Upload 168 files

docs/notebooks/ex0 - intro to python - creating pi.ipynb

@@ -0,0 +1,189 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3AArWF6_azgf"
+      },
+      "source": [
+        "## Libraries\n",
+        "\n",
+        "We use `numpy` to generate random numbers. We use `imgeio` to create animations, e.g. the `GIF` file. We use `matplotlib` to create plots of which we will save as images. We use `tqdm` to create progress bar for the for loop we will use."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "89k2LKWbazgh"
+      },
+      "outputs": [],
+      "source": [
+        "import numpy as np\n",
+        "import imageio as iio\n",
+        "import matplotlib.pyplot as plt\n",
+        "from tqdm import tqdm"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4xzyEzA7azgi"
+      },
+      "source": [
+        "## Algorithm\n",
+        "\n",
+        "We need to create 2 plots using `matplotlib`. The first plot we drop a bunch of dots on the circle of which we color code the dots that has $x$ and $y$ coordinates that is under the constraint: $x^2 + y^2 < 1$. The second plot we plot the value of estimated $\\pi$."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!mkdir output"
+      ],
+      "metadata": {
+        "id": "ZN5wfTxPbFGX"
+      },
+      "execution_count": 4,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import math"
+      ],
+      "metadata": {
+        "id": "UcXghRnIbgZF"
+      },
+      "execution_count": 8,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "RqUtYt9Eazgj",
+        "outputId": "fab38634-2a58-4a24-dbc2-7f9dd90816fe"
+      },
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "100%|██████████| 100/100 [01:23<00:00,  1.20it/s]\n"
+          ]
+        }
+      ],
+      "source": [
+        "n_range = np.arange(100, 20001, 100)\n",
+        "list_of_pi = list(np.repeat(None, len(n_range)))\n",
+        "\n",
+        "for i in tqdm(range(len(n_range))):\n",
+        "    plt.figure(figsize=(10, 4))  # Create a new figure with specified size\n",
+        "    n = n_range[i]  # Current sample size\n",
+        "    x = np.random.uniform(0, 1, n)  # Generate n random x-coordinates in [0, 1]\n",
+        "    y = np.random.uniform(0, 1, n)  # Generate n random y-coordinates in [0, 1]\n",
+        "    colors = ((x**2 + y**2) < 1).astype(int)  # Determine points inside unit circle\n",
+        "    this_pi = colors.sum()/n*4  # Calculate the estimate of pi\n",
+        "    list_of_pi[i] = this_pi  # Store the pi estimate in the list\n",
+        "    plt.subplot(1, 2, 1)  # Prepare to plot in the first subplot\n",
+        "    plt.title(f'Sample size: {n_range[i]}')  # Set title with current sample size\n",
+        "    plt.scatter(x, y, c=colors)  # Plot points with colors based on their position\n",
+        "    plt.subplot(1, 2, 2)  # Prepare to plot in the second subplot\n",
+        "    plt.title(f'Estimated pi = {np.round(this_pi,3)} vs. Actual pi = {np.round(math.pi,3)}')  # Set title with current pi estimate\n",
+        "    plt.plot(n_range[:i+1], list_of_pi[:i+1], '-ok')  # Plot the pi estimate progression\n",
+        "    plt.axhline(y=math.pi, color='r', linestyle='-')  # Add a line for the actual value of pi (3.14)\n",
+        "    plt.savefig(\n",
+        "        f'./output/img_{i}.png',\n",
+        "        transparent=False,\n",
+        "        facecolor='white'\n",
+        "    )  # Save the current figure\n",
+        "    plt.close()  # Close the figure to free memory\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3Mt3SQJzazgj"
+      },
+      "source": [
+        "## Animation\n",
+        "\n",
+        "We use `imageio` to create the animation. Running the above code, we have saved many plots in a sub-folder called `output`. We will now go into the folder and grab all the images. We use `imageio` to pack all these images together into an animation and save it in the main directory."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 13,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "EA4u1xltazgj",
+        "outputId": "c6c82194-e0bb-4ffa-fb69-1ae40d5f6998"
+      },
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "100%|██████████| 100/100 [00:00<00:00, 117.60it/s]\n"
+          ]
+        }
+      ],
+      "source": [
+        "frames = []\n",
+        "for i in tqdm(range(len(n_range))):\n",
+        "    image = iio.v2.imread(f'./output/img_{i}.png')\n",
+        "    frames.append(image)\n",
+        "\n",
+        "iio.mimsave(\n",
+        "    './pi.gif', # output gif\n",
+        "    frames,          # array of input frames\n",
+        "    fps = 5)         # optional: frames per second\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FykOuyjDazgk"
+      },
+      "source": [
+        "Enjoy your animation! You can check out the final product [here](https://github.com/yiqiao-yin/WYNAssociates/blob/main/figs/pi.gif)."
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": ".venv_specimen",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.9.7"
+    },
+    "orig_nbformat": 4,
+    "vscode": {
+      "interpreter": {
+        "hash": "c806452b2bc06941f39b1e669d8e4e56c296f09b7ac9df653e29bb221a7c7e72"
+      }
+    },
+    "colab": {
+      "provenance": []
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/notebooks/ex0 - intro to python.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.3"},"colab":{"name":"ex0 - intro to python.ipynb","provenance":[{"file_id":"1JvFihr4n2BRQk5Z2yq8We3-kUVyh3E2a","timestamp":1653574498662},{"file_id":"1CyCkorDS44LaXsnpKIahzkgPfoVlypcc","timestamp":1645423944383},{"file_id":"1FwtNuKQIcKOsIQyfJiti8crBrYuhY3jA","timestamp":1638891872672}],"collapsed_sections":["-Kg-TBkI0mS-"],"toc_visible":true}},"cells":[{"cell_type":"markdown","metadata":{"id":"UTt709mbMcfP"},"source":["# Introduction to Basic Python Command, Data Structures, and Variables"]},{"cell_type":"markdown","metadata":{"id":"k8mvv6_oMcfU"},"source":["\n","\n","```\n","# This is formatted as code\n","```\n","\n","## Motivating Example\n","\n","This is an example Python program that displays the text “Hello, world!” followed by a description of how many pets someone has."]},{"cell_type":"code","metadata":{"id":"eIgsD9AbMcfV","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1637366242300,"user_tz":480,"elapsed":4,"user":{"displayName":"Sam Scarborough","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Ghk5loPmSbpeoTj0WqLVWGK-2gSQlGreDneEmgK=s64","userId":"18324888543492053717"}},"outputId":"051904a6-93ef-4a0f-9d15-35c63d16b440"},"source":["# This lines prints hello world \n","print (\"Hello World!\")\n","print (\"I have no dogs\")"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Hello World!\n","I have no dogs\n"]}]},{"cell_type":"code","metadata":{"id":"gAKwkkfpMcfY","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1637366244880,"user_tz":480,"elapsed":566,"user":{"displayName":"Sam Scarborough","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Ghk5loPmSbpeoTj0WqLVWGK-2gSQlGreDneEmgK=s64","userId":"18324888543492053717"}},"outputId":"f3d1a769-9735-430b-ca14-f4bbd39faed9"},"source":["# Printing text\n","print(\"Hello World!\")\n","\n","# Printing how many pets you have\n","print(\"I have 2 dogs.\")"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Hello World!\n","I have 2 dogs.\n"]}]},{"cell_type":"markdown","metadata":{"id":"d_yElViMMcfd"},"source":["When you use print, each new thing that you want to display will be shown on a new line. If you want to display text, you need to write the word print, followed by the text in quotes between parentheses: (\"insert your text here\").\n","\n","Do you notice those lines with # signs in front? Those are called comments. Comments are really useful to tell people what's happening in your code, as well as a reminder for yourself when you review what you've written. The comments aren't actual code, so you don't need to worry about Python reading them when your programs are running. In other words, they don't get printed out."]},{"cell_type":"markdown","metadata":{"id":"7aS9vB1ieicC"},"source":["### Try it out: Introduce yourself!\n","\n","Write a program that prints your name and something about yourself. Here’s an example of what your program could display: \n"," > `My name is Arya.`\n"," \n"," > `I like writing Python programs.`\n"," \n"," Please comment each line of your code informatively."]},{"cell_type":"code","metadata":{"id":"r6MOXpihMcff"},"source":["# Write your code here. \n","print (\"My name is Patrick\")\n","print (\"I like writing Python programs.\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BnnugRhxMcfh"},"source":["Watch [this](https://youtu.be/Jto6IBy9z9k) video introducing you to **variables.**\n","\n","Often, when we're programming, we need to store some pieces of information. A good way to do this is by using **variables**. Variables are like containers: you can put things into them and extract their contents later.\n","\n","To declare a variable, there are three steps:\n","\n","1.   Come up with a name for your variable\n","2.   Write an equals sign (=)\n","3.   Put the **value** you want to store in your variable after the = sign"]},{"cell_type":"markdown","metadata":{"id":"ki6JxFG8wsKz"},"source":["### Example: What I ate today \n","\n","Let's see how we can use a variable to store information about what someone ate for breakfast and lunch today, and then print it."]},{"cell_type":"code","metadata":{"id":"feE_Pgk9tLgs","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1635690826718,"user_tz":240,"elapsed":201,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"0ee1429e-1724-4b92-c062-ce4c5a04a5b0"},"source":["# a variable to store what I ate tooday \n","meal = \"english muffin\" \n","print (\"breakfast:\" + meal)\n","\n","# what I ate for lunch \n","meal = \"pizza with ranch dip\" \n","print (\"lunch: \" + meal) \n","\n","number_of_meals = 5\n","\n","print(number_of_meals)"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["breakfast:english muffin\n","lunch: pizza with ranch dip\n","5\n"]}]},{"cell_type":"markdown","metadata":{"id":"MhBR2vFFwgBg"},"source":["Here, we made a variable called `meal`, and initially assigned it the value of `\"English muffin\"`. Later on, we reassigned the variable with a different value, `\"Pizza with ranch dip\"`.  Since we reassigned the variable, `\"English muffin\"` is completely replaced and we don't have access to that original value any more. If you wanted to keep the original value, you could consider defining a second variable, calling it something like `meal2`.\n","\n","Do you also notice that `+` sign in the `print` statements? You can use it to chain piece of text together when you want to display more than one word or phrase. This is called **concatenation**."]},{"cell_type":"markdown","metadata":{"id":"eK52VvZr1VwS"},"source":["## Variable types\n","\n","All of the code above has been used to record and display text information. In Python, text between quotation marks is called a **string**. Referring to examples from before, `\"English muffin\"` is a string, as is `\"I have 2 dogs.\"`\n","\n","However, we can store other pieces of information in variables aside from strings. For instance, we can store numbers in variables. There are two primary numeric variable types in Python: **integers** and **floats** (or floating-point numbers). Integers are whole numbers (e.g. 4), and floats are numbers with a decimal part (e.g. 8.26).\n","\n","Let's see an example of how to define two numeric variables and add them together."]},{"cell_type":"code","metadata":{"id":"QTwzSrG01UXt","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1636909661461,"user_tz":480,"elapsed":3,"user":{"displayName":"Braden Monroe","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjWHGIVTAWJFJX-igMgPLiTl88SyYPNavFiV4MCXA=s64","userId":"14803880866214127972"}},"outputId":"d21177d1-d9d1-4d06-ed40-1523c684c57f"},"source":["# define an integer \n","our_integer = 5 \n","\n","#define a float vriable \n","our_float = 3.65\n","\n","#add them together \n","result = our_integer + our_float \n","print(result)"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["8.65\n"]}]},{"cell_type":"markdown","metadata":{"id":"fbEDo3ki2Bm4"},"source":["Do you notice anything different about how these variables look? With strings, we need to use quotation marks to define them. But with numbers, we can leave out the quotes. That means that `2` and `\"2\"` are not the same thing: the first is an integer, while the second is a string.\n","\n","Notice that we can add numeric variables in Python using the `+` operator. So it behaves like traditional addition as well as the concatenation (putting strings together) that you saw earlier. The values of `our_integer` and `our_float` are being combined and stored in `our_result` in the above example.\n","\n","We have other mathematical operators available to us in Python too:\n","\n","*   `-` subtraction (e.g. `4 - 2` evaluates to `2`)\n","*   `*` multiplication (e.g. `3.4 * 10` evaluates to `34`)\n","*   `/` exact division (e.g. `5/2` evaluates to `2.5`)\n","*  `//` integer division (leaving off the decimal part, e.g. `5//2` evaluates to `2`)\n","\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"vy71G7A8tfER"},"source":["### Example: Currency Converter \n","\n","Watch [this](https://youtu.be/kQ-_WkdO5dY) video with an explanation of this example.\n","\n","This example demonstrates how to create and update integer and float variables. The following program converts any amount (in dollars) to rupees, and displays the result. The amount is stored in a variable `amt_in_dollars`, and the conversion rate is stored in a variable, `conversion_rate`. The converted amount is stored in `amt_in_rupees` and displayed."]},{"cell_type":"code","metadata":{"id":"LZdlzMoDtpUv","colab":{"base_uri":"https://localhost:8080/","height":197},"executionInfo":{"status":"error","timestamp":1636909787321,"user_tz":480,"elapsed":338,"user":{"displayName":"Braden Monroe","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjWHGIVTAWJFJX-igMgPLiTl88SyYPNavFiV4MCXA=s64","userId":"14803880866214127972"}},"outputId":"973e208b-5f77-444f-b5b4-1575033cd342"},"source":["#define variables: store values in variables \n","amt_in_dollars =  20\n","conversion_rate = 74.93\n","\n","#convert dollars in rupees\n","amt_in_rupees =  amt_in_dollars * conversion_rate\n","\n","print(\"$\" + amt_in_dollars + \" is Rs. \" + str(amt_in_rupees))"],"execution_count":null,"outputs":[{"output_type":"error","ename":"TypeError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-2-56ce25b6eb6d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mamt_in_rupees\u001b[0m \u001b[0;34m=\u001b[0m  \u001b[0mamt_in_dollars\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mconversion_rate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"$\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mamt_in_dollars\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\" is Rs. \"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mamt_in_rupees\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str"]}]},{"cell_type":"markdown","metadata":{"id":"Pk4qJvuM4rjB"},"source":["When we printed this time, we couldn't use the `+` operator directly between `\"$\"` and `amt_in_dollars`. That's because `\"$\"` is a string, but `amt_in_dollars` is an integer. So we had to convert the `amt_in_dollars` to a string, by using `str()` surrounding that variable.\n","\n","Do you notice that there are loads of decimal places after the decimal point in the final Rs. amount? Decimal mathematics isn't quite exact in Python, but you can see the result is pretty close."]},{"cell_type":"markdown","metadata":{"id":"QLDw0ZGXEsfZ"},"source":["### Try it out: Judge for the Olympics \n","\n","Write an Olympic Judging program that outputs the average scores from 3 different judges. Write a program with three variables for each score, called `score_a`, `score_b` and `score_C`  respectively. Set the scores (out of 10) to integers of your choice, and print them one after the other. Then, calculate the average of the three scores, and print the result.\n","\n","Your output should appear like this: \n","\n","```\n","Judge 1: 10\n","Judge 2: 9\n","Judge 3: 8\n","Average Score: 9\n","```\n","\n","This example combines several concepts from previous exercises and examples. If you're unsure about how to start, break down the problem into parts that are similar to the things we've just covered.\n","\n","*Hint: calculate the **total** first (and store this in a variable) before you calculate and print your average.*"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"02VgY3iFs3zZ","executionInfo":{"status":"ok","timestamp":1635691315393,"user_tz":240,"elapsed":195,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"e4c8065b-2e0a-4bc1-e6c5-9a858f36f073"},"source":["# Write your Olympic Judging program here:\n","score_a = 7\n","print(\"Judge 1: \" + str(score_a)) \n","\n","score_b =  6.5\n","print(\"Judge 2: \" + str(score_b)) \n","\n","score_c =  8\n","print(\"Judge 3: \" + str(score_c)) \n","\n","avg = (score_a + score_b + score_c)/3\n","\n","print (\"Average Score: \" + str(avg))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Judge 1: 7\n","Judge 2: 6.5\n","Judge 3: 8\n","Average Score: 7.166666666666667\n"]}]},{"cell_type":"markdown","metadata":{"id":"MxI-LK78lGMu"},"source":["## User Input\n","\n","Watch [this](https://youtu.be/BKT9A2N0vpU) video in preparation for this section."]},{"cell_type":"markdown","metadata":{"id":"WoMYCl8c9A4V"},"source":["\n","\n","When we run Python programs, we aren't restricted to specifying all the strings, integers and floats ourselves. We can also request input from users of your program. This is a powerful tool to make your programs flexible and personalised. AI applications frequently make use of user input to tailor an experience appropriately to a particular person."]},{"cell_type":"markdown","metadata":{"id":"998Efy5y-cdk"},"source":["### Example: A better Currency Converter\n","\n","Let's suppose that you just got back from a holiday to the US. You still have some dollar bills remaining, and you want to convert these dollars back to Rs. This program will respond to your requests and convert your money back for you. Give it a run! "]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"fJb70YHelOAu","executionInfo":{"status":"ok","timestamp":1635681351489,"user_tz":240,"elapsed":12359,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"0059b723-2c69-491e-f3cc-a29b4b986a2f"},"source":["#ask user for name \n","name = input(\"What is your name? \" )\n","\n","\n","amt_in_dollars = input(\"Hi, \" + name + \"! How many dollars do you want to convert? \")\n","print(amt_in_dollars)\n","\n","conversion_rate = 74.93\n","\n","amt_in_rupees = float(amt_in_dollars) * conversion_rate\n","\n","print(\"$\" + amt_in_dollars + \" is Rs. \" + str(amt_in_rupees))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["What is your name? Patrick\n","Hi, Patrick! How many dollars do you want to convert? 10\n","10\n","$10 is Rs. 691.8000000000001\n"]}]},{"cell_type":"markdown","metadata":{"id":"1-GHbmTs_T_A"},"source":["Isn't that neat? This code has read in your name, addressed you personally asking for how much money you want to convert, and performed the conversion for you.\n","\n","To get user input, we perform the following steps:\n","\n","\n","1.   Come up with a variable name to store your input\n","2.   Write `=`\n","3.   Write `input()`\n","4.   Within the parentheses, write the message that prompts the user to enter something\n","\n","By default, user input gets read in as a string. In the above example, we wanted to retrieve the dollar amount to convert from the user, but we don't want to store that as a string since we want to use it in calculations. So we need to convert it to an integer. We do this by enclosing that expression in `int()`: \n","\n","`amt_in_dollars = int(input(name + \", what dollar amount are you looking to convert? \"))`"]},{"cell_type":"markdown","metadata":{"id":"nJDueUiOmNxD"},"source":["### Try it out: A better way to Judge the Olympics\n","\n","Instead of setting the scores yourself, extend the Olympics judging program so that it asks for input from each judge, and then calculates the average of the three scores. The output should look like this:\n","\n","```\n","Judge 1, what’s your score? 10\n","Judge 2, what’s your score? 9\n","Judge 3, what’s your score? 8\n","Average score: 9\n","```\n","\n","This time, instead of storing the total in a separate variable from the average, try to jump to the average in just one step. Note that you can use parentheses to specify that you want an expression to be calculated before something else, like this:\n","\n","```\n","ans = (6 + 2 + 5) / 13\n","```\n","This way, the sum `6 + 2 + 5` gets calculated first, giving `13`. Then `13` is divided by `13` to give the answer, which is `1`. This value is stored in `ans`.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":528},"id":"Tlin7gBXsNZG","executionInfo":{"status":"error","timestamp":1635681821446,"user_tz":240,"elapsed":4578,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"349172e3-a88f-465d-bbc1-172fe57cebad"},"source":["# Write your Extended Olympic Judging program here: \n","score_a = input(\"Judge 1, what's your score? \")\n","score_b = input(\"Judge 2, what's your score? \")\n","score_c = input(\"Judge 3, what's your score? \")\n","sum = float(score_a) + float(score_b) + float(score_c)\n","\n","average = (sum/3)\n","print (\"Average Score: \" + str(average))"],"execution_count":null,"outputs":[{"output_type":"error","ename":"KeyboardInterrupt","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)","\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py\u001b[0m in \u001b[0;36m_input_request\u001b[0;34m(self, prompt, ident, parent, password)\u001b[0m\n\u001b[1;32m    728\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 729\u001b[0;31m                 \u001b[0mident\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreply\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstdin_socket\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    730\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.7/dist-packages/jupyter_client/session.py\u001b[0m in \u001b[0;36mrecv\u001b[0;34m(self, socket, mode, content, copy)\u001b[0m\n\u001b[1;32m    802\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 803\u001b[0;31m             \u001b[0mmsg_list\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_multipart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    804\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mzmq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mZMQError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.7/dist-packages/zmq/sugar/socket.py\u001b[0m in \u001b[0;36mrecv_multipart\u001b[0;34m(self, flags, copy, track)\u001b[0m\n\u001b[1;32m    624\u001b[0m         \"\"\"\n\u001b[0;32m--> 625\u001b[0;31m         \u001b[0mparts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mflags\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrack\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrack\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    626\u001b[0m         \u001b[0;31m# have first part already, only loop while more to receive\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32mzmq/backend/cython/socket.pyx\u001b[0m in \u001b[0;36mzmq.backend.cython.socket.Socket.recv\u001b[0;34m()\u001b[0m\n","\u001b[0;32mzmq/backend/cython/socket.pyx\u001b[0m in \u001b[0;36mzmq.backend.cython.socket.Socket.recv\u001b[0;34m()\u001b[0m\n","\u001b[0;32mzmq/backend/cython/socket.pyx\u001b[0m in \u001b[0;36mzmq.backend.cython.socket._recv_copy\u001b[0;34m()\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.7/dist-packages/zmq/backend/cython/checkrc.pxd\u001b[0m in \u001b[0;36mzmq.backend.cython.checkrc._check_rc\u001b[0;34m()\u001b[0m\n","\u001b[0;31mKeyboardInterrupt\u001b[0m: ","\nDuring handling of the above exception, another exception occurred:\n","\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)","\u001b[0;32m<ipython-input-6-88e91363812b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Write your Extended Olympic Judging program here:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mscore_a\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Judge 1, what's your score? \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mscore_b\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Judge 2, what's your score? \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py\u001b[0m in \u001b[0;36mraw_input\u001b[0;34m(self, prompt)\u001b[0m\n\u001b[1;32m    702\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parent_ident\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    703\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parent_header\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 704\u001b[0;31m             \u001b[0mpassword\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    705\u001b[0m         )\n\u001b[1;32m    706\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py\u001b[0m in \u001b[0;36m_input_request\u001b[0;34m(self, prompt, ident, parent, password)\u001b[0m\n\u001b[1;32m    732\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    733\u001b[0m                 \u001b[0;31m# re-raise KeyboardInterrupt, to truncate traceback\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 734\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mKeyboardInterrupt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    735\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    736\u001b[0m                 \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mKeyboardInterrupt\u001b[0m: "]}]},{"cell_type":"markdown","metadata":{"id":"N0iZS9dBC8k2"},"source":["## Conditionals\n","\n","Watch [this](https://youtu.be/jkM2dfuHeu4) video in preparation for this exercise.\n","\n","In all the programs above, we have a fixed order in which things will happen. For instance, with our judge scores above, we know that we will be taking in input from 3 judges, calculating the average and printing it out.\n","\n","However, we might want to specify some alternative ways in which a program could run. In plain English, this means something like:\n","\n","\"if something is the case, execute the program in one particular way; otherwise do it another way\"\n","\n","We can achieve this in Python by using **Boolean** variables. A Boolean is another type of variable that can take on just two values: `True` and `False`.  "]},{"cell_type":"markdown","metadata":{"id":"M0s5cBa9GD0d"},"source":["### Example: Self-Driving Car at a Traffic Light\n","\n","We're sitting in a nice Tesla that is approaching a traffic light. The Tesla is looking out to see whether the green light is illuminated."]},{"cell_type":"code","metadata":{"id":"plfnn9dYuTxH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1635691432033,"user_tz":240,"elapsed":183,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"4503212a-43db-49df-b1d8-fe3208e13b37"},"source":["# We define a new Boolean variable called green_light \n","# In this case, the green light is illuminated, so we set the variable to True\n","green_light = True\n","\n","if green_light: #<- single variable\n","  print(\"Good to go!\")\n","else: \n","  print(\"Don't go yet...\")"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Good to go!\n"]}]},{"cell_type":"markdown","metadata":{"id":"N-4vV-kGHAji"},"source":["Nice, our Tesla can sail through the green light with no problems. But now we're approaching another traffic light that is not green:"]},{"cell_type":"code","metadata":{"id":"I31ZNrO0HJ5s","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1635691465407,"user_tz":240,"elapsed":329,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"d2a0052b-536d-475b-8a22-979bf8b47e6f"},"source":["# In this case, we're approaching a red light, so we set the variable to False\n","green_light = False\n","\n","if green_light: \n","  print(\"Good to go!\")\n","else: \n","  print(\"Don't go yet...\")"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Don't go yet...\n"]}]},{"cell_type":"markdown","metadata":{"id":"CqQMCBj2HV1A"},"source":["Notice above that each `print()` statement above is indented and preceded by one of two lines:\n","\n","```\n","if green_light:\n","```\n","or\n","```\n","else:\n","...\n","```\n","\n","When you write `if`, Python checks to see whether the variable or expression after it evaluates to `True`. If it does, then the indented code after it will execute, and Python will skip the indented code after the `else`. If it evaluates to `False`, then the indented code after the `if` statement is skipped, and the indented code after `else` will execute. The examples above show both of these in action.\n","\n","Note that when using `if` and `else` statements, you need to end each of these lines with a colon (`:` ) and indent the subsequent code with a tab."]},{"cell_type":"markdown","metadata":{"id":"0VEEKk8CkoYU"},"source":["## Comparison operators: `==, !=, >, <, >=, <=`\n","\n","So far, we've been setting variables directly to Boolean values. Sometimes, however, we want to compare other values or expressions to each other. To do this, we can use **comparison operators**. \n","\n","There are six main types of comparison operator:\n","\n","1.   `==` checks to see if two values/expressions are equal\n","2.   `!=` checks to see if two values/expressions are not equal\n","3.   `>` checks to see if the left-hand value/expression is greater than the right-hand value/expression\n","4.   `<` checks to see if the left-hand value/expression is less than the right-hand value/expression\n","5. `>=` checks to see if the left-hand value/expression is greater than or equal to the right-hand value/expression\n","6. `<=` checks to see if the left-hand value/expression is less than or equal to the right-hand value/expression\n","\n","If the comparison condition is satistfied, the whole expression will evaluate to `True`. If it's not, the whole expression evaluates to `False`.\n","\n","Let's take a look at some simple examples of this. Run the following cell and examine the outputs:"]},{"cell_type":"code","metadata":{"id":"pFQfeN1inQlB","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1635691820336,"user_tz":240,"elapsed":198,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"01a33815-9a92-4148-f69f-a9221301b3ec"},"source":["# Define a variable called test\n","test = 2\n","\n","# Let's see if test is equal to 2, and print out the result\n","result = test == 2\n","print(result)\n","\n","# Let's see if test is greater than or equal to 4, and print out the result.\n","# This time, we're going to do it all in one line\n","print(test >= 4)\n","\n","# Finally, let's see if the result of 3 - test is not equal to the result of test/2.\n","print(3 - test != test/2)"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["True\n","False\n","False\n"]}]},{"cell_type":"markdown","metadata":{"id":"SiduqIyXoLYM"},"source":["Comparison operators also work on strings, as you can see below if you run the cell. This program combines the comparison operators with an `if` statement to get some different outputs. Will anything be printed out in this case?"]},{"cell_type":"code","metadata":{"id":"DtHeOXtVhYyF"},"source":["# Define a test string\n","test_string = \"hello\"\n","\n","# Will anything be printed here?\n","if test_string == \"pink\":\n","  print(\"The test string is pink\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"YJxyjt9Lv5C4"},"source":["Here, since we didn't specify an alternative output to print out, nothing gets printed. It's fine not to have an `else` statement if you don't want it!"]},{"cell_type":"markdown","metadata":{"id":"NKorZAoBhdPt"},"source":["**For Fun**"]},{"cell_type":"code","metadata":{"id":"cBLughVGoWQl","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1635691920319,"user_tz":240,"elapsed":7182,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"3486b24e-817a-442d-d9a1-f6414eb23c79"},"source":["# Define a test string\n","test_string = input(\"hello, what colour is the test string? \")\n","\n","# Will anything be printed here?\n","if test_string == \"pink\":\n","  print(\"The test string is pink\")\n","\n","else: \n","  print(\"you are incorrect\")"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["hello, what colour is the test string? pink\n","The test string is pink\n"]}]},{"cell_type":"code","metadata":{"id":"ZBZxLnCTiBSc","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1635691931912,"user_tz":240,"elapsed":4382,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"0678a79a-0333-4cd5-cb7e-58442dc36ad0"},"source":["# Define a test string\n","test_string = input(\"hello, what colour is the test string? \")\n","\n","# Will anything be printed here?\n","result = test_string == \"pink\"\n","print(result)"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["hello, what colour is the test string? pink\n","True\n"]}]},{"cell_type":"markdown","metadata":{"id":"liKM-vZIpfOh"},"source":["## Now you try!"]},{"cell_type":"markdown","metadata":{"id":"hq73OZ5GptOi"},"source":["**Write** two expressions of your own, with the first one evaluating to `True` and the second one evaluating to `False`. Print out the values of your expressions as you go."]},{"cell_type":"code","metadata":{"id":"apqxIhF7pq4F","outputId":"71c4bf4f-e20a-448a-91d4-8e1d841e55f6"},"source":["# Write your first expression here\n","test = 4 \n","\n","result = test >= 1 \n","print(result)\n","\n","result = test<=2\n","print(result)\n","\n","# Write your second expression here"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["True\n","False\n"]}]},{"cell_type":"markdown","metadata":{"id":"rLLB9mKwrMIQ"},"source":["## Controlling the traffic lights\n","\n","Remember the self-driving car example from before? It was pretty good, but we want to make it better. There, we only specified whether or not a light was green, but there are actually two alternatives for when a light is not green: it could be red or yellow.\n","\n","This time, we're going to let a user input what colour they want the traffic light to be. Then we'll tell the Tesla what it needs to do depending on the colour of the light that was entered."]},{"cell_type":"code","metadata":{"id":"SQ2uiLJkspI8"},"source":["# Get user input about the traffic light colour\n","light = input(\"What colour is the traffic light? \")\n","\n","# Don't worry about this line; it's just formatting the string so that there's\n","# no extra whitespace at the ends and it's all in lower case\n","light = light.lower().strip()\n","\n","# Tell the car what it needs to do based on the colour of the light\n","if light == \"red\": \n","  print(\"Don't go yet... \")\n","elif light == \"yellow\":\n","  print(\"Get ready to go... \")\n","elif light == \"green\":\n","  print(\"Good to go!\")\n","else:\n","  print(\"Huh?\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lkOmgMeLv3Hd"},"source":["This example is a little more involved than before. There are two main things to notice here:\n","\n","\n","1.   The above program uses the word `elif`. This is useful if you have multiple conditions you want to check. `if`/`else` on their own is fine if there's just one condition you want to check, but here we're seeing if the light is red, yellow or green.\n","2.   The final `else` statement might seem unnecessary, but we've included it in case the user enters something other than red, yellow or green. If a Tesla saw a purple light, it might be confused!\n","\n"]},{"cell_type":"markdown","metadata":{"id":"UjyGTyglxG1e"},"source":["## Logical operators: `and`, `or`, `not`\n","\n","In the above examples, we were only checking one expression at a time. But sometimes we want to check more than one at once, or negate some expression that we've already checked. There are three keywords in Python for this:\n","\n","\n","1. `and` checks if both the left-hand and right-hand expressions are `True`. If they are, the whole expression evaluates to `True`; otherwise it evaluates to `False`.\n","\n","2. `or` checks if either the left-hand or right-hand expressions are `True` (or both of them). If at least one of these expressions evaluates to `True`, then the whole expression evaluates to `True`; otherwise it evaluates to `False`.\n","\n","3. `not` will reverse the value of whatever expression we are checking. So `not True` is the same as `False`, and `not False` is the same as `True`.\n","\n","Run the cell below to see the results of each of the expressions within."]},{"cell_type":"code","metadata":{"id":"WBh9TwOQ6xSG"},"source":["# Define some Booleans\n","bool_a = 2 * 3 == 6  \n","bool_b = 9 - 4 != 5  \n","\n","print(bool_a)\n","\n","print(bool_b)\n","\n","# Expression 1\n","print(bool_a and bool_b)\n","\n","# Expression 2\n","print(bool_a or bool_b)\n","\n","# Expression 3\n","print(not bool_b)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-Kg-TBkI0mS-"},"source":["## Loops\n","\n","To motivate the need for while loops, let's re-visit the if-condition. Recall that an `if` statement allows you to do something UNLESS a condition is `False`. If the condition is `False`, nothing is executed. What if we want to do something UNTIL a condition becomes `False`?  This is exactly what loops are for: to execute a particular task repeatedly until some condition no longer holds `True`. \n","\n","Watch [this](https://youtu.be/ckltZHCqNnA) video in preparation for this topic."]},{"cell_type":"markdown","metadata":{"id":"V4oGzvQS05Gu"},"source":["## While loops: A personalized Self-Driving Car\n","\n","Let's explore this with an example. Let’s go back to our self-driving Tesla, but this time, let's personalize it. Suppose that our Tesla (let's call it *Jarvis*), only starts when the user summons it with a command, say \"*Jarvis, energize!*\". If we don’t enter that command, the car doesn’t start. We know this can be programmed with a simple `if` condition, like so:  "]},{"cell_type":"code","metadata":{"id":"7pWsULq03k9d","outputId":"02644562-8714-4fe9-a542-b901b3d16ff9"},"source":["# Define the predetermined summon in a variable\n","summon = \"Jarvis, energize!\"\n","\n","# Ask the user for the command\n","command = input(\"What’s your command? \")\n","\n","# Tell the car what to do based on the correctness of the command\n","if command == summon:\n","  print(\"Your wish is my command!\")\n","else: \n","  print(\"Sorry, I don’t understand.\")"],"execution_count":null,"outputs":[{"name":"stdin","output_type":"stream","text":["What’s your command?  Jarvis, energize!\n"]},{"name":"stdout","output_type":"stream","text":["Your wish is my command!\n"]}]},{"cell_type":"markdown","metadata":{"id":"Xk9njZY44Num"},"source":["… Now, instead of just giving the user a single try, what if we want to let the user keep guessing until they guess correctly?\n","\n","We can do that with something called a `while` loop. Take a look at this program, and give it a whirl:  \n","\n"]},{"cell_type":"code","metadata":{"id":"OI5jHY3q4p6M","outputId":"84390d81-62cd-44d4-91d9-2d950931637b"},"source":["# Define the predetermined summon in a variable\n","summon = \"Jarvis, energize!\"\n","\n","# Ask the user for the command\n","command = input(\"What’s your command? \")\n","\n","# While the command isn't the summon we're looking for... then execute the indented block\n","while command != summon: \n","  print(\"Sorry, I don't understand.\") \n","  command = input(\"What's your command? \")\n","  \n","# This line of code will only execute when the loop is finished, i.e., when command is the predetermined summon\n","# Note how this line is NOT indented like the statements inside the while-loop\n","#will exit the loop when command == summon\n","print(\"Your wish is my command!\")"],"execution_count":null,"outputs":[{"name":"stdin","output_type":"stream","text":["What’s your command?  Jarvis!\n"]},{"name":"stdout","output_type":"stream","text":["Sorry, I don't understand.\n"]},{"name":"stdin","output_type":"stream","text":["What's your command?  Jarvis!\n"]},{"name":"stdout","output_type":"stream","text":["Sorry, I don't understand.\n"]},{"name":"stdin","output_type":"stream","text":["What's your command?  Jarvis, energize!\n"]},{"name":"stdout","output_type":"stream","text":["Your wish is my command!\n"]}]},{"cell_type":"markdown","metadata":{"id":"_mr0HZ516N73"},"source":["In simple English, the `while` loop can be translated as: \"Until `command` is not the same as `summon`, ask user for input.\"\n","\n","Here's what a `while` loop looks like: \n","```\n","while condition\n","  ... do something ... \n"," ```\n","It looks a lot like an `if` statement. The only difference is the word \"while\" in place of the word \"if\". The condition still has to be something that evaluates to a boolean. The code under the `while` loop is the body. If the condition is `True`, it executes. \n","\n","EVERY TIME it executes, the condition is checked again. Unless the condition has become `False`, the body runs again."]},{"cell_type":"markdown","metadata":{"id":"hw_tTwE7-wRp"},"source":["## Try it out:  Free tickets for the IPL \n","\n","You just landed 20 tickets for the IPL finals and generously want to give it away. With your recently acquired programming chops, you decide to write a program that keeps track of the tickets you have left. While there are still tickets left with you, ask the user how many tickets they would like to buy. Then print out how many are left with you after the purchase.\n"," \n","Bonus points if your program can catch the case where your user tries to buy more tickets than you have left with you. In that case, you should print a message to the yser saying that their request isn’t possible. \n","\n","Here's an example of what one run of the program might appear like: \n","\n","```\n","I'm giving away free tickets for the IPL. \n","\n","How many do you need? 5 \n","Great! I have 15 tickets left. \n","\n","How many do you need? 17\n","Sorry, I don't have that many. \n","\n","How many tickets do you need? 15\n","Great! I have 0 tickets left. \n","\n","All sold out!\n","```\n"]},{"cell_type":"markdown","metadata":{"id":"PuSx33yMuHXc"},"source":["Hints:\n","* You can set a `while` loop header condition to *anything* that evaluates to a Boolean. In this case, we want to make sure that there are still tickets left. This means that we'll want to compare the ticket count to something. What will this comparison look like?\n","* Suppose that you store the number of tickets that someone requests in a variable called `tickets`. Then after that value is stored, you can subtract it from `tickets_left` to figure out the new number of remaining tickets. This updated value will then get checked again in the next run of the while loop."]},{"cell_type":"markdown","metadata":{"id":"Y4SjoSCit3Tk"},"source":["### Enter your code below"]},{"cell_type":"code","metadata":{"id":"SdAMVtvreaur","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1637470059955,"user_tz":480,"elapsed":6794,"user":{"displayName":"Sam Scarborough","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Ghk5loPmSbpeoTj0WqLVWGK-2gSQlGreDneEmgK=s64","userId":"18324888543492053717"}},"outputId":"ecd86f35-f21f-4981-fc5d-76f0119d3360"},"source":["# Complete the IPL ticket-giveaway program. \n","print(\"I'm giving away free tickets for the IPL\")\n","\n","total = 21\n","request = input(\"How many tickets do you need? \")\n","\n","while float (request) >= total: \n","  print(\"Sorry, I don't have that many\")\n","  request = input(\"How many tickets would you like to request now? \")\n","\n","sum = total - 1 - int(request)\n","\n","print(\"Great, I have \" + str(sum) + \" tickets left\")"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["I'm giving away free tickets for the IPL\n","How many tickets do you need? 3\n","Great, I have 17 tickets left\n"]}]},{"cell_type":"markdown","metadata":{"id":"srLwyhk9Djkh"},"source":["## For loops\n","\n","A while loop works well when we don't know in advance how many times we will have to do something. But what if we know exactly how many times you'll have to do something?\n","\n","We can use a `for` loop! Here's an example of what it looks like. \n","\n","Watch [this](https://youtu.be/jBVv-5UuFQc) video in preparation for this topic."]},{"cell_type":"code","metadata":{"id":"WHx66S3UHNMO","outputId":"21d5f4f3-6d85-4d41-c9f6-4950e5fab348"},"source":["for i in range(4): \n","  # Do this 4 times\n","  print(\"This is iteration number: \" + str(i))"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["This is iteration number: 0\n","This is iteration number: 1\n","This is iteration number: 2\n","This is iteration number: 3\n"]}]},{"cell_type":"markdown","metadata":{"id":"2aYA1T5hHZZV"},"source":["A `for` loop consists of three main things: \n","*   You say \"`for i in range`\"\n","*  An integer in parentheses, which is the number of times you want to do something\n","*  The indented code, called body, which will execute repeatedly"]},{"cell_type":"markdown","metadata":{"id":"iwMKMB78nsvp"},"source":["### Example: A more secure Self-Driving Car \n","\n","Our friend at Tesla, Elon Musk, just told us it isn’t safe to let the user have unlimited go’s at guessing the command. He recommends we give the user only five tries to guess the command. Let’s see how we might use our handy-dandy `for` loop here: "]},{"cell_type":"code","metadata":{"id":"2TNFWR9gn9on","outputId":"82ab3e47-9446-4093-e932-bf30ce0bc42a"},"source":["# Define the predetermined summon in a variable\n","summon = \"Jarvis, energize!\"\n","\n","# Try the command five times\n","for i in range(5): \n","  command = input(\"What’s your command? \")\n","  if command == summon:\n","     print(\"Your wish is my command!\")\n","     break\n","  else: \n","     print(\"Sorry, I don’t understand!\")"],"execution_count":null,"outputs":[{"name":"stdin","output_type":"stream","text":["What’s your command?  Jarvis, energize!\n"]},{"name":"stdout","output_type":"stream","text":["Your wish is my command!\n"]}]},{"cell_type":"code","metadata":{"id":"zDSBTuU84oJ9","outputId":"4d590581-591e-4379-dbce-78007234b22f"},"source":["for i in range(4):\n","  command = input(\"what's up\")"],"execution_count":null,"outputs":[{"name":"stdin","output_type":"stream","text":["what's up 1\n","what's up 2\n","what's up a\n","what's up b\n"]}]},{"cell_type":"markdown","metadata":{"id":"iYXMPBbmo-s8"},"source":["Notice a new `break` statement in the `if` condition? When the condition is `True`, the break statement skips the entire loop even before it finishes all its iterations. In the above program, it ensures that the user is not asked to give the command again if they've gotten it right before the first five tries."]},{"cell_type":"markdown","metadata":{"id":"dx76IiuieRn3"},"source":["### Try it out: Judging for the Olympics v3 \n","\n","Remember the Olympics Judging program from earlier? Re-write it with a for-loop. As a reminder, the program asks a score from each of the three judges, and then calculates the average. The output looks like this: \n","\n","```\n","Judge 1, what’s your score? 10\n","Judge 2, what’s your score? 9\n","Judge 3, what’s your score? 8\n","The average score is 9. \n","\n","```\n","\n"]},{"cell_type":"code","metadata":{"id":"VnklZEgbeYR7"},"source":["# Write your new version of the Olympics Judging program with for loops here.\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"87_LZVcq1TeU"},"source":["## Lists\n","\n","Watch [this](https://youtu.be/da8cmMaZ_8E) video in preparation for this topic.\n","\n","Another common data type is lists.  You can think of lists as storing multiple variables, or a collection of variables, in one place.  It's also important to note that the variables are in a particular order and are each assigned an index in the list. "]},{"cell_type":"code","metadata":{"id":"higKQZtE2DAY","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1636913570792,"user_tz":300,"elapsed":9,"user":{"displayName":"Patrick Funk","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"06363259130993623477"}},"outputId":"c334a616-d5b0-4e17-aff3-628567a9eb43"},"source":["# Let's create a grocery list for when we go to the store\n","groceries = []\n","groceries.append('apples')\n","groceries.append('milk')\n","groceries.append('bread')\n","groceries.append('cheese')\n","\n","\n","\n","for var, item in enumerate(groceries):\n","  print(item + ' is at index: ' + str(var))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["apples is at index: 0\n","milk is at index: 1\n","bread is at index: 2\n","cheese is at index: 3\n"]}]},{"cell_type":"markdown","metadata":{"id":"6sRqp6-ty_ZA"},"source":["## Try it out: Best Foot Forward\n","You’re applying to Stanford and you realize you want to submit only the best score from all your attempts at the SAT. The scores from all the attempts is stored in a list, called `scores`. Write a program that traverses the list and then prints the highest score from the list. \n","\n","**Note**: Do not use the built-in function `max` in your program!"]},{"cell_type":"code","metadata":{"id":"YdtnOAaPy-x7"},"source":["# Complete this program to find and print the highest score from the list\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Z3fkjPLaqsHg"},"source":["## Functions\n","\n","Watch [this](https://youtu.be/_vCWEAZLXXs) video in preparation for this topic.\n","\n","A Python function is similar to how we think of functions in math. You've actually already made use of functions earlier-- `range`, `input` and `append` are all examples of functions. You can think of these as pre-defined functions in Python that can be called for various uses. Now, you'll learn how to make your own functions.\n","\n","Let's say you're building a chatbot, Jarvis, which needs to greet the user every time it's summoned, like so:"]},{"cell_type":"code","metadata":{"id":"OHKWTDRRBZRA","outputId":"af0682aa-ef43-463c-8358-6e93853883a3"},"source":["print(\"Hi!\")\n","print(\"Allow me to introduce myself. I'm Jarvis, and I can call and mesage people for you. I can also share a joke or two.\")\n","print(\"What can I help you with?\")"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi!\n","Allow me to introduce myself. I'm Jarvis, and I can call and mesage people for you. I can also share a joke or two.\n","What can I help you with?\n"]}]},{"cell_type":"markdown","metadata":{"id":"3XpSjoulDNzk"},"source":["Every time a user summons Jarvis, we call these three lines of code. Even if Jarvis is summoned 3-4 times, that's a lot of lines of code required. Instead, we can assign these lines of code to a `function`. We'll call the function `greeting`, and define it in the following way. \n","\n","Every function you define contains the word `def`, the name of your function, a pair of parenthses, and a colon, followed by some code underneath. When we define a function, nothing actually happens in the console. Try this:"]},{"cell_type":"code","metadata":{"id":"fSI9u8E3FmkD"},"source":["def greeting(): \n","  print(\"Hi!\")\n","  print(\"Allow me to introduce myself. I'm Jarvis, and I can call and mesage people for you. I can also share a joke or two.\")\n","  print(\"What can I help you with?\") "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_5SX8EKeGKYK"},"source":["At this point, we have merely taught Jarvis how to print a greeting. Now we have to tell Jarvis to actually do it, and we do this by calling the function, like so:"]},{"cell_type":"code","metadata":{"id":"2UuaZo-qGVGd","outputId":"72cd619f-5efe-47dc-c525-dc82c00a6932"},"source":["greeting()"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi Alex!\n"]}]},{"cell_type":"markdown","metadata":{"id":"XQBUr90cHHVq"},"source":["Each time we call it, the code in the function gets executed. This is part of why functions are valuable. They allow us to save some code for later, and then use and re-use it without typing it out each time.\n","\n","## Variable Scope in Functions (optional)\n","\n","For the purpose exploring functions further, let's use a shorter greeting and add the user's name to it. Then we can declare a variable with the name, like we normally would."]},{"cell_type":"markdown","metadata":{"id":"Kc7Dixf6IlNq"},"source":["But watch out, if you do this, the variable `name` is only available within the function. If you try to use `name` outside the function, it causes an error. Do you see an error when you run this program?"]},{"cell_type":"code","metadata":{"id":"tf07EEMBIeaf","outputId":"6334b6ea-5c00-401b-f771-7ea64e5bc8a0"},"source":["def greeting(): \n","  name = \"Arya\"\n","  print(\"Hi \" + name + \"!\")  \n","  \n","greeting()\n","print(name)"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi Arya!\n","Yin\n"]}]},{"cell_type":"markdown","metadata":{"id":"2qYfeoApJMxm"},"source":["But here's where Python gets a little funky. If you declare the variable OUTSIDE a function, you can still access it INSIDE the function. Take a look at this:\n"]},{"cell_type":"code","metadata":{"id":"bC-BRSDBItU8","outputId":"1acaa706-21f7-4e34-9f7c-5ab540420bb5"},"source":["name = \"Arya\"\n","\n","def greeting(): \n","  print(\"Hi \" + name + \"!\")\n","  \n","greeting()\n","print(name)"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi Arya!\n","Arya\n"]}]},{"cell_type":"markdown","metadata":{"id":"tfpyxxy4JgGV"},"source":["Here's another place where Python gets REALLY funky. If a variable is declared outside a function, and you try to CHANGE it inside the function, you actually create ANOTHER variable of the same name! In this example, there are two variables called `name`. One of them lives inside `greeting`, and the other lives outside of it. Run this code and see what happens."]},{"cell_type":"code","metadata":{"id":"pHDPFzZPJY7O","outputId":"a11db91a-1f8d-46ec-ae44-c856ecdf4ab3"},"source":["name = \"Arya\"\n","\n","def greeting():\n","  name = \"Alex\"\n","  print(\"Hi \" + name + \"!\")\n","  \n","greeting()  # This print statement will refer to the version of name inside greeting()\n","print(name) # This print statement will refer to the version of name on the first line"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi Alex!\n","Arya\n"]}]},{"cell_type":"markdown","metadata":{"id":"BITv9AWDL8Z_"},"source":["## Functions with Parameters\n","\n","Watch [this](https://youtu.be/wHiMhywV8j0) video in preparation for this topic.\n","\n","As it stands now, functions are pretty limited; they always do the same thing. The function `greeting` always greets Arya.\n","\n","We can make a function behave differently by re-assigning the variable `name` each time, but that would be annoying. Fortunately, there's a better way to do this, and it exists in the form of **parameters**, which are pieces of information you can give the function when you call them. \n","\n","A parameter is basically a variable name. You put the variable name in the function's parentheses, and you can then use the name in the function as though it were a variable you declared."]},{"cell_type":"code","metadata":{"id":"m7tWzdhkL_Xl"},"source":["def greeting(name): \n","  print(\"Hi \" + name + \"!\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FtFb-88eOwdK"},"source":["But where does this variable get its value from?\n","\n","You actually give the variable a value when you call the function, like the following lines of code. Give it a whirl, what do you get?\n"]},{"cell_type":"code","metadata":{"id":"hmSm7QnhO8gb","outputId":"d19c87ba-1457-46ab-eee2-b6a48f55c91c"},"source":["greeting(\"Arya\")\n","greeting(\"Alex\")\n","greeting(\"Adi\")"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi Arya!\n","Hi Alex!\n","Hi Adi!\n"]}]},{"cell_type":"markdown","metadata":{"id":"zzJ0rkydPyrJ"},"source":["Functions can also have **multiple** parameters. Let's say we want to print the current temperature with the greeting. Then the provide the paramters in paretheses, separated by commas."]},{"cell_type":"code","metadata":{"id":"CjtVvXorPl8n","outputId":"eaefb85f-0973-48d4-c984-1ff7c1b723bc"},"source":["# Function definition includes two parameters for name and temp\n","def greeting(name, temp): \n","  print(\"Hi \" + name + \"!\")\n","  print(\"It's \" + str(temp) + \" degree celsius today.\")\n","\n","# While calling the function, we provide the values for BOTH parameters\n","greeting(\"Arya\", 21)\n","greeting(\"Alex\", 37)"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["Hi Arya!\n","It's 21 degree celsius today.\n","Hi Alex!\n","It's 37 degree celsius today.\n"]}]},{"cell_type":"markdown","metadata":{"id":"3fcjs9xRUVqZ"},"source":["## Functions with Return Values\n","\n","When you write a function, you have the option of making it return a value. Here's an example of a function that returns a `string variable` with a greeting. \n","\n","Run the program. Notice how the first print statement is empty, and the second print statement prints the string returned by the `greeting` function."]},{"cell_type":"code","metadata":{"id":"vUPmszQhW5ix","outputId":"5234a06b-6028-445f-8de6-fa42e829711e"},"source":["# Defining a function that returns a greeting string \n","def greeting(name): \n","  return \"Hey \" + name + \"!\"\n","\n","# Here, a variable is declared and printed\n","my_greeting = \"\"\n","print(my_greeting)\n","\n","# Here, the variables is assigned to the value returned by the greeting function\n","my_greeting = greeting(\"Arya\")\n","print(my_greeting)"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["\n","Hey Arya!\n"]}]},{"cell_type":"markdown","metadata":{"id":"Kl6SojLhd0LU"},"source":["## Try it out: Can you predict what Dory sings? (optional)\n","\n","*Finding Dory* is an amazing film! Dory, the friendly but forgetful blue tang fish, often sings a song to herself and her friends to keep motivation high when they are feeling lost. The following program prints out what Dory sings. **Without running the program, can you predict its output?** "]},{"cell_type":"code","metadata":{"id":"euCyitMOZEpL","outputId":"45b7764a-4035-4013-a252-d90d615df01b"},"source":["def sing_song(friend, quote):\n","  song = \"I sing for \" + friend + \": \"\n","  for i in range(3): \n","    song = song + quote\n","  return song\n","\n","def sing_for_all(quote): \n","  friends = [\"Nemo\", \"Marlin\", \"Destiny\"]\n","  song = \"\"\n","  for friend in friends: \n","    print(song + sing_song(friend, quote))\n","    \n","sing_for_all(\"Just keep swimming. \")\n"],"execution_count":null,"outputs":[{"name":"stdout","output_type":"stream","text":["I sing for Nemo: Just keep swimming. Just keep swimming. Just keep swimming. \n","I sing for Marlin: Just keep swimming. Just keep swimming. Just keep swimming. \n","I sing for Destiny: Just keep swimming. Just keep swimming. Just keep swimming. \n"]}]},{"cell_type":"markdown","metadata":{"id":"BkmSWh9Lh1ju"},"source":["Now run the program and compare your solution to the output. How did you do? :)"]},{"cell_type":"markdown","metadata":{"id":"rBCUCuQUma7X"},"source":["## Try it out: Turn up the temperature\n","\n","You're probably used to reading the temperature in Celsius units. Unfortunately, your soon-to-be friends from Stanford don't understand Celsius; they are only used to Farenheit. To make any discussion about weather easier with them, write a program that helps you convert the temperature between the Celsius and Farenheit: \n","\n","1.   Write a function called `c_to_f` that takes an input `c_temp`, a temperature in Celsius, and converts it to `f_temp`, that temperature in Fahrenheit. It should then return `f_temp`. The equation you should use is: \n","\n",">  $F = (1.8 * C) + 32$\n","\n","\n","2.   Let’s test your function with a value of 0 Celsius. Print out your output.\n","\n"]},{"cell_type":"code","metadata":{"id":"DycsOkuspAXe"},"source":["# Define your function here\n","\n","# Call your function and print output here\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"erdiMaacuYLr"},"source":["# CONGRATULATIONS!\n","Well done for finishing this notebook! If you have any remaining questions, be sure to ask them when you get to the camp. We are excited to meet you!"]}]}

docs/notebooks/ex03 - run an installed neuralnet.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"ex3 - run an installed neuralnet.ipynb","provenance":[{"file_id":"196zIUCpGGm7Eb3oPxLqnRmXxK96Ejd1m","timestamp":1654164418111}],"collapsed_sections":[],"machine_shape":"hm","background_execution":"on","toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","source":["# MNIST Data\n","\n","According to Yann LeCun on this [site](http://yann.lecun.com/exdb/mnist/#:~:text=It%20is%20a%20good%20database,efforts%20on%20preprocessing%20and%20formatting.), the MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.\n","\n","It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. \n","\n","<p align=\"center\">\n","    <img src=\"https://github.com/AviatorMoser/keras-mnist-tutorial/blob/master/mnist.png?raw=1\"></img>\n","</p>\n","\n","## Research Question\n","\n","Can AI recognize pictures?\n","\n","## Agenda\n","\n","This is the standrad image data for developing neural network models and convolutional neural network models. \n","\n","- Neural Networks (1st meeting)\n","- Convolutional Neural Networks (2nd meeting)\n","\n","One you have the basic knowledge, there two tasks:\n","- First, you build a model to make good predictions on MNIST dataset\n","- Next, you will get a new data set that is similar and you will build advanced models to make good predictions on it. Plan: We use the MNIST Fashion dataset.\n","\n","Goal: To develop state-of-the-art CNN models.\n","\n","## Notes\n","\n","Most of the notes are taken using markdown area. To learn more diverse editting techniques for you to take quick notes using markdown, please click [here](https://colab.research.google.com/notebooks/markdown_guide.ipynb)."],"metadata":{"id":"ZgxIJ7VDkatT"}},{"cell_type":"markdown","source":["# Artificial Neural Networks"],"metadata":{"id":"VV3Qwh9lEiAp"}},{"cell_type":"markdown","source":["## Load Data"],"metadata":{"id":"Rcuw1nh5jWmO"}},{"cell_type":"code","source":["# import library\n","import tensorflow as tf"],"metadata":{"id":"HqWLoP6nlSDP","executionInfo":{"status":"ok","timestamp":1654356288931,"user_tz":240,"elapsed":4921,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":1,"outputs":[]},{"cell_type":"code","source":["# use tf API to get data\n","mnist = tf.keras.datasets.mnist\n","(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n","print(training_images.shape)\n","print(training_images[0].shape)"],"metadata":{"id":"_rKryrdplThI","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1654356288931,"user_tz":240,"elapsed":26,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"9c423693-b26c-4b57-d67f-59f14e4c3946"},"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n","11493376/11490434 [==============================] - 0s 0us/step\n","11501568/11490434 [==============================] - 0s 0us/step\n","(60000, 28, 28)\n","(28, 28)\n"]}]},{"cell_type":"markdown","source":["### Save Using Pickle\n","\n","Sometimes the data you loaded is extremely large in sizes so we use pickle file to dump your data inside. This is to prevent that you lose the memory and record. The way to do this is whenever you run a chunk of cell with large memory you want to make sure have a pickle file ready to dump your data inside."],"metadata":{"id":"JYncHvoPpUp_"}},{"cell_type":"code","source":["# import\n","import os\n","import pickle"],"metadata":{"id":"8u_UMgRZpi4H","executionInfo":{"status":"ok","timestamp":1654356288931,"user_tz":240,"elapsed":20,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":3,"outputs":[]},{"cell_type":"code","source":["# list directory\n","os.listdir()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Jf8ZZukupnOn","executionInfo":{"status":"ok","timestamp":1654356288932,"user_tz":240,"elapsed":21,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"dd21c7da-1cf0-4644-9742-744e389e2631"},"execution_count":4,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['.config', 'sample_data']"]},"metadata":{},"execution_count":4}]},{"cell_type":"code","source":["# dump: this means save the desired file into a pickle file\n","# note: a pickle file is a file ends with .pkl format\n","a = (training_images, training_labels), (test_images, test_labels)\n","with open('mnist_data.pkl', 'wb') as handle:\n","    pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL)"],"metadata":{"id":"aXLSlnL-plEB","executionInfo":{"status":"ok","timestamp":1654356288932,"user_tz":240,"elapsed":19,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":5,"outputs":[]},{"cell_type":"code","source":["# check if it exists in the current directory\n","os.listdir()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"el3SiQwoqACF","executionInfo":{"status":"ok","timestamp":1654356288932,"user_tz":240,"elapsed":18,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"b3face06-0bfd-437b-a019-fdbf18d3137b"},"execution_count":6,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['.config', 'mnist_data.pkl', 'sample_data']"]},"metadata":{},"execution_count":6}]},{"cell_type":"code","source":["# load: this means start loading the desired file from the pickle file\n","# note: this means that you do a with-open clause, and grab whatever \n","#       inside of the .pkl file and defined it as a new object\n","#       please make sure the name and the directory are correct\n","with open('mnist_data.pkl', 'rb') as handle:\n","    b = pickle.load(handle)\n","\n","# preview\n","type(b), len(b)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"SvEpbvb5p22H","executionInfo":{"status":"ok","timestamp":1654356288933,"user_tz":240,"elapsed":18,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"1ed0f6ac-e92a-48c4-cace-f78f7995cc60"},"execution_count":7,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(tuple, 2)"]},"metadata":{},"execution_count":7}]},{"cell_type":"code","source":["# what is b?\n","print(b[0][0].shape, b[0][1].shape, b[1][0].shape, b[1][1].shape)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Us3wRy54qWR6","executionInfo":{"status":"ok","timestamp":1654356288933,"user_tz":240,"elapsed":16,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"06dbcb3f-13ea-4957-8ede-cec1d0a9d0c3"},"execution_count":8,"outputs":[{"output_type":"stream","name":"stdout","text":["(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)\n"]}]},{"cell_type":"markdown","source":["The dimension says 60,000 by 28 by 28. This means the dataset has 60,000 images. Each image has 28 rows and 28 columns of pixels."],"metadata":{"id":"bfq_Gs8gmFem"}},{"cell_type":"code","source":["# library\n","import matplotlib.pyplot as plt"],"metadata":{"id":"T2lZxDY7mKbs","executionInfo":{"status":"ok","timestamp":1654356288933,"user_tz":240,"elapsed":15,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":9,"outputs":[]},{"cell_type":"code","source":["# plot one sample\n","plt.imshow(training_images[0])\n","plt.title('title: this is digit ' + str(training_labels[0]))\n","plt.colorbar()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":298},"id":"HOtoxmiGmPwg","executionInfo":{"status":"ok","timestamp":1654356288934,"user_tz":240,"elapsed":15,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"3be090dc-3dfd-448f-a65a-2f46337f5632"},"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.colorbar.Colorbar at 0x7fb4a1271810>"]},"metadata":{},"execution_count":10},{"output_type":"display_data","data":{"text/plain":["<Figure size 432x288 with 2 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAATEAAAEICAYAAAA3EMMNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAavklEQVR4nO3df5RdZX3v8ffHEEITUJJG0gDBIIZCwBpw5MciIi4VkeVdgdWKUJaiFw1FUqWXtmJWb8FbuKX38uOCUHqDpIRbEFCgRC9XBNqKtBAIFPkVlRSDEIeJIUDCDyGZ+d4/9h48M+ec5+yZc2bO3pPPy7XX7LO/+8fjRr88z7Of/WxFBGZmVfW2bhfAzKwdTmJmVmlOYmZWaU5iZlZpTmJmVmlOYmZWaU5iY0TSXpJekTQpsU9Ies8YXf8aSecl4q9IevcIz3mypB90oGznSvqHfL3lfao5rvC+tv1wEusQSeskfXTwd0T8IiJ2joj+PP4vkr4wRtf+nKR7R3JMXranR3jMdRFx9MhK1/KcQ+7TSPYtck/zf1G8mie/VyR9s1Nlt3LYodsFMBsH74uItd0uhI0N18Q6QNL/AfYCvpv/2/7PJc3NawE7SDof+CBweR6/vME5pki6UNIvJPVJ+jtJv1Xg2vsDfwccnp/7pZrwdEn/V9IWSask7VNz3FtNWUnHSnoy32+9pD9tcq23anzKXCJpg6TNkh6TdGCT4/aW9MP8/HcCM2tib92nmn3vyfe9S9IVNU3PEd1T205EhJcOLMA64KM1v+cCAeyQ//4X4AvDjgngPfn6JcBKYAawC/Bd4K9r9n0JWNjk2p8D7h227RrgBeAQshr3dcANTa7dC3wwX58OHNzqOsDHgYeAXQEB+wOzmxx3H3AxMAU4EtgC/EOT+3QfcCGwI7AQ2JzYt+6eNrh2AL8EngduAeZ2+38rXjq7uCZWApIELAb+JCI2RcQW4L8DJw7uExG7RsSI+r2AWyPigYjYRpbEFjTZbyswX9LbI+LFiHi4wLm3kiXb/QBFxJqI6B2+k6S9gA8A/zUi3oiIe8gSdJ2aff8yIt7M//uuLFCWlA+RJb/9yJLZ9wZrfTYxOImVwzuBqcBDkl7Km4Tfz7e34/ma9deAnZvs9/vAscAzebPv8FYnjoh/Ai4HrgA2SFom6e0Ndt0deDEiXq3Z9kyT0+4ObIqI12q2PduqLC3KeU+eEF8CvgLsTVZrtAnCSaxzWk0HkopvBF4HDshrXLtGxDsiolnSGem10wdHPBgRi4DdgH8Ebip43GUR8X5gPrAv8GcNdusl65ubVrNtryan7AVmSJpas21OqghFytngGI3iOCspJ7HO6QNS466axiNiALgKuETSbgCS9pD08RFce09JO46gvOTX2TEf//WOiNhK1gc1UOC4D0g6VNJk4FXg142Oi4hngNXA1/NrLQT+U6Nz1ux7br7v4c32zSXvuaQDJC2QNEnSzsBFwHpgTav/flYdTmKd89fAX+TNwUZP9y4F/kDSi5IuaxD/KrAWuF/SZuAu4HcHg/kTuA82ufY/AU8Az0vaOIqyfwZYl1/3j4CTCxzzdrLE+yJZ8/AF4H822fcPgUOBTcA5wLWJ854MHJ6f7zzgRuCNJvu2uqez8uM3A0+T9Y19Mk/WNkEowpMiWnlJuhH4SUSc0+2yWDm5JmalkjdT95H0NknHAIvI+unMGvKjZiub3yEbz/XbwHPA6RHx790tkpWZm5NmVmluTppZpY1rc3JHTYmdmNZ6RzMblV/zKm/GG22Ng/v4h6fFC5taTioCwEOPvnFHRBzTzvXa1VYSyzteLwUmAd+MiAtS++/ENA7VR9q5pJklrIq72z7HC5v6eeCOZuORh5o0+6mZqbikOWRDamaRDTReFhGXSjoX+CLwq3zXpRFxe37M14BTgX7gyxFxR+oao05iyiamuwL4GFkH7IOSVkbEk6M9p5l1XwADrcc7F7UNOCsiHpa0C9mrdXfmsUsi4sLanSXNJ3tn+ACy19DukrRvJOaba6cmdgiwNvKJ9STdQPY43EnMrMKCYGvrOSqLnSubFKA3X98iaQ2wR+KQRWSzrbwB/FzSWrJcc1+zA9rp2N+DoS/nPteocJIWS1otafXWpgOvzaxMBgr+ZyQkzQUOAlblm5ZIelTScknT822F8kqtMX86GRHLIqInInomM2WsL2dmbQqC/ii2ADMHKyn5srjROfN3V28GzoyIzcCVwD5k00P1kr3XOirtNCfXM3SGgT3zbWZWcQPFJwjZGBE9qR3ySQJuBq6LiFsAIqKvJn4V8L3854jzSjs1sQeBefl0wjuSdca1O4GdmXVZAP1EoaWVfMLPq4E1EXFxzfbZNbsdDzyer68ETlQ2XfvewDzggdQ1Rl0Ti4htkpYAd5ANsVgeEU+M9nxmVh4jqIm1cgTZLCmPSXok37YUOEnSArKcuQ44DSAinpB0E9kDwm3AGaknk9DmOLF8XMft7ZzDzMolgK0deh0xn2K80eDbpnkjIs4Hzi96Db8AbmZDRMGmYlk4iZnZUAH91clhTmJmNlQ2Yr86nMTMbBjRX6FvqTiJmdkQWce+k5iZVVQ2TsxJzMwqbMA1MTOrKtfEzKzSAtFfoZnrncTMrI6bk2ZWWYF4MyZ1uxiFOYmZ2RDZYFc3J82swtyxb2aVFSH6wzUxM6uwAdfEzKyqso796qSG6pTUzMaFO/bNrPL6PU7MzKrKI/bNrPIG/HTSzKoqewHcSczMKioQW/3akZlVVQQe7GpmVSYPdjWz6gpcEzOzinPHvplVViBPimhm1ZV9sq06qaE6JTWzceKP51qJaIf0P+JJ75w5ptf/6Z/ObRrrnzqQPPZd+2xIxqd+Kf1/tOcv3rFp7OGeG5PHbux/NRk/9NtnJePv+S/3J+NlFmxHI/YlrQO2AP3Atojo6UShzKy7trea2IcjYmMHzmNmJRCh7acmZmYTT9axv/28dhTADyQF8L8jYtnwHSQtBhYD7MTUNi9nZmOvWnPst1vShRFxMPAJ4AxJRw7fISKWRURPRPRMZkqblzOzsZZ17KvQ0oqkOZL+WdKTkp6Q9JV8+wxJd0p6Kv87Pd8uSZdJWivpUUkHt7pGW0ksItbnfzcAtwKHtHM+MyuHft5WaClgG3BWRMwHDiOr7MwHzgbujoh5wN35b8gqRPPyZTFwZasLjDqJSZomaZfBdeBo4PHRns/MymFwxH4namIR0RsRD+frW4A1wB7AImBFvtsK4Lh8fRFwbWTuB3aVNDt1jXb6xGYBt0oaPM/1EfH9Ns43YU3af14yHlMmJ+O//NCuyfjrhzUf0zTjHenxTj96X3q8VDf9v9d2Scb/5vJjkvFV772+aeznW19PHntB38eS8d1/FMl41Y3gQyEzJa2u+b2sUd84gKS5wEHAKmBWRPTmoefJ8glkCe7ZmsOey7f10sSok1hEPA28b7THm1k5RcDWgcJJbGOR8aGSdgZuBs6MiM155Se/XkT+cHBUPMTCzIbImpOdezopaTJZArsuIm7JN/dJmh0RvXlzcfD1jPXAnJrD98y3NVWd56hmNm768/cnWy2tKKtyXQ2siYiLa0IrgVPy9VOA22q2fzZ/SnkY8HJNs7Mh18TMbIjBIRYdcgTwGeAxSY/k25YCFwA3SToVeAY4IY/dDhwLrAVeAz7f6gJOYmY2TOeakxFxLzStsn2kwf4BnDGSaziJmVkdz7G/nek/Kj2o+OJrrkjG953cfMqYiWxr9Cfjf/mNzyXjO7yafqB1+LeXNI3tsn5b8tgpG9NDMKauXpWMV1n2dHL7eXfSzCYYT09tZpXn5qSZVVaHn06OOScxM6vjSRHNrLIixDYnMTOrMjcnzayy3Ce2HZry018m4w/9ek4yvu/kvk4Wp6PO6j0sGX/6lfQn367Z5ztNYy8PpMd5zbrs35LxsTSxJ9ppzUnMzCrL48TMrPI8TszMKisCthWfFLHrnMTMrI6bk2ZWWe4TM7PKCycxM6syd+xvZ7b1Pp+Mf+NvPpWMn39M+rNqkx7dORn/8Ze+kYynnLfx95LxtR+dmoz3v5Sc/pw/PPxLTWPrvpw8lL35cXoHGxMR7hMzs0oT/X46aWZV5j4xM6ssvztpZtUWWb9YVTiJmVkdP500s8oKd+ybWdW5OWlDzPj7+5Lxd373t5Px/hc2JeMHHPifm8aeOHJ58tiVyz6UjO/2Untzeum+5mO99k7fFuuiKj2dbFlnlLRc0gZJj9dsmyHpTklP5X+nj20xzWy8RGRJrMhSBkUavtcAxwzbdjZwd0TMA+7Of5vZBDEQKrSUQcskFhH3AMPbM4uAFfn6CuC4DpfLzLooothSBqPtE5sVEYMvzT0PzGq2o6TFwGKAnUi/h2dm3ReIgQo9nWy7pBERJL6rEBHLIqInInomM6Xdy5nZOIiCSxmMNon1SZoNkP/d0LkimVlXTcCO/UZWAqfk66cAt3WmOGZWChWqirXsE5P0LeAoYKak54BzgAuAmySdCjwDnDCWhZzo+je+0NbxWzfvOOpjDzj5yWT8V1dOSp9goH/U17byKkstq4iWSSwiTmoS+kiHy2JmJRDAwEBnkpik5cAngQ0RcWC+7Vzgi8Cv8t2WRsTteexrwKlAP/DliLij1TWq8wjCzMZHAKFiS2vXUD/OFOCSiFiQL4MJbD5wInBAfszfSmrRFHASM7MGOjVOrMk402YWATdExBsR8XNgLXBIq4OcxMysXvGO/ZmSVtcsiwteYYmkR/PXGgdfW9wDeLZmn+fybUl+AdzMhhnR8ImNEdEzwgtcCfwVWRr8K+AioPksBi24JmZm9cZwiEVE9EVEf0QMAFfxmybjemBOza575tuSXBObAPb/6s+axj7/3vRD5L9/193J+Ic+dUYyvsuN9yfjVkEB0aGnk41Iml3z2uLxwOAMOSuB6yVdDOwOzAMeaHU+JzEza6BjQywajTM9StICsrrcOuA0gIh4QtJNwJPANuCMiGg5ENFJzMzqdWg0fpNxplcn9j8fOH8k13ASM7N6JXmlqAgnMTMbanCwa0U4iZlZnbJMeFiEk5iZ1RvDp5Od5iRmZnXkmpiNp/6XXm4ae+H0/ZPH/mLl68n42eddm4x/7YTjk/H493c0jc05v8U326rUpplISjRXWBFOYmY2TOEZKkrBSczM6rkmZmaVNtDtAhTnJGZmQ3mcmJlVnZ9Omlm1VSiJeT4xM6s018QmuIEfr0nGT/z6nyXj151zYTL+yGHpcWQc1jx0wLQlyUPnXdWbjG97el362jZqbk6aWXUFfu3IzCrONTEzqzI3J82s2pzEzKzSnMTMrKoUbk6aWdX56aRVxYzl6Tm9lvw0/d3Jt1/wXDL+rXff0TT2xGcvTx6735wvJOO/+/X0WO3+p55Oxq25KtXEWo7Yl7Rc0gZJj9dsO1fSekmP5MuxY1tMMxtXY/gF8E4r8trRNcAxDbZfEhEL8uX2zhbLzLomftMv1mopg5ZJLCLuATaNQ1nMrCwmWE2smSWSHs2bm9Ob7SRpsaTVklZv5Y02Lmdm40UDxZYyGG0SuxLYB1gA9AIXNdsxIpZFRE9E9ExmyigvZ2bW2KiSWET0RUR/RAwAVwGHdLZYZtZVE705KWl2zc/jgceb7WtmFVOxjv2W48QkfQs4Cpgp6TngHOAoSQvIcvE64LQxLKN1kf71kWT8tT/YLRn/wKf/uGls1VcvTR77kw9/Mxk/ee7RyfjLC5NhSylJgiqiZRKLiJMabL56DMpiZmUxkZKYmW1fRHmePBbhJGZmQ5Wov6sIfyjEzOp16Olkk9cWZ0i6U9JT+d/p+XZJukzS2nwM6sFFiuokZmb1OjfE4hrqX1s8G7g7IuYBd+e/AT4BzMuXxWTjUVtyEjOzOp0aYtHktcVFwIp8fQVwXM32ayNzP7DrsOFcDblPzNrS37chGZ91WfP4r/98W/LYqdoxGb9q7veS8U8ef2bzc9+6Knnsdm9s+8RmRcTg9/ieB2bl63sAz9bs91y+LfntPicxMxsqRvR0cqak1TW/l0XEssKXigipvccITmJmVq94WtkYET0jPHufpNkR0Zs3Fwer6+uBOTX77ZlvS3KfmJnVGePXjlYCp+TrpwC31Wz/bP6U8jDg5ZpmZ1OuiZlZvQ71iTV5bfEC4CZJpwLPACfku98OHAusBV4DPl/kGk5iZjZUB2eoaPLaIsBHGuwbQPqjDg04iZnZEKJaI/adxMysjpOYTRgDCxck4//xqZ2S8QMXrGsaazUOrJVvbDooGZ962+pk3BKcxMys0pzEzKyyKjaLhZOYmdVzEjOzKvOkiGZWaW5Omll1lehzbEU4iZlZPScxKwv1HJiM/+zLLebsOmJFMn7kTm+OuExFvRFbk/H7N+2dPsFAy3eHrQGP2DezytNAdbKYk5iZDeU+MTOrOjcnzazanMTMrMpcEzOzanMSM7PKGtnXjrquZRKTNAe4luzbcEH2SaZLJc0AbgTmAuuAEyLixbEr6vZrh73flYz/x+d3bxo799M3JI/9/Z03jqpMnbC0L/2RnB9eelgyPn3FfZ0sjuWqNk6syNeOtgFnRcR84DDgDEnzaf4pcjOruohiSwm0TGIR0RsRD+frW4A1ZF/lbfYpcjOruDH+ZFtHjahPTNJc4CBgFc0/RW5mVTZRB7tK2hm4GTgzIjZLeiuW+hS5pMXAYoCdmNpeac1sXFSpY7/QF8AlTSZLYNdFxC355r78E+QM+xT5EBGxLCJ6IqJnMlM6UWYzG2MaKLaUQcskpqzKdTWwJiIurgk1+xS5mVVZUKmO/SLNySOAzwCPSXok37aU5p8it2F2mLtXMv7y+2cn45/+b99Pxv9o11uS8bF0Vm96GMR9f9t8GMWMax5IHjt9wEMouqUsnfZFtExiEXEv2dCRRuo+RW5mE8BESmJmtn2p2mBXJzEzGyrCkyKaWcVVJ4c5iZlZPTcnzay6AnBz0swqrTo5zEmsqB1m/07T2Kbl05LHnr73D5Pxk3bpG1WZOmHJ+oXJ+MNXLkjGZ37n8WR8xhaP9aoiNyfNrNI6+XRS0jpgC9APbIuInk7OR1jo3Ukz247ECJbiPhwRCyJi8BWOjs1H6CRmZkNkg12j0NKGjs1H6CRmZvUGCi4wU9LqmmVxg7MF8ANJD9XEOzYfofvEzKzOCGpZG2uaiM0sjIj1knYD7pT0k9pgaj7CIlwTM7OhOtwnFhHr878bgFuBQyg4H2ERTmJmNkz27mSRpRVJ0yTtMrgOHA08TgfnI9xumpNvfjxd433zTzYl40vfc3vT2NG/9eqoytQpff2vN40dufKs5LH7/cVPkvEZL6XHeZVkck/rtM5NeDgLuDWfzn4H4PqI+L6kB+nQfITbTRIzs4I6+PHciHgaeF+D7S/QofkIncTMrF5Jpp4uwknMzOpVJ4c5iZlZPQ1Up7fTSczMhgoq9cTGSczMhhBtv1I0rpzEzKyek1j5rDsuPa73Z+/99phd+4qX9knGL/3h0cm4+pt9MS+z33k/bxqb17cqeWx/MmrbLScxM6ss94mZWdX56aSZVVi4OWlmFRY4iZlZxVWnNekkZmb1PE7MzKptIiUxSXOAa8nmBQpgWURcKulc4IvAr/Jdl0ZE80m3umzf0x9Ixj95+vvHqST19iVdtlY81ss6KgL6q9OeLFIT2wacFREP5zM0PiTpzjx2SURcOHbFM7OumEg1sfyLJL35+hZJa4A9xrpgZtZFFUpiI5pjX9Jc4CBg8F2WJZIelbRc0vQmxywe/JzTVt5oq7BmNg4CGIhiSwkUTmKSdgZuBs6MiM3AlcA+wAKymtpFjY6LiGUR0RMRPZOZ0oEim9nYCoiBYksJFHo6KWkyWQK7LiJuAYiIvpr4VcD3xqSEZja+gkp17LesiSn7TMnVwJqIuLhm++ya3Y4n+wyTmU0EEcWWEihSEzsC+AzwmKRH8m1LgZMkLSDL2+uA08akhGY2/kqSoIoo8nTyXqDRhFalHRNmZu0oTy2rCI/YN7OhAvBUPGZWaa6JmVl1TbzXjsxsexIQJRkDVoSTmJnVK8lo/CKcxMysnvvEzKyyIvx00swqzjUxM6uuIPqrM9Wmk5iZDTU4FU9FOImZWb0KDbEY0aSIZjbxBRADUWgpQtIxkn4qaa2ksztdXicxMxsqOjcpoqRJwBXAJ4D5ZLPfzO9kcd2cNLM6HezYPwRYGxFPA0i6AVgEPNmpC4xrEtvCixvviu88U7NpJrBxPMswAmUtW1nLBS7baHWybO9q9wRbePGOu+I7MwvuvpOk1TW/l0XEsprfewDP1vx+Dji03TLWGtckFhHvrP0taXVE9IxnGYoqa9nKWi5w2UarbGWLiGO6XYaRcJ+YmY2l9cCcmt975ts6xknMzMbSg8A8SXtL2hE4EVjZyQt0u2N/WetduqasZStrucBlG60yl60tEbFN0hLgDmASsDwinujkNRQVekfKzGw4NyfNrNKcxMys0rqSxMb6NYR2SFon6TFJjwwb/9KNsiyXtEHS4zXbZki6U9JT+d/pJSrbuZLW5/fuEUnHdqlscyT9s6QnJT0h6Sv59q7eu0S5SnHfqmrc+8Ty1xB+BnyMbODbg8BJEdGxEbztkLQO6ImIrg+MlHQk8ApwbUQcmG/7H8CmiLgg/xfA9Ij4aknKdi7wSkRcON7lGVa22cDsiHhY0i7AQ8BxwOfo4r1LlOsESnDfqqobNbG3XkOIiDeBwdcQbJiIuAfYNGzzImBFvr6C7P8E465J2UohInoj4uF8fQuwhmzkeFfvXaJc1oZuJLFGryGU6R9kAD+Q9JCkxd0uTAOzIqI3X38emNXNwjSwRNKjeXOzK03dWpLmAgcBqyjRvRtWLijZfasSd+zXWxgRB5O9dX9G3mwqpcj6Aso0RuZKYB9gAdALXNTNwkjaGbgZODMiNtfGunnvGpSrVPetarqRxMb8NYR2RMT6/O8G4Fay5m+Z9OV9K4N9LBu6XJ63RERfRPRH9tHCq+jivZM0mSxRXBcRt+Sbu37vGpWrTPetirqRxMb8NYTRkjQt73BF0jTgaODx9FHjbiVwSr5+CnBbF8syxGCCyB1Pl+6dJAFXA2si4uKaUFfvXbNyleW+VVVXRuznj5D/F795DeH8cS9EA5LeTVb7guyVrOu7WTZJ3wKOIpuqpQ84B/hH4CZgL+AZ4ISIGPcO9iZlO4qsSRTAOuC0mj6o8SzbQuBHwGPA4Mx9S8n6n7p27xLlOokS3Leq8mtHZlZp7tg3s0pzEjOzSnMSM7NKcxIzs0pzEjOzSnMSM7NKcxIzs0r7/8ohR50UuBLGAAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","source":["# library\n","import numpy as np"],"metadata":{"id":"bhTfDKGmrAAv","executionInfo":{"status":"ok","timestamp":1654356288934,"user_tz":240,"elapsed":13,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":11,"outputs":[]},{"cell_type":"code","source":["# unique digits\n","np.unique(training_labels)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YlMsNHasq4fd","executionInfo":{"status":"ok","timestamp":1654356288934,"user_tz":240,"elapsed":13,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"66f72149-788d-41a6-a73e-f89ab8456226"},"execution_count":12,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8)"]},"metadata":{},"execution_count":12}]},{"cell_type":"code","source":["# display shape\n","training_labels.shape"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GPupey3-uwYz","executionInfo":{"status":"ok","timestamp":1654356288935,"user_tz":240,"elapsed":13,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"23eca1be-f21a-41f5-cb9d-cdf30590dc93"},"execution_count":13,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(60000,)"]},"metadata":{},"execution_count":13}]},{"cell_type":"markdown","source":["### One hot encode\n","\n","We need to use *to_categorical* function to do one-hot encode. The operation of one-hot encode will check each unique level of the data and create a new binary vector for that level.\n","\n","For example, the vector below has 3 levels: 1, 3, 4. Hence, for each level (for each of the value 1, 3, 4), there will be a new vector created that only takes values 1 or 0.\n","\n","$$\n","\\begin{bmatrix}\n","1 \\\\\n","3 \\\\\n","1 \\\\\n","4 \\\\\n","\\end{bmatrix}\n","\\rightarrow_\\text{one hot}\n","\\begin{bmatrix}\n","1 & 0 & 0\\\\\n","0 & 1 & 0\\\\\n","1 & 0 & 0\\\\\n","0 & 0 & 1\\\\\n","\\end{bmatrix}\n","$$\n","\n","Source is [here](https://www.tensorflow.org/api_docs/python/tf/keras/utils/to_categorical)."],"metadata":{"id":"sESMsDDYu-p0"}},{"cell_type":"code","source":["# one hot encode\n","training_labels_one_hot_encoded = tf.keras.utils.to_categorical(training_labels, num_classes=10)\n","test_labels_one_hot_encoded = tf.keras.utils.to_categorical(test_labels, num_classes=10)"],"metadata":{"id":"_FwaQ3IKu98U","executionInfo":{"status":"ok","timestamp":1654356288935,"user_tz":240,"elapsed":12,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":14,"outputs":[]},{"cell_type":"code","source":["# display shape\n","print(training_labels_one_hot_encoded.shape)\n","print(test_labels_one_hot_encoded.shape)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"aTpXkpmJ9eEI","executionInfo":{"status":"ok","timestamp":1654356288935,"user_tz":240,"elapsed":11,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"c6e457f1-db35-40bd-c445-fa2b0888c395"},"execution_count":15,"outputs":[{"output_type":"stream","name":"stdout","text":["(60000, 10)\n","(10000, 10)\n"]}]},{"cell_type":"markdown","source":["This means there are 10 classes."],"metadata":{"id":"5NqgiOyFrCW8"}},{"cell_type":"markdown","source":["## Install\n","\n","Here we install a python installable github package from [this site](https://github.com/yiqiao-yin/DeepLearningSeries)."],"metadata":{"id":"u-hW3faCakU1"}},{"cell_type":"code","source":["# from python notebook\n","!pip install git+https://github.com/yiqiao-yin/DeepLearningSeries.git"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"fDIQusCfUtmv","executionInfo":{"status":"ok","timestamp":1654356466460,"user_tz":240,"elapsed":4439,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"7cfb566e-4822-41b9-908c-d6975a95abab"},"execution_count":17,"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting git+https://github.com/yiqiao-yin/DeepLearningSeries.git\n","  Cloning https://github.com/yiqiao-yin/DeepLearningSeries.git to /tmp/pip-req-build-sw_bd640\n","  Running command git clone -q https://github.com/yiqiao-yin/DeepLearningSeries.git /tmp/pip-req-build-sw_bd640\n","Building wheels for collected packages: DeepLearningSeries\n","  Building wheel for DeepLearningSeries (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for DeepLearningSeries: filename=DeepLearningSeries-0.0.0-py3-none-any.whl size=3347 sha256=1c1ccacb458a677919ef290aab3bed8349dcd7f852e205ae77e14bd9b1d546fe\n","  Stored in directory: /tmp/pip-ephem-wheel-cache-imwkt677/wheels/aa/1d/c5/d3b552b251276a8ebb974c22863c925470b39fdcee61503634\n","Successfully built DeepLearningSeries\n","Installing collected packages: DeepLearningSeries\n","Successfully installed DeepLearningSeries-0.0.0\n"]}]},{"cell_type":"code","source":["# import \n","from DeepLearning.modules import DL"],"metadata":{"id":"5gwyyH3_UuYU","executionInfo":{"status":"ok","timestamp":1654356573191,"user_tz":240,"elapsed":199,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":18,"outputs":[]},{"cell_type":"code","source":["# check out the function\n","DL.NN3"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"WyXjCafnVfmo","executionInfo":{"status":"ok","timestamp":1654356595587,"user_tz":240,"elapsed":219,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"66b881c0-cfec-405a-dbe8-9e8617d98ffa"},"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<function DeepLearning.modules.DL.NN3>"]},"metadata":{},"execution_count":19}]},{"cell_type":"code","source":["# save the output of the function into a new object called test_run\n","test_run = DL.NN3(x_train=training_images, y_train=training_labels_one_hot_encoded, \n","                  x_test=test_images, y_test=test_labels_one_hot_encoded, validation_split=0.2, epochs=10)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ubvoT30hVlAb","executionInfo":{"status":"ok","timestamp":1654356784964,"user_tz":240,"elapsed":50287,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"bdd252d6-824d-4cc0-906d-3e8873b16376"},"execution_count":20,"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/10\n","1500/1500 [==============================] - 7s 3ms/step - loss: 0.2072 - accuracy: 0.9360 - val_loss: 0.1122 - val_accuracy: 0.9672\n","Epoch 2/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0926 - accuracy: 0.9719 - val_loss: 0.1019 - val_accuracy: 0.9709\n","Epoch 3/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0654 - accuracy: 0.9795 - val_loss: 0.1004 - val_accuracy: 0.9708\n","Epoch 4/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0476 - accuracy: 0.9855 - val_loss: 0.0915 - val_accuracy: 0.9738\n","Epoch 5/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0419 - accuracy: 0.9867 - val_loss: 0.0952 - val_accuracy: 0.9751\n","Epoch 6/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0335 - accuracy: 0.9897 - val_loss: 0.1135 - val_accuracy: 0.9723\n","Epoch 7/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0305 - accuracy: 0.9909 - val_loss: 0.1088 - val_accuracy: 0.9750\n","Epoch 8/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0251 - accuracy: 0.9924 - val_loss: 0.1031 - val_accuracy: 0.9782\n","Epoch 9/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0230 - accuracy: 0.9933 - val_loss: 0.0979 - val_accuracy: 0.9755\n","Epoch 10/10\n","1500/1500 [==============================] - 4s 3ms/step - loss: 0.0216 - accuracy: 0.9933 - val_loss: 0.1007 - val_accuracy: 0.9778\n","313/313 [==============================] - 1s 2ms/step - loss: 0.1024 - accuracy: 0.9782\n","Test Result: Loss is 0.10238423943519592, and accuracy is 0.9782000184059143\n"]}]},{"cell_type":"code","source":["test_run.keys()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"u4yAKtOGWHEv","executionInfo":{"status":"ok","timestamp":1654356857709,"user_tz":240,"elapsed":2,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"d9cfa820-917d-4a9e-8137-f37f8037dc85"},"execution_count":21,"outputs":[{"output_type":"execute_result","data":{"text/plain":["dict_keys(['History', 'Model'])"]},"metadata":{},"execution_count":21}]},{"cell_type":"code","source":["test_run['Model']"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"FzXgr6kmWlGY","executionInfo":{"status":"ok","timestamp":1654356882319,"user_tz":240,"elapsed":214,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"67745268-fb97-4b5a-bb61-36ef3743ba9d"},"execution_count":22,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<keras.engine.sequential.Sequential at 0x7fb53f7a2090>"]},"metadata":{},"execution_count":22}]},{"cell_type":"code","source":["test_run['History'].history.keys()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"XKZ7OdBrWwy8","executionInfo":{"status":"ok","timestamp":1654356938115,"user_tz":240,"elapsed":2,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"693dccf9-2902-4d4b-9e7a-6418af5e3876"},"execution_count":25,"outputs":[{"output_type":"execute_result","data":{"text/plain":["dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])"]},"metadata":{},"execution_count":25}]}]}

docs/notebooks/ex04a - more in cnn (famous cnn).ipynb

@@ -0,0 +1,396 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "cX6vB-OppNfv"
+      },
+      "source": [
+        "## LeNet\n",
+        "\n",
+        "LeNet is a classic convolutional neural network (CNN) architecture designed by Yann LeCun, which was primarily used for handwritten digit recognition. Here's a simple implementation of the LeNet model using TensorFlow 2.x and Keras:\n",
+        "\n",
+        "```py\n",
+        "import tensorflow as tf\n",
+        "from tensorflow.keras import layers, models\n",
+        "\n",
+        "def build_lenet_model(input_shape=(32, 32, 1), num_classes=10):\n",
+        "    model = models.Sequential()\n",
+        "\n",
+        "    # Layer 1: Convolutional Layer + Activation + Pooling\n",
+        "    model.add(layers.Conv2D(filters=6, kernel_size=(5, 5), padding='same', activation='relu', input_shape=input_shape))\n",
+        "    model.add(layers.AveragePooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n",
+        "\n",
+        "    # Layer 2: Convolutional Layer + Activation + Pooling\n",
+        "    model.add(layers.Conv2D(filters=16, kernel_size=(5, 5), padding='valid', activation='relu'))\n",
+        "    model.add(layers.AveragePooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'))\n",
+        "\n",
+        "    # Layer 3: Convolutional Layer + Activation\n",
+        "    model.add(layers.Conv2D(filters=120, kernel_size=(5, 5), padding='valid', activation='relu'))\n",
+        "\n",
+        "    # Flatten the output\n",
+        "    model.add(layers.Flatten())\n",
+        "\n",
+        "    # Layer 4: Fully Connected (Dense) Layer + Activation\n",
+        "    model.add(layers.Dense(units=84, activation='relu'))\n",
+        "\n",
+        "    # Layer 5: Fully Connected (Dense) Layer + Activation (Output Layer)\n",
+        "    model.add(layers.Dense(units=num_classes, activation='softmax'))\n",
+        "\n",
+        "    return model\n",
+        "\n",
+        "# Build the LeNet model\n",
+        "lenet_model = build_lenet_model(input_shape=(32, 32, 1), num_classes=10)\n",
+        "lenet_model.summary()\n",
+        "```\n",
+        "\n",
+        "LeNet is a convolutional neural network, and its original paper is called \"Gradient-Based Learning Applied to Document Recognition\" by Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. It was published in 1998 and can be found here: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-kX1-wPypS8R"
+      },
+      "source": [
+        "## VGG16\n",
+        "\n",
+        "VGG16 is a deep convolutional neural network (CNN) architecture developed by the Visual Geometry Group at the University of Oxford. Here's a simple implementation of the VGG16 model using TensorFlow 2.x and Keras Sequential API:\n",
+        "\n",
+        "```py\n",
+        "import tensorflow as tf\n",
+        "from tensorflow.keras import layers, models\n",
+        "\n",
+        "def build_vgg16_model(input_shape=(224, 224, 3), num_classes=1000):\n",
+        "    model = models.Sequential()\n",
+        "\n",
+        "    # Block 1: 2x Convolutional Layers + Max Pooling\n",
+        "    model.add(layers.Conv2D(64, kernel_size=(3, 3), padding='same', activation='relu', input_shape=input_shape))\n",
+        "    model.add(layers.Conv2D(64, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n",
+        "\n",
+        "    # Block 2: 2x Convolutional Layers + Max Pooling\n",
+        "    model.add(layers.Conv2D(128, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(128, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n",
+        "\n",
+        "    # Block 3: 3x Convolutional Layers + Max Pooling\n",
+        "    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(256, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n",
+        "\n",
+        "    # Block 4: 3x Convolutional Layers + Max Pooling\n",
+        "    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n",
+        "\n",
+        "    # Block 5: 3x Convolutional Layers + Max Pooling\n",
+        "    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.Conv2D(512, kernel_size=(3, 3), padding='same', activation='relu'))\n",
+        "    model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))\n",
+        "\n",
+        "    # Flatten the output\n",
+        "    model.add(layers.Flatten())\n",
+        "\n",
+        "    # Fully Connected Layers\n",
+        "    model.add(layers.Dense(4096, activation='relu'))\n",
+        "    model.add(layers.Dropout(0.5))\n",
+        "    model.add(layers.Dense(4096, activation='relu'))\n",
+        "    model.add(layers.Dropout(0.5))\n",
+        "\n",
+        "    # Output Layer\n",
+        "    model.add(layers.Dense(num_classes, activation='softmax'))\n",
+        "\n",
+        "    return model\n",
+        "\n",
+        "# Build the VGG16 model\n",
+        "vgg16_model = build_vgg16_model(input_shape=(224, 224, 3), num_classes=1000)\n",
+        "vgg16_model.summary\n",
+        "```\n",
+        "\n",
+        "VGG16 is introduced in a paper titled \"Very Deep Convolutional Networks for Large-Scale Image Recognition\" by K. Simonyan and A. Zisserman.\n",
+        "\n",
+        "Here is the link to the VGG16 paper: \n",
+        "\n",
+        "[Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/abs/1409.1556)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "G6md-gIspYG-"
+      },
+      "source": [
+        "## ResNet\n",
+        "\n",
+        "ResNet (Residual Network) is a deep convolutional neural network (CNN) architecture that introduces residual connections to improve learning and avoid the vanishing gradient problem in deep networks. Here's a simple implementation of the ResNet-50 model using TensorFlow 2.x and Keras functional API:\n",
+        "\n",
+        "```py\n",
+        "import tensorflow as tf\n",
+        "from tensorflow.keras import layers, models, Input\n",
+        "\n",
+        "def identity_block(input_tensor, kernel_size, filters):\n",
+        "    filters1, filters2, filters3 = filters\n",
+        "\n",
+        "    x = layers.Conv2D(filters1, (1, 1))(input_tensor)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "\n",
+        "    x = layers.Conv2D(filters2, kernel_size, padding='same')(x)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "\n",
+        "    x = layers.Conv2D(filters3, (1, 1))(x)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "\n",
+        "    x = layers.add([x, input_tensor])\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "    return x\n",
+        "\n",
+        "def conv_block(input_tensor, kernel_size, filters, strides=(2, 2)):\n",
+        "    filters1, filters2, filters3 = filters\n",
+        "\n",
+        "    x = layers.Conv2D(filters1, (1, 1), strides=strides)(input_tensor)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "\n",
+        "    x = layers.Conv2D(filters2, kernel_size, padding='same')(x)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "\n",
+        "    x = layers.Conv2D(filters3, (1, 1))(x)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "\n",
+        "    shortcut = layers.Conv2D(filters3, (1, 1), strides=strides)(input_tensor)\n",
+        "    shortcut = layers.BatchNormalization()(shortcut)\n",
+        "\n",
+        "    x = layers.add([x, shortcut])\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "    return x\n",
+        "\n",
+        "def build_resnet50(input_shape=(224, 224, 3), num_classes=1000):\n",
+        "    input_tensor = Input(shape=input_shape)\n",
+        "\n",
+        "    # Initial convolution layer\n",
+        "    x = layers.ZeroPadding2D(padding=(3, 3))(input_tensor)\n",
+        "    x = layers.Conv2D(64, (7, 7), strides=(2, 2))(x)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "    x = layers.ZeroPadding2D(padding=(1, 1))(x)\n",
+        "    x = layers.MaxPooling2D((3, 3), strides=(2, 2))(x)\n",
+        "\n",
+        "    # Residual blocks\n",
+        "    x = conv_block(x, 3, [64, 64, 256], strides=(1, 1))\n",
+        "    x = identity_block(x, 3, [64, 64, 256])\n",
+        "    x = identity_block(x, 3, [64, 64, 256])\n",
+        "\n",
+        "    x = conv_block(x, 3, [128, 128, 512])\n",
+        "    x = identity_block(x, 3, [128, 128, 512])\n",
+        "    x = identity_block(x, 3, [128, 128, 512])\n",
+        "    x = identity_block(x, 3, [128, 128, 512])\n",
+        "\n",
+        "    x = conv_block(x, 3, [256, 256, 1024])\n",
+        "    x = identity_block(x, 3, [256, 256, 1024])\n",
+        "    x = identity_block(x, 3, [256, 256, 1024])\n",
+        "    x = identity_block(x, 3, [256, 256, 1024])\n",
+        "    x = identity_block(x, 3, [256, 256, 1024])\n",
+        "\n",
+        "    x = conv_block(x, 3, [512, 512, 2048])\n",
+        "    x = identity_block(x, 3, [512, 512, 2048])\n",
+        "    x = identity_block(x, 3, [512, 512, 2048])\n",
+        "\n",
+        "    # Final part\n",
+        "    x = layers.GlobalAveragePooling2D()(x)\n",
+        "    x = layers.Dense(num_classes, activation='softmax')(x)\n",
+        "\n",
+        "    # Create model\n",
+        "    model = Model(input_tensor, x)\n",
+        "\n",
+        "    return model\n",
+        "```\n",
+        "\n",
+        "ResNet was introduced in the paper \"Deep Residual Learning for Image Recognition\" by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Here's the link to the ResNet paper:\n",
+        "\n",
+        "https://arxiv.org/pdf/1512.03385.pdf"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "kIMwTxKhpcCs"
+      },
+      "source": [
+        "## DenseNet\n",
+        "\n",
+        "DenseNet (Densely Connected Convolutional Networks) is a deep convolutional neural network (CNN) architecture that introduces dense connections between layers to improve learning and parameter efficiency. Here's a simple implementation of the DenseNet-121 model using TensorFlow 2.x and Keras functional API:\n",
+        "\n",
+        "```py\n",
+        "import tensorflow as tf\n",
+        "from tensorflow.keras import layers, models, Input\n",
+        "\n",
+        "def dense_block(x, num_layers, growth_rate):\n",
+        "    for _ in range(num_layers):\n",
+        "        output = layers.BatchNormalization()(x)\n",
+        "        output = layers.Activation('relu')(output)\n",
+        "        output = layers.Conv2D(4 * growth_rate, (1, 1), padding='same', kernel_initializer='he_normal')(output)\n",
+        "        output = layers.BatchNormalization()(output)\n",
+        "        output = layers.Activation('relu')(output)\n",
+        "        output = layers.Conv2D(growth_rate, (3, 3), padding='same', kernel_initializer='he_normal')(output)\n",
+        "        x = layers.Concatenate()([x, output])\n",
+        "    return x\n",
+        "\n",
+        "def transition_layer(x, compression_factor):\n",
+        "    num_filters = int(x.shape[-1] * compression_factor)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "    x = layers.Conv2D(num_filters, (1, 1), padding='same', kernel_initializer='he_normal')(x)\n",
+        "    x = layers.AveragePooling2D((2, 2), strides=(2, 2))(x)\n",
+        "    return x\n",
+        "\n",
+        "def build_densenet121(input_shape=(224, 224, 3), num_classes=1000, growth_rate=32, compression_factor=0.5):\n",
+        "    input_tensor = Input(shape=input_shape)\n",
+        "\n",
+        "    # Initial convolution layer\n",
+        "    x = layers.Conv2D(2 * growth_rate, (7, 7), strides=(2, 2), padding='same', kernel_initializer='he_normal')(input_tensor)\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "    x = layers.Activation('relu')(x)\n",
+        "    x = layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)\n",
+        "\n",
+        "    # Dense blocks and transition layers\n",
+        "    x = dense_block(x, 6, growth_rate)\n",
+        "    x = transition_layer(x, compression_factor)\n",
+        "\n",
+        "    x = dense_block(x, 12, growth_rate)\n",
+        "    x = transition_layer(x, compression_factor)\n",
+        "\n",
+        "    x = dense_block(x, 24, growth_rate)\n",
+        "    x = transition_layer(x, compression_factor)\n",
+        "\n",
+        "    x = dense_block(x, 16, growth_rate)\n",
+        "\n",
+        "    # Global average pooling and output layer\n",
+        "    x = layers.GlobalAveragePooling2D()(x)\n",
+        "    x = layers.Dense(num_classes, activation='softmax')(x)\n",
+        "\n",
+        "    model = models.Model(input_tensor, x, name='densenet121')\n",
+        "    return model\n",
+        "\n",
+        "# Build the DenseNet-121 model\n",
+        "densenet121_model = build_densenet121(input_shape=(224, 224, 3), num_classes=1000)\n",
+        "densenet121_model.summary()\n",
+        "```\n",
+        "\n",
+        "DenseNet was introduced in the paper \"Densely Connected Convolutional Networks\" by Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Here is the link to the original paper for Densenet: \n",
+        "\n",
+        "https://arxiv.org/abs/1608.06993"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9omz0AL9qe3p"
+      },
+      "source": [
+        "## Inception\n",
+        "\n",
+        "Inception is a deep convolutional neural network (CNN) architecture that was introduced in the GoogLeNet model. It uses inception modules to efficiently learn different features at multiple scales. Here's a simple implementation of the Inception V1 (GoogLeNet) model using TensorFlow 2.x and Keras functional API:\n",
+        "\n",
+        "```py\n",
+        "import tensorflow as tf\n",
+        "from tensorflow.keras import layers, models, Input\n",
+        "\n",
+        "def inception_module(x, filters_1x1, filters_3x3_reduce, filters_3x3, filters_5x5_reduce, filters_5x5, filters_pool_proj):\n",
+        "    conv_1x1 = layers.Conv2D(filters_1x1, (1, 1), padding='same', activation='relu', kernel_initializer='he_normal')(x)\n",
+        "    \n",
+        "    conv_3x3_reduce = layers.Conv2D(filters_3x3_reduce, (1, 1), padding='same', activation='relu', kernel_initializer='he_normal')(x)\n",
+        "    conv_3x3 = layers.Conv2D(filters_3x3, (3, 3), padding='same', activation='relu', kernel_initializer='he_normal')(conv_3x3_reduce)\n",
+        "    \n",
+        "    conv_5x5_reduce = layers.Conv2D(filters_5x5_reduce, (1, 1), padding='same', activation='relu', kernel_initializer='he_normal')(x)\n",
+        "    conv_5x5 = layers.Conv2D(filters_5x5, (5, 5), padding='same', activation='relu', kernel_initializer='he_normal')(conv_5x5_reduce)\n",
+        "    \n",
+        "    max_pool = layers.MaxPooling2D((3, 3), strides=(1, 1), padding='same')(x)\n",
+        "    pool_proj = layers.Conv2D(filters_pool_proj, (1, 1), padding='same', activation='relu', kernel_initializer='he_normal')(max_pool)\n",
+        "    \n",
+        "    output = layers.Concatenate(axis=-1)([conv_1x1, conv_3x3, conv_5x5, pool_proj])\n",
+        "    return output\n",
+        "\n",
+        "def build_inception_v1(input_shape=(224, 224, 3), num_classes=1000):\n",
+        "    input_tensor = Input(shape=input_shape)\n",
+        "\n",
+        "    # Initial convolution and max pooling layers\n",
+        "    x = layers.Conv2D(64, (7, 7), strides=(2, 2), padding='same', activation='relu', kernel_initializer='he_normal')(input_tensor)\n",
+        "    x = layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)\n",
+        "\n",
+        "    # Local response normalization\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "\n",
+        "    # Convolution and max pooling layers\n",
+        "    x = layers.Conv2D(64, (1, 1), strides=(1, 1), padding='same', activation='relu', kernel_initializer='he_normal')(x)\n",
+        "    x = layers.Conv2D(192, (3, 3), strides=(1, 1), padding='same', activation='relu', kernel_initializer='he_normal')(x)\n",
+        "\n",
+        "    # Local response normalization\n",
+        "    x = layers.BatchNormalization()(x)\n",
+        "\n",
+        "    x = layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)\n",
+        "\n",
+        "    # Inception modules\n",
+        "    x = inception_module(x, filters_1x1=64, filters_3x3_reduce=96, filters_3x3=128, filters_5x5_reduce=16, filters_5x5=32, filters_pool_proj=32)\n",
+        "    x = inception_module(x, filters_1x1=128, filters_\n",
+        "    x = inception_module(x, filters_1x1=128, filters_3x3_reduce=128, filters_3x3=192, filters_5x5_reduce=32, filters_5x5=96, filters_pool_proj=64)\n",
+        "    x = layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)\n",
+        "    \n",
+        "    x = inception_module(x, filters_1x1=192, filters_3x3_reduce=96, filters_3x3=208, filters_5x5_reduce=16, filters_5x5=48, filters_pool_proj=64)\n",
+        "    x = inception_module(x, filters_1x1=160, filters_3x3_reduce=112, filters_3x3=224, filters_5x5_reduce=24, filters_5x5=64, filters_pool_proj=64)\n",
+        "    x = inception_module(x, filters_1x1=128, filters_3x3_reduce=128, filters_3x3=256, filters_5x5_reduce=24, filters_5x5=64, filters_pool_proj=64)\n",
+        "    x = inception_module(x, filters_1x1=112, filters_3x3_reduce=144, filters_3x3=288, filters_5x5_reduce=32, filters_5x5=64, filters_pool_proj=64)\n",
+        "    x = inception_module(x, filters_1x1=256, filters_3x3_reduce=160, filters_3x3=320, filters_5x5_reduce=32, filters_5x5=128, filters_pool_proj=128)\n",
+        "    x = layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)\n",
+        "    \n",
+        "    x = inception_module(x, filters_1x1=256, filters_3x3_reduce=160, filters_3x3=320, filters_5x5_reduce=32, filters_5x5=128, filters_pool_proj=128)\n",
+        "    x = inception_module(x, filters_1x1=384, filters_3x3_reduce=192, filters_3x3=384, filters_5x5_reduce=48, filters_5x5=128, filters_pool_proj=128)\n",
+        "\n",
+        "    # Global average pooling and output layer\n",
+        "    x = layers.GlobalAveragePooling2D()(x)\n",
+        "    x = layers.Dropout(0.4)(x)\n",
+        "    x = layers.Dense(num_classes, activation='softmax')(x)\n",
+        "\n",
+        "    model = models.Model(input_tensor, x, name='inception_v1')\n",
+        "    return model\n",
+        "\n",
+        "# Build the Inception V1 (GoogLeNet) model\n",
+        "inception_v1_model = build_inception_v1(input_shape=(224, 224, 3), num_classes=1000)\n",
+        "inception_v1_model.summary()\n",
+        "```\n",
+        "\n",
+        "The paper that introduced the Inception model is titled \"Going deeper with convolutions\" by Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. The paper for Inception model can be found at this link: https://arxiv.org/abs/1409.4842\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "uJw0Ih5DpJSi"
+      },
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/notebooks/ex04b - vit classifier on mnist.ipynb

@@ -0,0 +1,373 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "T4",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bRxeOhhTinjp"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Load the MNIST dataset\n",
+        "mnist = tf.keras.datasets.mnist\n",
+        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
+        "\n",
+        "# Normalize the pixel values\n",
+        "x_train, x_test = x_train / 255.0, x_test / 255.0"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "dAuruOiWivWV",
+        "outputId": "6dac1d1f-508d-4717-e8f9-e85e13bcbcec"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
+            "11490434/11490434 [==============================] - 0s 0us/step\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "x_train = x_train.reshape(-1, 28, 28, 1)\n",
+        "x_test = x_test.reshape(-1, 28, 28, 1)\n",
+        "print(x_train.shape)\n",
+        "print(x_test.shape)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "YcLq_AcLj0Ae",
+        "outputId": "6ea60a8d-db68-42b1-dc8c-c50b20c9d54a"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "(60000, 28, 28, 1)\n",
+            "(10000, 28, 28, 1)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "source": [
+        "import numpy as np\n",
+        "import matplotlib.pyplot as plt\n",
+        "\n",
+        "# Select 5 random images from the training set\n",
+        "images = x_train[np.random.randint(0, len(x_train), size=25)]\n",
+        "\n",
+        "# Create a 5x5 grid of subplots\n",
+        "plt.figure(figsize=(10, 12))\n",
+        "fig, axes = plt.subplots(5, 5)\n",
+        "\n",
+        "# Iterate through the images and labels, and plot each image with its corresponding label\n",
+        "for i, ax in enumerate(axes.flat):\n",
+        "    ax.imshow(images[i], cmap='gray')\n",
+        "    ax.set_title(f\"Label: {y_train[i]}\")\n",
+        "    ax.axis('off')\n",
+        "\n",
+        "# Display the plot\n",
+        "plt.show()"
+      ],
+      "cell_type": "code",
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 445
+        },
+        "id": "eI-O3WwFi5GT",
+        "outputId": "4776351b-4802-4828-b961-546f86a5057d"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<Figure size 1000x1200 with 0 Axes>"
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<Figure size 640x480 with 25 Axes>"
+            ],
+            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAGbCAYAAAA7n8J/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADwbUlEQVR4nOy9d3Cd13nn/729997QAQIgwd5EURKpEkmW5MixbDmOI+1unE1iO+vJROuUmViend849ipOtLEc25NiW9lktLEiybZiSaYlUpWdBAsIgGgXuL33Xt7fH8w5viBAEiRB4F7wfGbuSLwN7/ue+57nnKd8Hx7HcRwYDAaDwWC0FPzVPgAGg8FgMBjXDzPgDAaDwWC0IMyAMxgMBoPRgjADzmAwGAxGC8IMOIPBYDAYLQgz4AwGg8FgtCDMgDMYDAaD0YIwA85gMBgMRgvCDDiDwWAwGC3Iqhlwt9sNHo+Hv/qrv1q27zx06BB4PB4OHTq0bN95O8LGpjlh49K8sLFpXtby2FyXAf/hD38IHo+HEydO3KrjWVW+9rWvgcfjLXhIpdLVPrRrstbHBgB8Ph8+/elPQ6vVQq1W49d//dcxPT292od1VW6HcWnkgQceAI/Hw5e+9KXVPpRrstbHZnx8HH/0R3+EPXv2QCqVgsfjwe12r/ZhLYm1PjYA8NJLL2Hr1q2QSqUwmUz4nd/5HUSj0ev6DuEtOraW5rvf/S6USiX9t0AgWMWjYQBANpvF/v37kUql8Od//ucQiUT4m7/5G9xzzz0YHh6GwWBY7UO87XnllVdw+PDh1T4Mxn9y+PBh/O3f/i0GBwcxMDCA4eHh1T4kxn/y3e9+F1/4whdw33334a//+q/h9Xrxf/7P/8GJEydw9OjRJW8amQFfhCeeeAJGo3G1D4PRwN/93d9hYmICx44dw44dOwAADz/8MDZs2IBvfetb+PrXv77KR3h7UywW8cd//Mf4kz/5E3z1q19d7cNhAPj4xz+OZDIJlUqFv/qrv2IGvEkol8v48z//c9x99904cOAAeDweAGDPnj147LHH8Pd///f4wz/8wyV917LHwMvlMr761a9i27Zt0Gg0UCgUuOuuu3Dw4MErfuZv/uZv0N7eDplMhnvuuQfnz59f8J6xsTE88cQT0Ov1kEql2L59O376059e83jy+TzGxsauyzXBcRzS6TTWWqO2Vh6bl19+GTt27KDGGwD6+/tx33334d/+7d+u+flmppXHhfC///f/Rr1exzPPPLPkz7QCrTw2er0eKpXqmu9rVVp1bM6fP49kMoknn3ySGm8AePTRR6FUKvHSSy9d828Rlt2Ap9Np/MM//AP27duHb37zm/ja176GSCSCBx98cNEV4Isvvoi//du/xRe/+EX82Z/9Gc6fP497770XoVCIvmdkZAS7d+/G6Ogo/vRP/xTf+ta3oFAo8Pjjj+PVV1+96vEcO3YMAwMDeOGFF5Z8Dl1dXdBoNFCpVPjc5z4371hamVYdm3q9jrNnz2L79u0LXtu5cyempqaQyWSWdhGakFYdF8Lc3By+8Y1v4Jvf/CZkMtl1nXuz0+pjs5Zp1bEplUoAsOi9IpPJcPr0adTr9SVcAQDcdfCDH/yAA8AdP378iu+pVqtcqVSa91wikeAsFgv33/7bf6PPzczMcAA4mUzGeb1e+vzRo0c5ANwf/dEf0efuu+8+bmhoiCsWi/S5er3O7dmzh+vt7aXPHTx4kAPAHTx4cMFzzz777DXP7/nnn+e+9KUvcf/yL//Cvfzyy9yXv/xlTigUcr29vVwqlbrm51eTtTw2kUiEA8D9r//1vxa89p3vfIcDwI2NjV31O1aLtTwuhCeeeILbs2cP/TcA7otf/OKSPrua3A5jQ3juuec4ANzMzMx1fW61WMtjE4lEOB6Px/3O7/zOvOfHxsY4ABwALhqNXvU7CMu+AxcIBBCLxQAu7Zzi8Tiq1Sq2b9+OU6dOLXj/448/DofDQf+9c+dO7Nq1Cz//+c8BAPF4HO+88w4+/elPI5PJIBqNIhqNIhaL4cEHH8TExAR8Pt8Vj2ffvn3gOA5f+9rXrnnsX/7yl/Htb38bn/3sZ/HJT34Szz//PH70ox9hYmICf/d3f3edV6L5aNWxKRQKAACJRLLgNZLsQd7TirTquADAwYMH8e///u94/vnnr++kW4RWHpu1TquOjdFoxKc//Wn86Ec/wre+9S1MT0/j/fffx5NPPgmRSARg6fPZLakD/9GPfoSNGzdCKpXCYDDAZDLhP/7jP5BKpRa8t7e3d8FzfX19tNxhcnISHMfhL/7iL2AymeY9nn32WQBAOBy+FacBAPjsZz8Lq9WKX/7yl7fsb6wkrTg2xNVEXE+NFIvFee9pVVpxXKrVKv7H//gf+O3f/u15uQlrjVYcm9uFVh2b73//+/jYxz6GZ555Bt3d3bj77rsxNDSExx57DADmVUFdjWXPQv+///f/4r/8l/+Cxx9/HP/zf/5PmM1mCAQC/OVf/iWmpqau+/tILOCZZ57Bgw8+uOh7enp6buqYr4XL5UI8Hr+lf2MlaNWx0ev1kEgkCAQCC14jz9nt9pv+O6tFq47Liy++iPHxcXz/+99fUF+cyWTgdrthNpshl8tv+m+tFq06NrcDrTw2Go0GP/nJTzA3Nwe324329na0t7djz549MJlM0Gq1S/qeZTfgL7/8Mrq6uvDKK6/My7AjK5jLmZiYWPDcxYsX0dHRAeBSQhkAiEQi3H///ct9uNeE4zi43W5s2bJlxf/2ctOqY8Pn8zE0NLSoqMPRo0fR1dXV0tm2rTouc3NzqFQquPPOOxe89uKLL+LFF1/Eq6++iscff/yWHcOtplXH5nZgLYxNW1sb2traAADJZBInT57EJz/5ySV//pbEwAHMK8E6evToFQUeXnvttXlxhWPHjuHo0aN4+OGHAQBmsxn79u3D97///UV3YJFI5KrHcz1lF4t913e/+11EIhE89NBD1/x8s9PKY/PEE0/g+PHj84z4+Pg43nnnHXzqU5+65uebmVYdl8985jN49dVXFzwA4GMf+xheffVV7Nq166rf0ey06tjcDqy1sfmzP/szVKtV/NEf/dGSP3NDO/B/+qd/wptvvrng+S9/+ct49NFH8corr+ATn/gEHnnkEczMzOB73/seBgcHkc1mF3ymp6cHe/fuxR/8wR+gVCrh+eefh8FgwFe+8hX6nu985zvYu3cvhoaG8Lu/+7vo6upCKBTC4cOH4fV6cebMmSse67Fjx7B//348++yz10wuaG9vx5NPPomhoSFIpVJ88MEHeOmll7B582b83u/93tIv0CqyVsfmC1/4Av7+7/8ejzzyCJ555hmIRCL89V//NSwWC/74j/946RdolViL49Lf34/+/v5FX+vs7GyZnfdaHBsASKVS+Pa3vw0A+PDDDwEAL7zwArRaLbRabUvI3a7VsfnGN76B8+fPY9euXRAKhXjttdfwi1/8Av/f//f/XV8+yZJy1f8Tktp/pYfH4+Hq9Tr39a9/nWtvb+ckEgm3ZcsW7vXXX+eefvpprr29nX4XSe1/7rnnuG9961ucy+XiJBIJd9ddd3FnzpxZ8Lenpqa4p556irNarZxIJOIcDgf36KOPci+//DJ9z82WXXz+85/nBgcHOZVKxYlEIq6np4f7kz/5Ey6dTl/PZVoV1vrYcBzHeTwe7oknnuDUajWnVCq5Rx99lJuYmLjRS7Yi3A7jcjlosTKytTo25JgWezQeezOy1sfm9ddf53bu3MmpVCpOLpdzu3fv5v7t3/7tuq8Tj+PWmNwYg8FgMBi3AawfOIPBYDAYLQgz4AwGg8FgtCDMgDMYDAaD0YIwA85gMBgMRgvCDDiDwWAwGC0IM+AMBoPBYLQg1yXk0ihXdzvSzBV3bGzY2DQrzTo2bFyac1wANjZLHRu2A2cwGAwGowVhBpzBYDAYjBaEGXAGg8FgMFoQZsAZDAaDwWhBmAFnMBgMBqMFuaF2os2CUCiEWCyGVCqF0WgEn/+r9Ug4HEY8Hl/Fo2Mwlg+BQICOjg4YDAYIhULw+XwUi0XE43GUSiVEo1GUSqXVPkwGg7GCtLQBVygU0Ol0cDqd2L9/PxQKBQCgUqngrbfewgcffLDKR8hgLA9SqRS/8Ru/gbvvvhtKpRIqlQperxeHDh1CKBTCu+++C7/fv9qHyWAwVpCWNeA8Hg8KhQIWiwU2mw1OpxNKpRL1eh31eh1WqxVGoxHFYhG5XK6pax7XIgKBgD6kUin4fD7dOZIHcGkcq9UqcrkcarUaSqUSqtUqG6/L4PF4UCqVMBqN0Gg00Ov14PF4cLlcEAgEkMvlEIlEqNVqqNfrq324DAZjBWhJAy4WiyEUCrFlyxY88sgjsFgs2LhxI6RSKQCgXq9DJpNhYGAAZ8+exZtvvsnciyuMVquFyWSC2WzG5s2boVKpYLVaoVQqodVqoVKpqFhDNBrFL37xCwSDQYyMjCAQCKBWq6Fara7yWTQPHMchm80iGo1CqVRCrVZDIpFAo9EgEAhgenoa1WoViUQCqVRqtQ+XwWCsAC1nwHk8Ho19m81mDAwMwGAwwOFwQCwWA7hkwLu6ulCr1RCNRufFxhnLD4/HW/BQKBQwGAyw2+0YGBiATqdDR0cH1Go1jEYjdDodfa/f74fb7YZYLMbs7CwEAgHbRV4Gx3Eol8soFAqo1WoQi8WQSCRQq9WQyWQwGAxQq9XI5XKrfaiMy1js/rheOI6j9wS7NxiEljPgQqEQGzduRHt7O3bs2IHOzk7w+XyEw2HU63XkcjlUKhWMj49jenqaPs9YPsgkJJFIoFAooFAo0NnZCaVSCafTCa1WC4PBAIvFAqVSCYfDAZFIBI7jkM/nceHCBeRyOYjFYigUCnAchx07dmBgYADBYBCxWAzZbBaVSmW1T7VpqNVqmJubg0ajgVqtxvr166khEAgEMJlMcDgcSKfTiEQiq3y0DODSuBiNRshkMrS1tcFms0Gv16OjowMCgQDAryRDeTweDRtdHj7iOA7j4+Nwu92IRqPwer0ol8s07MS4fWlJA97X14dt27Zhw4YNcDgcyGazmJmZQaFQQDweR6FQwPT0NObm5hCLxZgBX2Z4PB4EAgEkEgk11tu2bYPZbMbWrVvhcrmg0Wig1WoBANVqFeVyGT6fD5lMBpOTk5iamoJSqYTBYIDRaMQ999wDoVCId955B+fOnUOlUkEmk1ndE20i6vU6QqEQpFIpent7503yAoEAOp0OZrMZc3Nzq3iUjEbIuOj1emzduhVDQ0Po7u7G7t27IRKJ6PvIgpjjuEVzP2q1Gt566y189NFHcLvdyGazyOVy1BvDuH1pOQMOXLoxSBwcAMrlMqLRKBKJBC5cuIBYLIZgMIhoNIpgMMgM+E3C4/EgFovR3d0NtVoNhUIBuVwOtVoNs9kMtVqNvr4+qNVqWK1WqNVq8Hg8ZDIZpNNpTE9P00VWMpmE3+9HJBKBRCKBUqmE2WyGTqeDQqGAQCCAy+UCx3GIxWIsme0/4TgO6XSaLlAbEYlEcLlcqFarmJmZAZ/Pv6IxYNx6RCIR1Go1VCoVdu7cCYfDgb6+PnR0dMBsNl93SI/H48HhcGDz5s0QCoWYmJgAACQSCealus1pSQMukUho1i2Px0M+n8fMzAy8Xi9eeeUVTE9P02z0er3OVqk3AY/HA5/Ph0qlwv3334/e3l7Y7XYYjUbo9XqaBS0UCunOnMfjIRaLIRQKYXJyEq+99hpCoRBGRkYQj8dppjTZeVitVtRqNZjNZojFYgwNDaFSqWBmZoYZof+kXq8jHo/TRLXG6yKRSLB582a0tbVhdHQUR44cQb1eZ0mAq4RcLkdHRwfsdjs+9alPYf369XTh21iBQbiS65zA4/EwODiIdevWQaVS4fz58xCLxQgGg7f8XBjNTcsZcD6fD6VSCb1eD7lcDuBS3Xc8Hkc0GkU2m12wQ2HcGDweDzKZDCaTCUajEU6nE06nE0ajEQaDASqVCkqlEnw+ny6WMpkMSqUSfD4f5ubm4Ha7EQgEEIvFkEqlFk2yyufziMViEAqFdGevUqlW4Yybm0qlgmKxiFKphHK5DAC0NE8qlUKhUEAsFtMdOGNlIcm1JGHTbrfT5EKZTDbPbU7Gh8TAG8drsbETCoUQCoUQiUTg8/k3nAzHuDakLLPR01upVOg9JxAI6D1HNi9kA8Pn81EqlRAIBFAqlVCv12/pvdhyBlwgEGDdunW48847IZFIwOfzkUwmceLECXi9XlZCs0yQnUJbWxs+/elPw263484774TdbodIJIJQKKQ/5Hq9jnw+j0KhgJMnT2J2dhZnz57FyZMnkc1mEQgEUC6XUSwWF/1bhUIB4+PjiMfj+PVf/3Vs2rQJwWAQBw4cYOGP/4QkANZqNSQSCYTDYbqQJR4SHo8HuVwOoVDIDPgqoNPpYLPZMDg4iN/6rd+C1WpFR0cHVCrVTVfCNHoTyYON8a1BLpdj/fr10Gq1cDgcUKvViEQi8Pl8EIlEUCgUkEql6OrqmldVIxKJIJPJ4Ha78fzzz2N2dhaFQuGWesJazoCTSUqj0dAVaK1WQzabRTabZW7DZYKsJpVKJdrb2+F0OmGxWKDT6QBcMii1Wo3+QMnu2u/305331NQUyuUystnsVSebWq2GXC4HuVwOqVQKg8FAvSuMS5AyolqthkqlgkqlQn/rZDfGyiVXB7IjVigUMJvN1HAbjUaa13GzEF2ExrFnBnx5IPcOEZsimhU6nQ4ulwt6vR4SiQQcx9HX5XI5Ojs7afKowWCAWCyGTCYDn8+n436r78mWM+AAUCqVkM/nIRKJqBa63W6nmbrZbHa1D7HlIQpqBoMBQ0NDcDgc1K1N3Lhzc3M4ffo0EokERkdHkU6n4fV6kUwmkUgkkM1ml7RT4PP5kMlkkMvlqNVqyOfzLDnnMkh+AXGjisVi6pKtVCoIhUKIxWKIx+MoFovMc7FC8Pl8mM1mKJVK7Nu3Dw899BBMJhNsNhskEglNtL0ZOI5DKBRCOBzGhQsXMDo6ilQqxcSplgm9Xg+tVgubzYYNGzbAaDRi586dVHBKIpEgn88jl8tRIy8UCqFWq6nRlkgkdHF9M/X+10vLGXCO42hZEon1CYVCaLVaZLPZZblhGKA/VLlcDofDAZvNRl8jsdhAIICTJ08iGAziyJEjSCaTN+Qy4vP51CBxHIdSqcQSDxeh0YiTEAZwyb1KMtRzuRzzQq0gROLWbDajv78fd999N6RSKWQy2bJO4Ol0GoFAAMFgEIFAgOX5LBONolPd3d244447YDabsWPHDqjVapqUuxRvR6lUQiaTmVfbf6tpOWtHdtnT09OwWCyw2+0Qi8UwmUwoFotUjY2xPHAch2KxiEwmA5/Ph2w2C7fbDZ/PB7fbjdOnTyOdTiOTyaBcLt/Qzk8ikaC9vR0WiwXZbBYTExOIRCLMRdiAQCBAe3s77HY72traoFQqaRVGtVqlanZLyQEh7kK5XE4z/5VKJYRCIU26SaVSiEQiNHmHZLWznf0liCKkTCbDHXfcgU2bNmHTpk2QSqV0XG4WjuOQy+VQKpVw4cIFHD16FKOjo2xxuwwQwy2RSLBt2zZs374dDocDg4ODEIvFiEajiEaj8Pl8SKVSKBaLKBQKEIvFMBgM4PP5KBQK4DgO69evR09PD2q1GsrlMkqlEorFIorF4i0fq5Y04H6/H2NjY+DxeLDb7ZBIJLBYLCiXy8yALyPE25HP55FMJnHu3Dn4/X4cPXoUZ8+eRSqVQigUuumEGqlUip6eHtjtdsTjcQSDQYRCIWbAGxAIBOjt7cXQ0BB6e3vnJUZVKhXMzc3RRMBrQTweOp0O69evpzE/mUxGjTQJjxSLRaRSKVQqFZpIxfiVNoJKpcK+ffvw2GOPQSqVLmvuRq1Wo4vjU6dO4Y033kAqlWIelmVAIBBArVZDrVZj7969eOKJJ2huVS6Xw7lz5xCLxfDhhx9iamoKyWQSsVgMWq0W69atg0AgQDweR6VSwdNPP42enh7U63VqwAuFAvL5/C0/j5Yz4ABo3Ju4yyUSCcxmMyqVCpRKJcRiMc3UZNwY9XodlUoF0WgUhw8fhkwmw4ULFxCPx+H3+5FOp2lW9I0aWpFIBLlcDr1eD5vNBovFgkAggEAggHQ6vcxn1Nrw+Xzo9XrY7Xao1ep5r3Ech0qlctXQA4/Ho/eMxWKBy+WCwWDA4OAgbcsrlUppjb5Wq4VcLqeLt3K5jEQigVwuh3Q6jUQiQWN+tzM8Hg8ikYjuvK8EEdYpFApIp9MQCoU0c/lK1Ot1JBIJxGIxJBIJZDIZFItFtrBdBvh8PrRaLe3uJ5fLwXEcEokE4vE4RkZGEA6HaRlsNptFOp1GtVqFTCaj5WWN3pZSqYR4PL6ii6yWM+A8Hg9qtRomkwlKpRLApc5X27dvh81mg8PhoK5eFie6ccrlMiqVCoaHh/EXf/EX4PF4KJfL1E1EdmQ3M5mo1Wp0dHSgr68PO3fuhMlkwsGDB3H06FFEIhG222tAIBCgp6cHu3fvhslkmpfdSgxDLpejtaqNkNi5yWSCVqvFfffdh8ceewxqtZrq1F8e6yPjXC6XEYlEkM/nMT4+jlAohOHhYRw9epQa99t1nEiiEklkuprbvFqt0lDH+fPnoVQqaaLUlSiVSlQDfXJyEqFQ6JbXFd8uiMVi9PT0oL29HW1tbdBoNAiFQpiYmMD09DR++MMfYm5ujt5TxPvE5/PhdruhUCiwa9cumM1m2gUzlUphbGwM09PTK5Zg2PQGvDFxR6FQ0GYOcrkcYrGYvq5QKKBUKqFUKqFQKGjnJsaNQ+LfV6rfvlkkEgl0Oh3UajXdRZIdyq36m60G2eHJZDIoFAqaFQuA9k8nJZTZbHaBASexbolEAqvVCpPJBKfTCYfDQbXoBQIBLUsiO3CSwFitViEQCFAqlZDL5SASiWhsMJlM0hg5WdDdjiwl45iUgFWr1SX3bCcJnYVCAaVSibnOlxGinaDT6WjpV6lUokmCkUgEsVhs0byPSqUCoVAIiUQClUpFk2+LxSL1lrAd+H8il8uh0+lgsVjw8MMPw263Y/fu3ejs7IREIqEGXCaTQaPRYN26dSiXyzh37hwTdWlSSN2s1WrFHXfcAaVSiaNHj6JSqeD8+fM0eYoByGQydHV1wWg0wuFwQKvV0jwPkpfg8/lw9OhRXLhwYUEJpVqtpjuF+++/H/39/TAajTCbzVTFrVqtIhwOo1Ao0H7iUqkUarUaUqkUFouFltSUSiVs2bIFjz32GCYmJvDyyy8jEolgZmaGNZ+5CoVCAclkkgpRSaVSlq+ziohEInR3d2Pjxo3Q6/WoVCqYmJjAj3/8Y4RCIUSjUVQqlQXeDj6fD5FIBKVSifXr12P9+vUwmUyoVCrweDz45S9/iWAwuGL3QtMbcNIYwGw2Y+PGjWhra4PT6YRGowEwX5KQJCZotVrq1mA0H0Q4QS6Xw263g8fj0bh3IpFYkeSPVkEoFEKv19Na48b672KxCL/fD5/Ph2AwuKCNKJF7tNvtaG9vx8aNG7Fp0yb6OhGHqVarNMZHJi+lUolyuUwT3BpLo0jCqEwmw0cffUQTS68l2HM7U61WUSqVIBKJoNPp5o3jtWCyqcsLqSAgMXCxWIxKpYJkMomJiQkkEgmaYX6lz5JsdKvVSj1VmUwGHo8HsVhsxTYgTWvA+Xw+BAIBuru78cgjj8Bms2FgYIDGjBrdd6VSCYlEApFIBCdOnMDY2BhCodDqngDjisjlcigUCnR0dGDr1q2Ix+N45513MDc3h3A4zIxAA1qtFg899BC6u7vR1dVF3eckwenUqVPw+XwLkv4MBgNcLhfa29vx4IMPwuVywW63A/hVfWo0GsXp06cRj8dx7NgxhMNhKolLErM0Gg02bdoEg8GAnTt3or29ncZ929ra8KlPfQp+vx/1eh0XL15EOp1eVO/+doeU6RF9hcWamlyOQCCAXq9HLpejoiEsOffmkMvlsNlssNlsaG9vh8PhQC6Xw8jICCYnJxGLxZDJZK54jWUyGQ1BdXV1oaOjA7VaDcFgkHbAJMluK0HTGnAS93a5XHjwwQdpMw2JRIJEIkHjpCRmOjs7i1AohPPnz+PixYssXtTEyGQyqnw0MDCA6elpTE9P4/z588x1fhmkzGX9+vVQKBS0VpuUGI2NjdGkzUY0Gg0GBgbQ09ODPXv2wOl0LvjuZDKJY8eOwePx4M0331y0l7hWq8Xc3BxsNhvsdjvtPsfn82Gz2WAymRAMBnH8+HG682AGfCFyufy6S8z4fD7UajUMBgMUCgXNeGYG/MaRyWRwuVzUCFssFoyMjGBiYoLmdVwtd0oqlcJms9HGTg6Hg248YrEYksnkiv7+m9aA63Q66PV6OBwOWm7hdrtRLpcxOTmJcDiMYrGIfD6PYrGIaDSKVCqFZDLJhP6bEFLGJBaLsXHjRgwNDWHdunW0TIZkerJxuwRRh3I4HPO6jAFAIBCA2+3G+fPnEQ6H5/WFJq5Zm82GzZs3w+l0QiaTAQBNNJubm8P09DRmZ2dx7tw5RCKRK0465XIZfr8fpVIJp0+fBsdxcDqddCfO5/MhkUjQ3d2NYrGIarWKaDS6MhdpjSMUCmEymSASibB161ZUKhVEIhFMTU1R1S9mzK8PkUgErVYLnU4HiUQCgUCAVCqF2dlZRKPRKyYXEvlii8WCbdu2wW63QyAQIJlMYnR0FGfPnsXY2NiKbxyb0oDz+Xy4XC4MDAxgcHAQTqcT2WwW77//Pvx+P95++22MjY3RHTipsSS1y7drNmwzIxAIaCLUI488gs985jPIZDKYm5vD7OzsTdeUrzUMBgM2b96M3t5emilLyrzGxsbws5/9DLOzs5iYmEA2m6W/edKOdWBgAI8++ii95vV6nWrYv/fee/jXf/1XxONxzMzMoFQqXdHzkc/nMTo6Sjv/nT17Fvfffz+sVivtSqdUKrF79264XC7E43GMjY2t5KVas4hEInR2dtK6/N27d+P06dN49dVXkUgkMD09zSptrhOpVAqHw0GrMIRCIQKBAM6cOQO3233FBZFUKoVWq0VfXx8+8YlPwGg00tyPX/7yl/jpT39K1ShXkqY04MAl153L5YLRaKTtEVOpFE3TT6VSNP7NaH74fD40Gg3t487n85HL5eB2u+H1elEul+lCjHEpZtrR0UHrtHk8Hq1FzWaziEQidOfduGAlCU+kGQ0xvKRWPJPJ0HyRdDpNteuv1S2uXC4jFotBIpEgHo/P60dOFmeVSgUKheKWX5tmgWwa8vk80uk0LfdbTki4QqVSwWg0wmQywWq1gs/nLxryYFwd0k1MoVDQcASRir6SSA7RHnE4HLBarbS/OxG0IuItq2GLmtKA8/l8bNy4EZ/61Keg1+shEolQLBYxNTWFqakpKizBdtqtg0QiwYYNG9DR0QG5XI7Z2VmcOHECP/7xjxGLxa7qvrod6e7uxuc+9zkYDAYYDAZaZ1oul+H1enH27Fmk0+kFO2dSyw1gXqIUcZ17vV4qDlIul5fclrJarWJychJzc3Po6+tDLBajNelisRi9vb1wOp0wmUzLfzGaEKJbUCwWMT09jVOnTsHlcqGrq2vZM8Z5PB70ej3UajVqtRp1+U5MTLDSvetELpejq6sLNpuNbiIadQ0uvxfIAmr9+vV45JFH4HK5YLFYUKlUcOrUKUxMTGB0dJQKGq30BqQpDTiPx6MrTrlcTncfxWKRKuOw2M/KQOKcZFdH/k06YV1psmoUq+A4DnK5nNYfA0AsFkMwGMTc3BzS6fSKu56aHdIFTq1W03KjcrlM2xqSbO/LFz1kl37582QBkM1mkcvlUCgUrvseyufz9O9f3o9cLpdDJBLRLPnbAbJYSiQSCAQCUCqVKBaL9D4BQDPOr0WjCt5i95RYLIZYLIZarYbVakU2m53nXWGeq6XRKAxWLBbB4/GQy+WQz+dRKpUWXEeRSASRSAS9Xo+2tjYYjUaqnRCJRGj562olTTelAec4DrFYjHYc02g0EAgEUCqV0Gg0S66fZNw4ROdXKpVCr9fT7E21Wg2n0wmLxQKBQHBFMYpwOIxAIEB3KXK5HHfffTdsNhsOHz6M119/HV6vF5FIhLUPXQSSHCaVSumEMT09DZ/Ph8nJSSQSCRp2aKRQKKBSqSCdTiObzUIsFi9bgw2yeGt83K4Q453P5/HLX/4Sw8PD2L59O/bu3QuJRAKlUgmZTIa+vj6qWXElGuu8r2WItVot+vv7IRKJ0N7ejnq9jlgsxjL/l0gikcCRI0eg0WioLTl58iTC4fACb5RQKER/fz+cTid2796NTZs2oVQq0eTRI0eOYHR0dEkNhG4VTWnAASCbzSIajVKReSJKIZVK6e6Pcesgxlkul8NgMECtVqO/vx8mkwnr169HV1fXFTWgOY7DzMwMxsfHqdSnTCZDb28vjEYj3njjDRw+fBi5XA6ZTIbtHhaB1As39vyORqOYnZ29agipUqlQSVoiccpx3LK4dRsN92Lfd7uNI+k+1Zi0p9PpaIMejUYDl8t1VQN+vSItpI45l8tBr9fTJifMgC+NfD6P6elpKi/M4/EwOzu7oAwTuDQH2mw29Pb2orOzEy6XC8FgEKdOnYLH48HMzAzcbvfKn0QDTWnAOY5DKBTChQsXwOPxMDAwALlcjt27d6O7uxtGoxFzc3O0HIbs8m63CWQ5IQbD5XLREj6TyQS1Wo22tjYoFAo4nU4olUqYzWZotVrqirq8gT2Px4PD4YBEIqFiOyKRCBaLBTKZDDKZDFKpFOVyeZ7rkHF1iALUUna+ZBdOFKMkEgkMBgNqtRr1apVKJeTz+atef7JwlkgkGBoagsPhwI4dO2gZDp/PR7lcxsTEBKLRKPx+/3Keckvh9Xrx4Ycf0v7qWq0WIpEIHR0dsFqtMBqNCz5TrVbhdrsRi8Vol0WZTEZzfy4fb+KZUalUcDqdKJfLiMfjSCQSK3mqLUuhUMDs7Cy9rnw+f8G1Iy5zlUqFDRs2YPPmzdDr9bSE77333oPP51vVnTehKQ14vV6H1+vF6dOnIZfLUSqVIJPJsG/fPlQqFfT09MDv9+PQoUO0xSGLi984JC4kkUgwMDCA3t5etLW1oaenBxqNBp2dnfO8HyT7thEyyZD4H+k0dvnfqVar1L3ISmCWTqOEY+Oi6UqUy2Ukk0lIpVLaRclisUClUsFut0Ov19PM26vdN3w+nzYReuCBB7B79260tbXBYDDQYygWizh58iSmpqZu68zo2dlZzM3N0cUwSTyLxWLYvn37vGtGqFQqGBkZwejoKK3S0Ov16O/vp002GsNUpO9D4/01NTW1kqfZ0uRyuQXX6/IFrFgshs1mg9lsxtatW7Fnzx4UCgUEg0GMjY3hjTfeQCgUagrRqaY04MAlF3ooFMLU1BSOHz8OrVYLp9NJtdEBoK2tDV1dXYjFYkin08yAXydEEU0mk8Fms0GpVGJoaAgdHR0wm80wmUwQi8W03j4Wi9Fa4sakMzIpCQQCmM1mqNVq6HQ6uuMrFApUPaxSqUAkEsFqtaJWqyEQCLDs80Vo9GYQF7hGo4HZbIbRaIRWq6UdwhbbQRcKBUQiEQiFQtRqtXl9q10uF7Zv345QKIQzZ87QuDlJNiSLNZVKBblcDqfTCa1Wi+7ubtrGl8fjUQljkq8yOTmJZDK5wleqebg8mYzo/ZOOVeQ9jdTrdQQCAVy8eBEKhQJqtRp6vR7lchlarRYDAwNXzPkhOQmM6+NaHj9iwO12Ow2JJBIJeDwehEIh2hmuGTyHTWnAOY6D1+tFMBiE1+vF6Ogo2tra8NnPfhYOh4PGJUjLw7GxMczOzrJM5uuErDCdTicefvhhWCwWmEwmqFQqGptLpVK0dO9nP/sZZmZmEI/HF52opVIpfu3Xfg0DAwPYunUr7rjjDhQKBXi9XuquLZfLVPjj7NmzmJiYYLK3V6DRiAuFQnR1dcFut8Pr9eLMmTPUcC62E4jH4zh9+jSSySS2bt0KrVYLpVIJuVyO/fv3Y3BwEMPDw3jhhRcQiUSQSqVQqVSoPrTdbseGDRtgNBpp32oi50k8AMlkEocPH4bP58Mbb7xBcx5ud0gNvlKphNPpRFdX1xX7flcqFRw/fhyvvPIKdenq9XoMDAzA6XTi93//96/aM5yxvBD52jvuuAOdnZ3o7u6GTqfDmTNncOjQIbjdbuRyuabZLDalAQcuxYaq1SqSyST8fj8EAgECgQBEIhEMBgNkMhlUKhVNsGIr0aVDSiN0Oh1cLhedsE0mE2QyGYRCIQqFAvL5PGKxGLxeL6LRKLxeL7xeL+Lx+LzmGSROSsIdjb29U6kU/H4/CoUCLT8qFAp0lyeVSun7m2FF2yxUKhVks1nad5jEPvl8PgwGA+0LkMlkUCqVaA0qKd/j8/nIZDL0ddLXm1Rz8Hg82O12+j0qlQqVSmWexrPL5YLBYIDdbqc920mP6kwmg0gkAp/PR+OBrCb5VxA3OunjfqVdNBHYabyf6vU6QqEQxGIx25SsICKRCHK5nHYpIyVjpBVsMBhEIpFoGuMNNLEBJ+Tzedpt6cUXX4TVasXv//7vw2q1QqvVoqOjA+FwGEJh059KU8Dj8dDe3g6Xy4UdO3bgU5/6FFQqFbRaLXg8Hi5evIhIJIKxsTEMDw8jk8nA5/Mhl8vR7OfLd3wqlQpbt26FxWLBvffei40bN6JcLmN4eBhTU1P493//d1r2VK/X0d/fj66uLsjlcgwMDCCdTmNmZobFxBsIhUJ47733qKY5qQcXCARUtjQSieD8+fPI5/PIZrMolUq0oQLHcQgGgxCJRHThS3bRUqkUIpEIGzduxB//8R9TRcNarUalV4kLnVQacBxH80zOnTuH48ePw+Px4J133qGTG+MSJG4tlUphMplgt9uvq/d3oVCAx+Oh/89YGex2O7Zv346Ojg7s3r0bJpMJHo8HFy5cwLvvvosPPviA1os3C01v9arVKm2hNzMzg0wmQ1erpLH6YqVMjMXh8XhQKpWwWq1wuVzo6+uDRCJBoVBAsVhEJBKhP9ojR45Qw73YqpPE4GQyGdUXJm0rPR4PwuEwZmdnqbuXGH6SzyAUCmEwGCAQCOiExbhENpuF1+sFAAwMDKBer9NkQ5LRHI1GwePxkM/nkUqlaH/wUCiEbDZLG/4UCgUUCgWoVCpwHEd34kajkeqkEx160nCmEbKzL5fLKJfLCIVCGB8fx9zc3AItdsav7guyA7/eOnySN8LUJlcOIh7W1taG9vZ2WCwW6HQ6XLhwgSoY+v3+ptp9Ay1gwFUqFUwmE8xmM+677z7Y7Xb09vaCx+OhUqkgk8lcsfk6Y3G0Wi0cDge0Wi3q9ToikQjeeecd2o7V7/cjGAxSo3v5JELioC6XCxs3boTFYsG+ffvobqNWq2FychJvvPEG/H4/1a0nyVR+vx9nzpyB1WrFnj17EA6HMTMzs6iy2O1KJBLBRx99hLa2NtjtdpoVq1KpqHEgLUNJqV61WkU2m6W5Bvl8HkqlEj09PVT2tBGS2NZYVUA8WfV6nX7f2NgYkskkZmZmEI1GqXwkufdYF7nlxWAwYNeuXXC5XNDr9at9OGsek8kEjUaDLVu24IEHHoBGo6Fqhx988AGOHTsGt9vdlHNT0xtwkgXb1dWFxx57DO3t7VQYoVKp0HaibAJZGmSlSUqK6vU64vE4Dh48iMnJSVy8eBHhcPiq3yEUCiGVStHW1kYXVXv37oVWq6UNMmZnZ/Hee+8hk8kgm83OW7mGw2GUy2UYDAZs3LgRgUAA//Ef/0ET59hYXkpCO3v2LJLJJDZv3oxarQalUkkTDEksW6lUApgvv7nU60eS4xZ7PxEpSSaTGB4eht/vx7FjxzAzM4NkMtkUNbBrFa1Wix07dsDpdF5TxY1xc5Dqjo6ODvT392P37t3g8XjweDyIRCI4ceIEDh482LSdEm+5ATeZTOju7oZEIqGu7lQqhXw+v+C9JBGNx+PRWJzBYEBbWxssFgsMBgOkUimAS8ab7BjdbndT1OS1CjqdDu3t7ZBKpZibm4PH40EgEKAu18UQCAQ0ebCrqwtOpxM9PT3o7e2FUqmkHeLGx8cRCoUwMjJyxQ4/RHgnk8nQjHbiImZcgniXQqEQ3QHE43HY7XYYDAaYzeZFhXSux3gv9l5SHuPz+XDu3DlEo1GcPHkSsVgMgUCAJsUxrgxZ/OTzeQQCAXg8HjqfXQ6fz4fT6cT69eup29xms8HpdMJut9P5rvG7SYKj3+/H3NzconMp4+oQ75NQKITVaqUCYdVqFfl8ni5aSfiwGY03sAIGvLu7G0899RQ0Gg3Vz7548SJCodCi7yfxo8HBQXR3d1NZQrLrI8pPxWIRExMT+MUvfnFVw8OYD+m1vm3bNoRCIQwPD8PtdmN0dBQ+n++KbiKxWIyenh5YLBZ87GMfw9133w2lUgm9Xo9cLoeRkRFEIhH85Cc/wZkzZxCNRhGNRhdttECMeiQSgdvtpm0x2Q78V5B6+0QigWAwCJlMhm3btqG9vR1btmzBHXfcAalUCoVCsawVGCTOffToUXzve9+jjTqI4EszuhGbjWq1ilwuh0QigfHxcYhEIgwODi5qwIVCIYaGhlCtVhEKheD3+zEwMIChoSGYzWbqYSEQ4x2NRjEyMoKJiQmmwnYDkAY8MpkM69atwx133AG73Y5SqQS/34+f/vSntPtes8W9G7nlBpzUapPdtFAopJ10rgRRjdLpdLQMA7j0463VaohEIjQ7mpSvsInl+ri8eYJYLJ43JhKJBBKJBAKBACKRCAqFAj09PbBarbQCoFarIRwOI5lM0lpxUmpxtXg2ia+m02n4fD5kMhnWD3wROI6jO4JqtYpwOAyBQEAbMRCt+uU24KRxSjQapXFu5uFaOiTpj4xdNpu9YjmYQCCAyWRCR0cHDZG0t7fTSoDLx7ZcLiORSCCZTCKTydDfBuP6ID3siTiS2WyGVCpFMplELBaj17jZy/huuQEnBkAul8Nut0OpVMJut1/xR0cMC8mGJa0rSYZtKpXCO++8g9HRUYyPj2NsbGxea0PG1Wms463VanRh1dHRQRdKAOB0OtHZ2QmlUgmbzQaVSoWNGzfCaDTSUqSRkRG89dZbCAaDOHr0KHWH5/P5q65aSWXB9PQ0wuEwarUaVdJjBnw+ZLwqlQrOnz+P8fFxHDt2DP/2b/9GPRbLZcAb279ms1nEYjFqiBhLp7FXeCKRQDQavWI5mEgkwu7du7F582aajEg6AJJqgUYikQiOHz+O6elp2s2vmXeIzYpUKsWWLVvQ0dGBvXv3Yvfu3fB4PDh9+jRmZmYwMzNDuyk2M7fcgNdqNeoOJHFOkkl7ecyz8d+kdKxarVIVL7Iymp2dxdTUFILB4BWlJBlXhoip1Go16oY1Go3zrr/D4UB7ezttH0oWXlqtloYw4vE4pqenEQwG4Xa7kUqllpSRTHbapL804+qQHR3pmMRcps0PyewvlUooFApXXATx+XzqUWn87OWQRS/Ji4hEIswzcoOQ5E2DwQCbzQa9Xg+lUgmO4xAOhxGJRJDL5Voi1+OWG3C3242XX34ZBoMB4+PjMBgM2Lp1K9rb22m3ncshUqqBQIC2UEwmk5iYmJgn7cmM9/XDcRzm5uZw/PhxOJ1ODA4OoqenB93d3fQHS2rFyUJLIBDQHWA2m8Xs7Cz8fj9mZ2dx6tQp6iZk5UQMxq8g4T65XE49TDfiMeE4DmNjY5icnMT58+dx4MABJBKJeeptjKVBtEPMZjM2bdqEjRs3QiKRYHp6GqdPn8Z//Md/IBaLtYyq4C034LFYDKdOnaIxU7PZDKfTCZvNRjMBL4/HchyHRCJBW4aePn0aiUQCFy5coL1vmz020axwHIdkMom5uTlotVpYLBZIpVL09fUtmgVOsqHT6TT8fj88Hg/Onj2LixcvIpFIwOv1svwDBmMR6vU6stks4vE4isUilbi9ke8JBAI4f/48zp07h9OnT1PlPMb1IRAIaE6Wy+VCZ2cnEokEIpEI5ubmaPVMK+y+gRUw4JVKhSY0jY6OwuPxoFgs4v3334dAIFiwAydZyIFAAJFIBMlkEj6fj+oFsx/uzUF24CTOSeJtAwMDUCgUNK7qdrsxPT2NbDaLYDBIBT3i8TgdGyagw2BcmUqlAo/Hg3Q6jePHj0MqlcJisaCnp4eWMF2NarVKZaRPnz6No0ePwu/3o1qtskXzdULmNb1ej507d8LpdMJkMkEkEtGGWRMTEygWi03TaWwp3HIDTtScSOkDj8fD4cOH59WuLgaJkzZmJrfKRW1mOI7D9PQ0ZmZm4PV6kclk4HK5oFaraZkfAAwPD+PnP/85otEoLl68iHw+z7LFGYzroFKpYHp6GgKBACqVCtlsFps3b4bD4QBwaTd4Ne2DcrmMyclJ+Hw+HD58GIcOHaKStozrg4QuTCYT9u/fD4fDAavVCrFYDLfbjUOHDsHr9bZcVv+KKrGxib85IOOQzWbh8/lQLpdx5MgR6PV6ulI9d+4cXf2TRhmLyaoyGIwrQxIQY7EY3G435HI5Ojo6oFarYbVaaRe/Rk9kqVRCMplEOp2Gx+OBz+dDIpFgO++bgPRZb2trg8PhgF6vRyQSQTQahcfjQTAYRDqdbjn7xOOu44hvd6WsZh7cGxkbEsIgTRdIZUC1WkUkEoFQKKQJbM2++GrmY2P3TXOOzUqOi1gshkgkQkdHB+3cd88998BsNqOzsxMGg4G+l5RlfvDBB/jrv/5rbN68GfF4HHNzc8t6TM06LsCtGZvdu3fj05/+NJ588kkAwE9/+lNMTU3hgw8+wPDwMGq1WtNk9S91bJpeC51x6yBdjwAsmnVJerIzbpwf/vCHq30IjCaAKNzF43F4vV5Uq1V4vV6USiUIBALk83k6aQeDQXg8HkSjUQCgYjqM64ckSZfLZajVahr3LpVKiEaj8Pl8SCaTLavkyQw4g8FgrBCkSc34+DjOnTtH1fREIhF9D3GhJ5NJAGjKNpatgEAggMvlgkqlwpkzZ7B+/Xps2rQJwWAQkUgEp06donr/rQoz4AwGg7FClEolWqLk9/uX9Bm2+74xeDwe1Go1FcnR6/UwGo20MVAkEmn5PhrMgDMYDAZjzUF6athsNrz77rsQCAR44YUX8L3vfQ/pdBoikajlE3OZAWcwGAzGmoPP58NoNMJutwMA/vVf/xWJRAJDQ0OIx+M4f/58UyfyLYXla2PEYDAYDEaTUKvVMDMzg3PnzgEALl68iKeeegpKpRIKhWJe86ZWhe3AGQwGg7HmqFQqOH78OC1Jq9fr+Nu//VtaFrsWKmyYAWcwGAzGmqSxrnstlsVelwv9Bz/4AQDg+PHjC6ROyeOf//mfAQCPP/44XnzxRbz55ps4cOAA7r33XrS3t9P3zczMAAC++tWvLviO3/7t34ZEIgHHcTh8+DAA4JlnnsGBAwcWfRAFnYMHDwIADh48eMXjW8pjeHgYWq0W27dvRyaTaXoRE+D2GRuO4/Dcc88BAGZmZlpibIDbY1waH7/5m78Jq9Xa1GPD7pnmZS2PzT/+4z9CJBLhww8/pL3HyTE+9dRTmJmZWXKb5WXfgb/88svo6urCK6+8Mk9N59lnn130/RMTEwueu3jxIjo6OgAAXV1dAC4V5N9///3LfbgLmJqawkMPPQSz2Yyf//znUCqVt/xvrhStPjZrlbU4LoVCAalUalX+9nKyFsdmrdCqYzM3N4dKpYI777xzwWsvvvgiXnzxRbz66qt4/PHHr/ldy57ERpphNK7wjh49Slc3l/Paa6/B5/PRfx87dgxHjx7Fww8/DAAwm83Yt28fvv/97yMQCCz4fCQSuerx5PN5jI2NLalYPxgM4td+7dfA5/Px1ltvwWQyXfMzrUQrj81appXHJRwOL3jO7Xbj7bffxvbt26/5+WanlcdmrdOqY/OZz3wGr7766oIHAHzsYx/Dq6++il27dl31Owg3tAP/p3/6J7z55psLnv/yl7+MRx99FK+88go+8YlP4JFHHsHMzAy+973vYXBwENlsdsFnenp6sHfvXvzBH/wBSqUSnn/+eRgMBnzlK1+h7/nOd76DvXv3YmhoCL/7u7+Lrq4uhEIhHD58GF6vF2fOnLnisR47dgz79+/Hs88+i6997WtXPa+HHnoI09PT+MpXvoIPPvgAH3zwAX3NYrHggQceWMLVWV3W6tikUil8+9vfBgB8+OGHAIAXXngBWq0WWq0WX/rSl5ZyeVaNtTouQ0NDuO+++7B582bodDpMTEzgH//xH1GpVPCNb3xj6RdoFVmrY9Pq9wywNsemv78f/f39i77W2dm5pJ03hbsOfvCDH3AArvjweDxcvV7nvv71r3Pt7e2cRCLhtmzZwr3++uvc008/zbW3t9PvmpmZ4QBwzz33HPetb32Lc7lcnEQi4e666y7uzJkzC/721NQU99RTT3FWq5UTiUScw+HgHn30Ue7ll1+m7zl48CAHgDt48OCC55599tlrnt/Vzu2ee+65nku14qz1sSHHtNij8dibjbU+Ls8++yy3fft2TqfTcUKhkLPb7dxnPvMZ7uzZszdz2VaEtT42rXrPcNzaH5vFAMB98YtfvK7PXFc3MgaDwWAwGM0BE3JhMBgMBqMFYQacwWAwGIwWhBlwBoPBYDBaEGbAGQwGg8FoQZgBZzAYDAajBWEGnMFgMBiMFoQZcAaDwWAwWpDrUmJr1Ju9HWnmknk2NmxsmpVmHRs2Ls05LgAbm6WODduBMxgMBoPRgjADzmAwGAxGC8IMOIPBYDAYLQgz4AwGg8FgtCDMgDMYDAaD0YLcUD9wBoPA4/HA5/Mhk8kgFAphNBqhVCohlUqhVCrpeziOQ61WA8dx8Hq9iEQiKJfLKBQKq3wGDAaD0ZowA864YXg8HgQCAaRSKZxOJzQaDe6991709/ejra0NXV1d4PMvOXk4jkOxWESpVMJLL72EN954A/F4HHNzc6jX66t8JgwGg9F6MAPOuG74fD74fD7EYjHkcjkUCgUcDgd0Oh1cLhecTid9LGbAnU4nHA4HeDweQqEQKpUKqtXqKp8Vg8FgtBbMgDOWDJ/PB4/Hg0qlglwuh8vlwsaNG2GxWHDXXXfBYDDAYDBQFzowX5BAJBJBIBBg//796Ovrw0cffYRcLodUKoVoNIpKpbJap8ZgMBgtx5ow4Hw+HwKBAMAlg8FxHOr1elMrDbUiAoGAxrvVajUsFgt6e3ths9mwZcsWGAwGAL8y2pe7xsnO3eFwwGg0IhwOQ6vVolqtIpFIMAN+k/B4vAWPRmq1GgtXrDJkTBrHh3iplgoZQzbHrS6N48fn8+fZnZUal5Y24FKpFDKZDO3t7XjwwQchkUgwOzuLVCqF8+fPY3JycrUPcc0gk8kwNDQEk8mEDRs2oLe3FzqdDk6nE0qlEgqFAhzHIZVKIZ/PI51OIx6P0x+zRCJBX18fNBoNxGIxNeQ7d+6Ez+dDPB5HsVhc7dNsSQQCAXQ6HWQyGVwuFywWCzQaDSwWC8RiMZRKJcrlMv793/8dw8PDq324tyUSiQQikQhKpRJ6vR4ymQwOhwMKhQIdHR108dtIo7En91EikYDH40EikcDw8DBSqRSKxSKq1SrEYjEkEglqtRpKpRLq9Toz8rcIoVAIs9kMuVyOrVu3YmBgAD6fD+fOnUMqlcLs7OyKJOi2tAEXi8VQq9VYt24dnn76aahUKhw5cgQ+nw+pVIoZ8GWCx+NBIpFgYGAAPT09uPfee7F79+4F76vX68hkMkgmk/B4PHC73fR5lUoFm80GjUYDoVBIb4CNGzdCoVDg8OHDK3xWawc+nw+tVgutVouNGzdi/fr1sNvtGBgYgEKhgNlsRj6fx8jICDPgq4RIJIJCoYDBYEBnZyd0Oh02bdoEo9GIPXv2oLu7e9HPNeaQcBwHt9uNkydPYnZ2FoFAgHpV6vU6JBIJVCoVKpUKarUa87jcQgQCAUwmE4xGIx544AF87GMfw6lTp8BxHPx+P4LBIDPg10IsFkOlUtGYq0QigVQqhVwup49qtYpyubzah9rS8Hg8iEQi2O12dHR0QCKRIJlMIp1O0yS0UqmEcrkMr9eLRCKBaDSKcDhMV/8ajQadnZ3g8XjQaDR0zAwGAxKJBKRSKUQiEarVKtsxXCcikQjt7e2w2+3o7+/HunXroNVqoVKpIJVKqYvPYDDA6XQik8kglUqt9mGvOYRCIUQiEYRCIeRyOcRiMcxmMzXcer0earUadrsdSqUSbW1tUKlUUKlUV/xOjuPmhULI56RSKe666y7EYjEkEgnk83kYDAZYLBbk83mEQiHkcjl4PB5ks1lkMhnkcrmVuAxrisZ5qVqtQqVSweVyQaPRYMuWLbBYLOjp6aFltOVyeUXnsJY24HK5HCaTCXq9nrpxNRoN8vk83ZHk83lUKhVmFG4QUuctlUoxMDCAnTt3olarwe/3Y2JiAu+//z5yuRwikQgKhQLcbjfi8ThKpRJKpRKN9RmNRtjtdmSzWQwMDNDxam9vR6lUosamUCiwjPTrRCqVYuvWrVi/fj02bdqEdevWUaNNrr9AIEBbWxs2bdqEqakppNNpdk8sM2QHrFAoYLPZoNPpsGfPHtjtdnR1dcHhcFB9BB6PB6FQSMfmSpAxIv/V6/XQaDQoFovo7OxEsVhEJBJBNptFR0cHent7kUwmMT4+jkgkggMHDmB2dhbT09PMgF8nfD4farUaSqUShUIB+XwebW1tePTRR2G327Fv3z7Y7XZIJBKIxWKIRCLk83nk8/kV83y0tAEnO3CZTIZarYZisYhwOAyfz4d0Os1cSMsEx3GoVqsIBoNwu90olUqoVCqYmZmBz+dDPp+nMWyyMycrVuDSjZDL5RAOhxEIBOByuQD8yq2oVqthtVrpe9Lp9GqebktCDLVQKIRYLF70db1eD6fTiVgsRuOqjJtHKpXSkJDNZoNKpYLD4YBarYbT6YTFYoHBYIBWq4VIJIJMJluQYEgSoGq1GqrV6rxNR2NSokgkgkQigUQigVarpd5FhUJBNzIcx8FkMkEgENB7LRKJrOxFWQPw+XzY7XbYbDYUi0UUCgW4XC60tbXBbDZDo9FAoVAgl8shmUwiFoshlUohm82iVqutyDG2tAHXarXo7e2FyWRCJpNBPp/Ha6+9htOnTyMSiSCRSLAkjpuEKKjF43G8+OKLNGu8Wq0ik8kgGo3SSYfjOOpCaqReryOfz+PDDz/E1NQUDAYDhoaGaCmaWCzGI488Ao/Hg5///OcYGRlZpbNtTUjuQSKRuGIioEgkwrZt22C321Eul3H8+PEVPsq1iUAgoFUVu3fvxr59+6BWq+FwOCCRSCCXyyESiSASiSAWi6/a57pYLFJ3t8fjoVUZPB6PhggNBgPa29up6iEx1vV6nX6/QqFAb28vSqUSNBoN4vE48vk8Lly4sFKXZU0glUrx6U9/Gg899BBKpRKKxSKUSiUdW6I0OTIygnPnzuHUqVMYHh5GPp9fsYTcljbgYrEYCoUCYrEYpVIJuVwOwWAQHo8HxWKRlSUtI5VKBX6/H9FolBrwcrmMfD6/pM9Xq1XE43EAQC6XQ71eB5/PpzeCzWZDrVaDQqG4laexJuE4DoVCAblcju7cLjcUJPegVqtBpVJd1ZAwlg7RRTAajXA4HOju7oZarYbNZoNQeGl6XWwD0VhyRJLQyE4umUwiEAjMM+Akp6FRnlgkEtEE00YEAgFdOOh0OgCgugyMa0M8WVKpFHa7HT09PdSAS6VS6HQ6mlxYq9WQSqXg8/kQCoWQyWRQKpVW7Fhb2oDz+XyIRCKUy2XMzc0hlUohmUyiUCismAvjdoHjOGSzWQgEgnkTz1Kp1+tIp9Oo1+sIh8MIh8O0nlwsFsPhcEAoFNJyKOJGZFybYrGI4eFh+P1+dHR0YGhoiCZUEUislYjpMJYH4tnYsWMH1q1bB7vdDrFYfM1rXCqV4Pf7USgUEI1Gkc1mMTMzg4mJCaRSKboDJ7FVUvlxPWNXKBRw5MgRzMzMYGZm5mZP9bZBoVCgv78fJpMJOp2OblbIrposyGq1GsrlMmZnZ3HixAn4fL4Vtzstb8D5fD518RLjzSb+5Ye4x2/m86SsgrgJye6FlEFVq1WavctxHBvHJVKtVuH1epFMJqmHBMA8Aw78SvDoeoVDGFeGxJmHhoZomeRSrm+lUkE8Hkc6ncbs7Czi8ThGRkZw8uRJZDIZagxkMhmkUinWr19PkxKvRuNuv1KpwO12Y2RkBLFY7KbP9XZBLBbD6XTCZrNBLpejXq/TDYVIJJpnwIlncXZ2FslkcsVzrprKgPN4PLp6VSqVkMvlyGaziMVii7qhBAIBjftks1lks1mWwdykkAUAj8dDMplEKBQCx3EwGo0Qi8XQ6/Xg8XhQq9WQyWSo1+tM2GWJ1Ot1ujgqlUqo1WoLdmo8Hg9yuZxWBJCkwVQqxRI9bwKSI1KpVFCv1+eJrwC/Ur/z+XwIh8OIRCKYmZlBLpdDIBBAoVBAMplEPp9HMBhENBoFn8+nAkm7d++GzWbDjh074HA4oFQqFxhy8v/EzRsIBHDkyBFEIhF89NFH8Pv9zIBfB8ViERMTEwiHw6jVavjwww/R19eHDRs2UDtUqVQQCASQTCbp9S0UCiueb9V0Blwmk0EsFsNms8FkMiEYDCKRSCzqmhAIBJBIJOA4Dul0GqlUihnwJoXjOGpcYrEY/H4/Xc0KhUKYTCZIJBJoNBrI5XJWu38dcByHXC5H27NWq1UIhcJ5sfBGDXuLxQKn04l4PI5sNssM+E1CXKxkjmqUSSUVHFNTUzhz5gzOnTuHt956C4VCYV6ojwi1kEXtunXr0NbWht///d/Hxo0bryiP2/jvUqmEVCqFkZERvPDCCwgGg7SkkyXyLh2S8Mfj8XD06FHw+Xx87nOfQ39/P72O1WoVc3NzCAQCmJubm6d5sZI0lQEnghQ6nQ5WqxVGoxE8Hg8XL16cZ8Abm2pYrVaUSiUEg0H2I20RFpuMVlI/eK3CcRySySS8Xi/0ej3kcvk8dy75f4FAQF3pLJnt5qjVavB6vRgZGQGfz4fFYoFIJKJJY8ViEcViEcFgEJOTk/D7/cjn87QUs/E339jZb2hoCA6HA1qt9prJcI078FQqhXQ6jVwuh3w+z4SRbhAyHwkEAuoZlslkkEgk4PF4tKx2ZmYGiURi1a5xUxlwhUKBX/u1X8Pg4CCMRiMMBgPefPNNfPTRR/N2ZCRBx+VyYceOHYhGo0gkEmwyanKIKMzlyVRkl1KtVqnLkU061w/HcZicnMTBgwcxODgIm802z5VOjDYpaSKGgXHjlMtlHDp0CKdPn8Zjjz0Gi8VCdQ0A0NrgY8eO4Wc/+xny+Tyy2eyiC1az2Yx169ZhcHAQTz/9NO3uByxuvAnkNaLBTXoLkKRRxo1BPMKk5t5oNFJlw0KhgGPHjuHUqVOYnZ1dtWNsmjuYpO7r9XpYLBaqzHV5Ig4wvx+1UqlENptlwhRNDil7EYvFVO6WrGZJvLtQKNA6cjbx3Bjlchm5XA7FYnHR+4F4Psg9xLg5OI5DJpNBtVpFLBZDJBJBvV6HXq+n1/dKFQBkw0EanRiNRjidTtjtdpoB3Zg01TieRFuBJFLVajWEQiGEQiHE43Eak2fcOKR8T6PRUBEePp9PQyZEe2Ely8YupykMONkVyOVyOJ1OdHV1wefzwePxwO/3L9qWksS/iSFgnXeaG4FAALPZDLVajYGBAWzduhUSiQR8Ph/5fB7j4+MIh8OYm5ubl0nNWDqNcdQrvd74X8bNQ8orC4UChoeHIRaL0dXVBaFQCLVaDbVaDY1Gg/vuuw96vR7nz5/Hm2++SSd9kUhEm8/s3LkTd999N5WBJgZjMVKpFD788ENEo1GaxR4Oh+H3+6k2OuPmkEgkePTRR3HHHXdgYGAAcrkcpVKJKkpGIhHE4/EVaVpyJZrCgJMYg1QqhUqlgkajwezsLK2PbJxwGncQjatakurPVp3NCZ/Ph1KphFarhcFggNFopK+RkppIJIJMJsOyz5eBy405Cy/dOkj4JxqNYmpqCgKBAMlkEgKBgEo9O51OlMtlpNNpWg0AXCpZslqtVMd8YGCAeqoWM95ko5LP5zE3Nwe/34/R0VGaER2LxRZVQ2RcP0KhEJ2dndi6dSuMRiOEQiGKxSIymQzS6TRVXFtNzZGmMOA6nQ5DQ0NwuVwwmUwQiUSYnp7GO++8A4/HM+/HSJLX1Go1Ve3KZrMYHx+H1+tlK88mg7gPNRoNHnjgAfT19aG/vx/Ar+ook8kkRkZGMDc3R9XaGDdGvV6flxHNWDlisRhGR0dpXwC9Xo89e/bAZrNBIpFg/fr1VLWNaBwIhUJ0dXVRJTepVEoTpxopFosol8uYnJzEkSNHEA6HceLECdr5L5vNolQqoVAoUCPPuDFIuE8qldKQrkwmAwCEQiH89Kc/hc/ng9vtRjabXdWKmaYw4EqlEhs3boTT6YROp4NQKMTc3ByOHDmyQFWNaP1qtVpIpVLweDzk83m43W54vd5VdWcwFkJyG1QqFfbs2YPdu3dTecd6vY5SqYRMJoOpqSlMT0+zRiY3SWN/aMbKkkqlkEqlaGmRwWCATCZDNpvFpk2b0NPTA4fDgfXr11MpYbIhIQbiSpTLZWSzWYyNjeHHP/4xYrEYZmdnqcFmYZHlo7FpjEajmectjEajOHToEDweD23ktJqsqgEnLnC9Xo+uri5YrVYUCgXqDiJJTRzH0ZaWEokEHR0daGtro5mearUaGzZsgNlshslkojWxxK1eLpdRKpUQCoVYffEKI5PJ4HA44HA4oNPpoFAoaGJiuVymOwi/3w+fz8cWYDcBx3GIxWKYmpqC3W5nRnyVqNVqVKDl3LlziMViUCqVMJlMtDFJY30+0TS/nHq9Tnd4w8PDGBsbw9jYGEKhELLZLE1UY8Z7eSDhWY1Gg82bN8NqtcJisQAATbJNJBJIJpM0cXG1WVUDThpZ2O127N69G1qtFj6fD7OzswgEAvN6FhOdbJVKhR07dmDTpk3o6+sDn8+HyWTCQw89hEwmg2AwiHw+j1wuR8USUqkUotEoPvjgA2bAVxiNRoOtW7fC5XLB4XBQxTXgUlMT4ooaGRmB2+1mrt+bgOM4zM7OIhaLwWq1smu5SlSrVaRSKWQyGbzxxhu06sLpdEKr1cLhcCypBr9WqyEcDiORSOCll17Cyy+/TDck19uLgHFtyGLKYrHgySefRE9PD3p7ewEAmUwGgUAAXq+XJrA1gy1ZVQMuk8lgNBqp6IRMJkOlUkGhUIBEIoHZbKaxCKlUCqvVCpVKBZfLBbPZDIVCQXfncrkctVoNPB5vXi1xYyIPW6muHKTPt91uh8vlgt1un5e8A1xa1ZJMTpZ4szwQWU9mvFcfPp8PlUpFu4iR2vsrCRkRqtUqisUicrkcpqenEQqFEAgEkMlkVuM0bhsEAgGkUikUCgVNtCWd3lKpFNxuNwKBAEqlUtMI5KyqAXe5XNi9ezcGBgZgsVggFAqRz+cRj8fR1dUFpVIJq9WKdevWQalUwuVyQSqVwmAwQC6XQyAQUDUjIr8ZCATg8/lobSTpsZtOp9mktoIMDAxg7969cDqduOuuu6DRaGA2m+e9JxQK4cMPP4TH40Eul1ulI2UwlheS96FWq/HAAw+gs7MTe/bsQVtbGzXgwK9U1C43BJlMBhcvXkQgEMAPfvADjI6OIhqNrsap3FbI5XLYbDa0tbWht7cX3d3dNNxx9uxZ/PCHP0QwGKSS3be9AVcoFLBardDr9VTUg8g86vV6SKVS2Gw2dHR00EbqUqkUMpmMpvTn83nk83naDzmVSiGRSNAdOMnMLBaLzOW0CGQncHkcrnGXcCV3X71eR61Wm+flEAqFNKzR3t4Oh8MBm81GdyCNlMtlxONxpmG/zFyrHlwkEtGeA0zMZfkRCASQyWRQqVSw2+3o6OigCW2Xs5h3sFarIZ/PI51Ow+v1YmZmpimMxVpHLBZDo9FArVZDqVRSD2+tVkMymcTc3BySyeQCCdzVZNUMOI/Hg8vlwt69e6mrgs/nY/Pmzejr66NlE8R1wXEcRkdHIZFIsH37drS1tWFiYgLHjx9HIBDAqVOnkE6naSIUmcBIIlu1Wl1VxZxmhEzkWq0WW7duhVqtpokxJG6nUqngdDoXld2MxWJwu91UgxkA+vr6YLFY0NXVhXXr1kGhUFBFqcs7ZFWrVZqrwBZXywPJQCduWIlEQhdOZEE2ODiIz372szh37hx8Ph9isRjTUFhGLBYL9u3bB5vNhvvuuw/t7e1UEnUpSKVSOBwO8Hg82Gw2BINBZDIZ5qW6xdhsNuzfvx/t7e1QKpXgOA7xeJx2jguHw8jn8011n6zqDlyr1aK7uxtKpZLuvJ1OJ4BL4h7VahXj4+M4f/48TUgTiUTo6elBW1sbQqEQhoeH4Xa7cfDgQfYDv06EQiFkMhl0Oh02bNgAk8lE4ztEUMdgMGD9+vU0FtSIx+PB6dOnkcvlEA6HAQB79uxBd3c3dDodzbq9UrJOrVZDuVymSTmM5YF4RsjClSh6kXGw2+0Qi8W037RQKGThpWVEo9Fg48aNcLlc6O/vh81mu+r7L//ti0QiaLVaFAoFuhsslUpsfrvFqNVqrFu3Dna7nTajyeVySCQStElMs0nUrqoBr1QqtBwil8uhXq8jGo2iUCjQ/rmBQACjo6PULavRaGh5WSgUwsjICCKRCBVGYFwbo9FIs2FJ45idO3dCo9HQBEDSfYfswhfTpLdYLNi4cSOtUQWAtrY2mpTYaLyr1Sr8fj+N77ndbkxMTGBychKZTKYpMjrXAsS9l0wm4Xa7YTKZaO4I8CvVQ41GMy+xil3/m6e9vZ1mLm/evBkmkwlKpXLR95ZKJUQiEeTzeYyOjiIYDMJut6OtrQ1yuZw2c7r77rvhdDrx7rvv4sSJEyt8RrcHpDzZarWip6cHRqMRYrEYlUoFw8PDuHDhAkZGRmheVTOxqga8VCohnU6Dx+PR7HOSsHHixAmMjIzQMjCNRoOdO3eiXq/TZg0+nw+nT59GsVhkBnyJ8Hg8WCwWdHd3Y9OmTXjkkUeoMSeTfON7G/97OXK5HHa7fcFnFnt/uVzG1NQUfD4fXnnlFRw4cIDuEllJzPJRqVRQqVQQiUQwPT2NQqEAs9kMqVRKd3pkwlKr1fMyoxk3R09PDz7+8Y+jvb0du3btuqLxBi7NfW63G+FwGP/v//0/nDhxArt378b+/fvhdDphs9lgNpvx0EMPIZlMUuU1xvIjl8uhVqvphkalUkEoFCKbzeLo0aN44403EA6Hmyr2TVhVAx4IBHDs2DHaX7VUKmF2dhbpdBqhUIgKslSrVboDN5vNdHVEHiSRinFteDwelEolzGYzDAYDtFotlaRtzKwslUoolUq0tIJ0f2tMerqae/xKf5uU+ZVKJdYD/BZSKBTg9/vB5/PnLW4btdGlUiksFgsNY7BF8NIhngyhUEiT1DZv3oz29nZYLBZ6r5DrTZJqk8kkfD4fkskkhoeHqYhRNptFJBLBzMwMBAIByuUybfDEcRza2trQ39+PTCaDUCjE5rxlgsfjUeOt1+tprk8ul0Mmk0EymUQymbxid7/VZtUMOMdxOHz4MM6ePTvvORJjIBMKuWgKhQJbtmxBV1cX1Go1FZJnE8/1wePxYLfbsXHjRvT29sLhcEAgEKBQKND2hLVaDdFoFKFQiJZWkPK9xWLhS4XIExIjzrh1hMNhHD58GN3d3bjzzjvnyUEStFotdu7cCbvdjkwmQ8MgjGsjFApp17DPf/7zuOuuu6DX62nTC3KfkDIxklg4MjKCl156iebvkKYYlUoFIyMjiMVi2LFjB+655x6IxWL6N/bt2weVSoUzZ87g9ddfR7FYbJpSplaGx+Ohs7MT27dvR19fH0QiEarVKsLhMCKRCLxeL3w+X9POV6u6AyclYEuB1FVqNBpafpTL5dgP+AYQCoXUjSoSiVCv12kDBhKOaDTg1WoVCoUCCoXihg04n8+HRCKBTCajf5u40BnLD8nwv1r5pEAggEKhoJoKjKVBvFE6nQ5GoxF2ux1OpxNyuRwKhWJRr1Q+n0c2m0U4HIbX60UoFEIkEpmXmEbkV4lMZ71eh0AgAJ/Ph16vh8vlgt/vh1QqpQttNv/dOHw+n9oVi8UCtVpNw7mktzvJ0WpWmqKZydUgteFyuZzKcR46dAjj4+M4ffo0i50uA9FoFD/+8Y/h8XgwNzeHRCKBcrmMYrEIoVAIhUIBm82GP/3TP8WWLVtu6G+IRCJ0dHTAaDRi3bp16OnpQTKZRCAQYGN4C6jX67SS40qTPJnASO0+49oQZUi73Y5PfvKTaGtrw+DgIDQaDb2GRKCF/H+tVsPx48dx9OhRjI2N4dSpU9SD2AgJGcZiMeTzeZRKJdqdrKurC2azGZVKBR999BESiQS8Xi8rjb1B+Hw+FAoFZDIZBgcHce+990Kj0YDjOITDYbz++uuYmZnB7Ozsah/qVWl6A04mGbFYDJVKBYVCgUgkgvHxcUQiEbYCXQaKxSKmpqYwPj6OiYkJhEKheSIuIpEILpfrqp3CiBEmce1GIRjyPSqViu5cdDodKpUKS566hVyrKxlRDFusxp+xOAKBABKJBCqVCj09Pejq6oLBYKBVGperqxFp20AggLGxMczMzCAajS66qyM7apL307jDVqvVUKlUMJvN0Gg0KBaLzGtyEzS2DDUajfO0LgqFAmZmZjAxMdH03RGb/s41Go3o7e1Fb28vTbaKxWJUFYcZ8JtHIBBApVJBq9XSichut6O9vR1qtRpOpxMWi2VBPStxgefzeczNzSGbzcLn8yGVSsFoNMJisUCr1aKnp4cm/HAcB6vVit7eXohEIgSDQdotjo3lyqJSqTA4OAilUokDBw6s9uG0BHa7Hbt27YLL5cLAwABVGWyELJzy+TwVmnr//fcxMjKCRCJxxXhqX18fhoaGMDg4OE8umrH8CIVC2O12GI1GmEwmKBQK6rXK5XLwer2YnZ1t+tr7pjfgRGSkvb0dMpmMZnIGg8Gmv7itAomFkvIJADCZTNi0aROsVis2bdpEhVkaIf280+k0Ll68iEgkguHhYXg8HvT29mL9+vVwuVxob2+HWCyGQCAAj8eD0WhEZ2cnisUi5HI5ALAGHKuAXC5HV1cXRCIRXRwzro7JZMLu3bvhcDioYNHlcByHarVKy5DGxsYwMjKCqampqwqBtLe347777qNdy4j7nLH8CAQCmEwm2uZYJpOhXC5TZUjSQKbZaXoDTnSF+Xw+fD4fBAIB4vE4CoUCS4BaJuRyOdavXw+DwQCpVIpgMIh169ZhaGgIGo0GTqcTCoWCSnKS+F00GoXb7UY0GsXJkycRj8dpO0s+n0/7Ig8NDaFer0Mul4PP58Nms2FoaAhqtRoikQi5XA6xWAyFQgFutxuxWOya7l/G9XN59yuycFMqldBqtdDpdDT2ylgclUqFrq4uWiq2GMlkEiMjIwiHw9Rtnkgk5i1SRSIRdDodpFIpLUMjVTYGg4Hp1N8iJBIJ/a1v27YNXV1d89Q/ye+/VeaepjfgIpGIJoiMjIygXC7D5/Mhk8kwl+syodFo8MADD6BcLuPuu+9GOp2GxWKB3W6nMT+STMhxHBKJBMLhMEZGRnDgwAFEIhGcPn2adnzjOA5erxcXLlzA1q1bsWPHDpTLZdhsNsjlcvT396O3txepVAr79u1DJpPBzMwM4vE4Xn75ZZr5ydTBlofL21eS+4aUKRUKBdjtdjgcDgSDQWbAr4LJZMKuXbug0WiuuDv2+Xx46aWX4PV6cerUKUSjUapwSJBKpejt7YXJZMInP/lJbN26lcoX8/l82gWL5YgsLwqFAuvWrYPD4cATTzyBoaEhGjasVqu0nLJVNodNb8DJikkkElFB/2Ytqm9VyIQhEoloZzi1Wk09HwKBAPV6HalUiorteDwezMzMIBAIIJFI0HEhkCQbkmnO4/GoxCrRvSeNTiQSCYrFIu0+FwgEkEwmEYvFVvGqtDakmQkpX8pms3SMCSS5kJQVyuVyltB2DUhS52LXKZ/PI5fLIRKJIBQKzUtWUyqVtLGMVCqFSqWiBtxms9H7juy8LzfepF48Go0im82y7oo3iEQigclkgsVigUqloh3iOI5DPp9HMBhEJBJhBny50Ov12LJlC3K5HH72s5/B7/cjkUis9mGtKYgBBy79wEkDDDJJkR/38ePH4fP58P777+PkyZPIZrO0k1WhUJj3naQpitvtxs9//nO4XC4YjcZ5MUOpVAqz2Yx6vQ6r1YpCoYBMJgOn04mTJ0/igw8+YJPUDZLL5eDxeCAUCjE6OopyuUzL+BoXv0R0RK/Xw2w2s77TS4AY18s3EW63G6dPn8bo6CiGh4eRSqUgEolo6WRHRwfMZjN6enqg0WiobKdGo6GL5ctb95IytHPnzuHMmTM4deoULl68iEKhwDwlN4BOp8OePXvgcrnoXERq6qenp/Gzn/0MXq+36bPPCU1rwMlNIpVKodVqAVxa4SaTSeZavUnI7qxUKqFcLlP3OPkv8KvyF9LZKpPJwO/3w+PxwO12Y3p6GpVK5YqTCEnkIa34+Hw+EokEMpkM3YGTMW4UliHxqct12RnXB8k/IPdMIpGAzWZbYHQad+GkaxnjyjS2J77chU7a6uZyOdRqNfB4PLrLs1qtcLlcsNls6O3thUajQUdHB00evHxcSDlmo7b93NwcwuEwrRdnLB2yOJLL5TCbzbRhCQA6j5F21KFQqKnFWxppWgNOGqtbrVaYTCbafawVUvubmXq9jgsXLiCfz2PDhg0QCATQ6/VYt24dJBIJra+PRCIIBoP0R53L5ahOfTAYXLILL5PJ4Pz585idnaWNNUg7WIlEAqVSCaVSiZ6eHnAch8nJSRw9ehRzc3MsTHITEAMej8dx9uxZpFIp2Gy2Bc1nGNcHSVAjXd5IFQVwqeR1/fr1tHyS4zh0d3dDo9FAq9VCrVZDLpdDo9FQWWFgofEmyVSFQgEXLlxAJBLB22+/jaNHjyKRSLSMe7eZIBnnmzZtwtDQECwWC5RKJer1OsbHx3Hx4kWcPHkSp06dQiaTWSCy06w0pQHn8XiQyWRUB1ilUkEikSCTySCRSDDX0U3AcRyCwSAymQyEQiF6e3tRKBTQ0dEBoVCISCSCqakpzM7OYmJiAtFolHaFy+Vy1z15FItF+P1+CIVCJJNJyGQy7N69G9lsFgqFAnq9Hnq9Hm1tbRAIBLShQyKRYAb8JmisRfb7/QDAFr7LQKFQQCAQAMdxCzwaSqUSdrudzlkikQgbN26EXq9f8D2XJxQ2fk+tVkOxWEQ6ncbk5CTm5uZw/vx5nD9//haf3dpFpVLB4XDQh8FgAHDpWgeDQVy4cAETExPweDwtZV+a0oA3UiwWEYlEaJlRuVxmcdGbhIimTE1N4cCBA1CpVDh9+jTEYjEtA0skEohGo8jlcsjlcjfdyJ60ga1UKhgbG0MqlYJYLIZcLodUKsWJEyfA4/Fw/PhxxGIxZmyWiUKhgMnJSaTTaSSTydU+nJYnEong6NGj6OzsRG9vL1QqFX1NIpFAp9NRiU7Sye9aEOW1SCRCEzgnJiaQSqUwMjJCn2fcOFqtFn19fVRxrV6vI5vNolAo4OLFizh27FhTNy25Ek1vwAuFAoLB4LzYDzPgNwdpWpLJZDAxMQFg8R3BYruDG4XcMACQSCRoF7rLe46z+u/lhbhhWfLn8hAKhfD+++8jEongwQcfhMVioa/JZDKa1bxUiHQqkTM+fvw4PB4P3n//fWQyGUQiEVZ1swyYTCYMDg7C5XJRA55IJJBMJnHu3Dm8++67NLehlWhaA06yLwuFAmKxGJLJJG3MwH7MywO5xqvxd9kYrhwkGWpqagonTpyA0WiE1WqlbXsTiQTi8Tji8XhLuQ9Xg3K5jGQyiVAohDNnziCZTMJgMECpVEImky3ajawxGa1UKiGbzaJUKiEej6NcLiOfz6NcLmNiYoIqGja2GWUL2puH5BWQ2DZpGRoKhahtabXdN9CkBpwYFtLWbXR0FJFIBPl8nv2YGYwboFQq4Sc/+QlOnTqF+++/H4899hgqlQreeustfOELX8DWrVvh8XhYT/BrQHSyU6kUvvOd70Cv19NOVk899RS++c1v4plnnlnwuUwmg1QqhUAggPHxcYTDYbz77ruIx+NUoY3k+NRqNaoGxua75eHixYvwer1QKBTUtpw8eRLj4+OYmZlBpVJpyU1FUxpw4FJyQblcRjabRSgUQjwebzn3BoPxwx/+cLUPAQBoz3cA8Hg88Hg8KJfLCIfDAEDjgeweuzqk4QUR/SD19vl8HsCl8NDc3Ny8kBTHcYjH40ilUggGg/B4PLQveDwepyptpOzvdudW3DOVSgXxeBzhcBgejwf1eh2BQAChUAi5XK4ljTfQxAY8k8mgUCggnU7D7XajWq0ilUqt9mExGC0JybYlCYqHDh0CAKp25/P5mLrXEiDewWKxCK/XC6FQiHg8TmuKf/zjH+Po0aMLPne5C71cLtPdNzHy7NrfOuLxON58800oFAr8/Oc/R71epwuwVhFtWYymNeDkB09qWRkMxs1BdnepVApTU1PzXmNZ/9dHrVaj16xxYzE1NbXg2jJWn1KpBI/HAwAYGxtb5aNZPpjsEoPBYDAYLQgz4AwGg8FgtCDMgDMYDAaD0YIwA85gMBgMRgvCDDiDwWDchszNza32ITBukusy4D/4wQ8AAMePH6elD5c//vmf/xkA8Pjjj+PFF1/Em2++iQMHDuDee+9Fe3s7fd/MzAwA4Ktf/eqC7/jt3/5tSCQScByHw4cPAwCeeeYZHDhwYNFHOp0Gx3E4ePAgAODgwYNXPL4befzmb/4mrFbrsl30W8HtOjY7duzAli1bVuGKXx+327iQe6aZ62vX8j1Tq9UglUrpGDQ+vvvd7wJA00vrrsVxufzx3HPPAQBmZmbmPb9Ulr2M7OWXX0ZXVxdeeeWVeZKCzz777KLvJ1rcjVy8eBEdHR0AgK6uLgCASCTC/fffv9yHuyQKhcKaqEFfa2PDcRzcbndLGPCrsdbGBWD3TCOrMTZ8Ph+bN2/G8ePHUS6XaZ06ANqdzmQy3bK/f6tp1XFZbpbdhU6a3DeuIo4ePUpXOJfz2muvwefz0X8fO3YMR48excMPPwwAMJvN2LdvH77//e8v2pEnEolc9Xjy+TzGxsYQjUaveexElaoRt9uNt99+G9u3b7/m55udVh6bxb7ru9/9LiKRCB566KFrfr6ZaeVxYffMfJppbJ588knUajX86Ec/os8Vi0X8y7/8CwYHB1u6N3wrj8tyckM78H/6p3/Cm2++ueD5L3/5y3j00Ufxyiuv4BOf+AQeeeQRzMzM4Hvf+x4GBwcX1Vnu6enB3r178Qd/8AcolUp4/vnnYTAY8JWvfIW+5zvf+Q727t2LoaEh/O7v/i66uroQCoVw+PBheL1enDlz5orHeuzYMezfvx/PPvssvva1r131vIaGhnDfffdh8+bN0Ol0mJiYwD/+4z+iUqngG9/4xtIv0CqyVsemvb0dTz75JIaGhiCVSvHBBx/gpZdewubNm/F7v/d7S79Aq8RaHRd2z8ynmcbm937v9/AP//AP+OIXv4iLFy+ira0N//zP/4zZ2Vn87Gc/W/oFWiXW6rikUil8+9vfBgB8+OGHAIAXXngBWq0WWq0WX/rSl5ZyeS7BXQc/+MEPOABXfHg8Hq5er3Nf//rXufb2dk4ikXBbtmzhXn/9de7pp5/m2tvb6XfNzMxwALjnnnuO+9a3vsW5XC5OIpFwd911F3fmzJkFf3tqaop76qmnOKvVyolEIs7hcHCPPvoo9/LLL9P3HDx4kAPAHTx4cMFzzz777DXP79lnn+W2b9/O6XQ6TigUcna7nfvMZz7DnT179nou06qw1sfm85//PDc4OMipVCpOJBJxPT093J/8yZ9w6XT6Zi7bLWetjwu7Zy7RjGPDcRwXCoW4p59+mtPr9ZxEIuF27drFvfnmmzd6yVaEtT4u5JgWezQe+1LgcVwTZ5kwGAwGg8FYFFZGxmAwGAxGC8IMOIPBYDAYLQgz4AwGg8FgtCDMgDMYDAaD0YIwA85gMBgMRgvCDDiDwWAwGC3IdQm5NErW3Y40c8UdGxs2Ns1Ks44NG5fmHBeAjc1Sx4btwBkMBoPBaEGYAWcwGAwGowVhBpzBYDAYjBaEGXAGg8FgMFoQZsAZDAaDwWhBmAFnMBgMBqMFYQacwWAwGIwW5LrqwBkMRmvA4/HA4/EgEAggkUjA5/MhEAjA4/EgEokgFosX/Vy9Xke9XkelUkGhUECtVkOpVGrqmuFWh8/ng8/nQy6XQ6lUArhUB1yr1ZBOp1Eul8FxHBsDxgKYAWcw1iAikQgKhQIajQYbN26ERqOBWq2GTCZDZ2cnent7wecvdMCl02lks1lMT0/j6NGjiMfjGBkZQTabXYWzWPvweDxoNBrI5XLs378fjzzyCACgXC4jHo/jX/7lXzAxMYFisYhSqbTKR8toNprWgJNVKXDpR85xHOr1OluJNjl8Pp/u/hppHD/GrYXssokBb29vh8FggMFggFKpxPr167F161Y6Ro1jFYvFkEwmoVKp4Pf7IRKJMDk5iXw+j3q9vlqntGbh8XiQSCR0YXXnnXeCx+OhWCwiGAziF7/4BTweDyqVymofKqMJaUoDzufzsX37dqxfvx5qtRpmsxmJRALvvfceotEoYrEYstksNQqM1YXH40EoFEIkEmFwcBA2mw0GgwEmkwnVahXlchnpdBqHDh2Cz+dji7BbiMlkgl6vR1dXF/bs2QOtVou+vj7I5XLIZDKIxWIYDIZ5n+E4jhpxuVwOPp+PgYEBSCQSOl5+vx9utxuxWGw1TmvNwuPxoFQqYTabodPpoFarwefzoVAowHEc+vv7Ua1WcfHiRczOzq724TKajKY14OvXr8fjjz8Oq9WKvr4+eDwexGIxTE1NoVQq0fgcY/UhBlwqlWJwcBBDQ0Po7OxEX18fqtUq0uk0gsEgRkdHEQgE2E78FsHj8aDX69HR0YGdO3fiySefhFqthlarhVC48FYnY0A8XAAglUohkUigVqvR3t4Or9eLubk5aLVaJBIJZsCXGWLAdTodNBoNFAoFBAIBgEvj09XVhXK5jEgkwgw4YwFNacABQKFQQK/XQ6VSQSgUQqvV4o477kB7ezsUCgVmZmaQSCQQjUaZQVglJBIJVCoVlEol+vr6oNVqsW3bNnR1dcFisUCr1aJWq0EqlUIgEGDHjh1QqVTweDwIhUKoVCooFourfRotD5/Ph1KphFQqxYYNG7Bt2zb09vZCpVJBKpXSUBS5T9LpNFKpFKrVKorFIvh8PnQ6HaRSKaRSKWQyGf1eqVQKh8OBer2OM2fOrOZprjlIqKOtrQ2Dg4OwWq3zwhkCgQAqlYqODePWIBKJIBQKwePxaAhQIBBAIBDAZDLRxEIAqFQqyOVy4DgOIpEIfD4fsVgMqVQKtVptxUMdTWnAeTwetFotnE4nlEolZDIZZDIZfuu3fgu5XA56vR4nTpzA2NgYnYiq1epqH/Zth0qlQkdHB1wuF5588kk4HA50dHTAaDTSGwG4tJOw2Wx44okn4Pf78Ytf/AIffvghMpkMy3BeBoRCIcxmMwwGAx588EH8xm/8BsRiMWQy2TyDUK1WUavV4PV6MTY2hnw+j2g0CqFQiA0bNsBgMMDhcNDPCQQCGjM3mUz48MMPV/Es1x58Ph8SiQSbN2/GvffeC6fTucCAm0wmFItFqFSqVTzStY1UKoVcLodAIIBIJIJAIIBMJoNUKsWOHTvQ0dFB35vNZuHxeFCv16FWqyEUCnH69GmMjo6iWCyiWq2u6HzWlAYcAJ1AGpOiyMRit9vR1dWFQqGAeDyOUqmEXC6HWq1GdxmlUoklftwiyEpVqVTC4XDA4XDAbDbDaDRCqVTOK1EiP2axWAy9Xg+O42A2m2EymcDj8RCPx1ko5CbhOA7VahWlUgmlUgnlchnlchnZbJbuCur1OvL5PCqVCqanpzE5OYlisYh4PA6JRAKTyQSO46DT6eZ9N5/Ph1gspjsUxvIgEAigUChoiEOn00Eul4PH49GckXQ6jVAohFAohFwut9qH3NKQpGjyeybeDbFYDK1WS40x2Y1LJBJIJBJ0dnbC6XTS78nlchCLxajX61AqlRAIBEin0/T+SiaT1LNVq9WQz+dRLpdvmZe4aQ34lRCLxdi3bx927tyJc+fOYcOGDUgkEpiamkKxWEQ+n0epVILX60UwGFztw12TkB93b28vPv7xj8NsNqO/vx8qlQoSiWTRz4jFYvT09MDlciGZTEIoFGJkZAQej4cZ8JukWq0iEAggFovh7NmzcDgcKBQKyGQyNJu5WCwiFAohm80iHA7D7/ejVquhVqtBoVAgkUjA5XJBLpejvb19tU9pzSOVStHb2wuz2Yx169ahp6eH5ilEIhG8//77CAQCOHDgAObm5hAKhVb5iFsbiUQCuVwOhUIBm80GtVqNnTt3wmq1wuFwwGKxQCwW0yRO8iA5IYR6vY5yuQwANFdh//79SCQSSCQS8Pl8yGazmJqaQjqdxtmzZ+H3+1Eul29JuLDlDDiPx4NOp4NOp0MymUQ0GoVKpUKhUEChUEAul0OpVEI2m6XudbYTXx6IR0ShUECpVMJoNMJut9M4kVwuv+JniQdFIpHAaDTCarXC5/NBKBSiWq2yaoKbgHicqtUqotEofD4fCoUCEokECoUC/H4/8vk8gsEgMpkM4vE4IpEI/bxKpUI2m0WhUFgQiuI4DuVyGZVKhYU6lhGS12MwGKBSqWjeAQCUSiUEg0H4/X54vV74/X4UCoVVPNrWotF7S8SLlEol1Go1lEolLBYLdDodOjs74XA44HQ6YbPZ5hnwRki1E9loXH4fiEQiaDQa6kVJp9OoVCpIJBKYnZ1FNBq9ZZuUljPgjZB4XbFYxO7du6kbsVKp4OzZsxgfH8fU1BQOHz7MjPhNIhaL0dnZCa1WS0v8XC4X1q9fD6lUekVlr8vh8Xhob2+HRCJBuVzGkSNHkE6nkUgk2BjdJPV6HSdPnoTf76f3AnHnVatVFAoFVCqVBYIgMpkMe/bswbZt29DZ2TnvtXw+jwsXLsDj8SAej6/k6axplEoldu7cSRM+G8lkMjh37hx8Ph9isRjy+TzL8bkO5HI5+vv7oVar0dbWBp1OB4PBAIvFAplMBovFAqlUCovFQnflMplsXt7O5YTDYXi9XnpfNRrxSqWCSqUCrVaL7u5ucByH3t5eZDIZ6hEOh8O3JAzStAZ8sQt5+cpHq9VCq9XS95NYYLVapfEljuNw7NgxZhxuEpJQY7PZsH37dtx1111QKpUwGAzzktUa/38xSKmTXC7H5OQkjYunUqkVO5e1CsdxmJubw9zc3HV9TiwWo6urC0NDQwuyncvlMgKBAObm5lgcdhmRSqXo7OxET08P1Gr1vNeKxSL8fj98Ph8ymQx12TKWhkQigdPphNlsxtDQEBwOB3WVy2Qy6HQ66v5ejMvnMVK54fF4UC6XaRb65UilUphMJkgkEohEIuTzebz77ru4ePEiMpnMvHLN5aKpDDifz4darYZcLqfxVD6fj3K5jGQyiQ8++ACpVApqtRpSqZRqOjcaexLXO3XqFEZHRzE9Pc1irDeBTCaj4iD79u1DZ2cnBgYGoNFowHEc4vE4rVMtFAqYnZ2dVyusVqvR2dkJpVKJzs5OqFQqiEQi8Hg8GAwGtLe3QyqVIhwOM6nIFUKhUEClUsFisWD79u2w2Wzo7u6GVCqlyWqlUgn5fB6hUAijo6OYnJxEMplc7UO/LZBKpbDb7ajVajRvgXFlhEIhBAIBLBYL+vr6YDKZcPfdd8NgMNB4N3nw+XxUq1Xk83nMzMwgnU7T7ykWi3TBFIvF5mWUh8NhBAIBKkxFQn4kwZrP52PXrl0YGhqiJWikr8CtLHNuKgMuEAhgMBjozprEhcrlMrxeL1588UXMzMygvb0dOp2OTkSX79ar1SrOnj1LS2WY++nGkcvl6OrqgtPpxMMPP4wNGzZALBZDJBIhlUohGAwilUrh7NmziEQieOedd3Dx4kUAl1avbW1tePDBB2G326HT6Wjmp1gshtFoRE9PD6RSKS5cuIBMJrPKZ3t7oFKp4HQ6sXXrVvzhH/4hLBYLNBoNxGIx3XGQSczv9+PMmTMYHx9nMfAVQiqVwuVyAQDGxsZW+WiaH5I13t3djY9//OOw2+246667oNPp5sk683g8GkpKpVI4duwYPB4P/Z5EIgGv14t0Oo3x8XHqcSKeXZIHcvl9QBYQQqEQjz/+OJ0fSUMaUh11S879lnzrDSIUCmGxWKgIiEAgQK1Wo6uebDZLSyvy+Tyt37ucer2OaDSKQqHA3E83iEwmg1wuh9PpxKZNm2C326HX66nHo1qtIhaLYWRkBIlEAqOjo0gmk4jFYvNcrZlMBrFYDFKpFJVKZUmhEcbyw+PxqOequ7sbg4OD6Ovrg0ajoWIvJJSRyWQQDocxNTWFqampK7oMGbcGol6YTqfZ5uMK8Hg86jEiyWj9/f1ob2+HXq+n4lHlcpnaj3w+j3w+D7/fj2QyibGxMQQCAfqd2WwWkUgEuVyOxq7J754YYuBXLnZSXmmz2WA2m9HZ2QmJRIJ6vY5AIIB0Oo1IJEKTSdd8GZlUKsW2bdswMDCA7u5uSCQSFAoFFItF5HI5RCIR+P1+hEIhmuZ/paQDkljAdLdvDJPJhK6uLmzatAlf+MIXYDKZIJfLIRQKab3x+fPn8Q//8A+Ix+PweDx0rBonHbFYTFez+Xx+3t9gGc4rh0AgQHd3N5xOJ+666y48+OCDUKlUMJlMEAqF4PP5qNfrmJqawujoKMbHx/HOO+/QSg/GypHP5zE1NQWPx7PgnmFcguTkqNVqfOpTn8LDDz8MtVoNi8VCd+T1ep121wuFQvB6vfB6vTh06BDi8fgCFzrJNK/X6wsEWS6fn4h6oVKpxKOPPop77rkHTqcTer0eqVQKR44cgc/nw+nTpzExMXF7ZKETcRAiHcjj8VCr1ZDL5ZDL5ehqiq1Kbz1KpRI2mw02mw1GoxE6nY4qeSWTScTjcfh8PoRCIVoDuZi3gwiJNLqRiNHOZrPIZDJUhIdx4xBXIdkVEFEKUkojFovhdDrhcrlgt9thNpvpDga4tPsolUoIhULw+Xzw+XwIBoPI5XLsfltGSIlTo2jI5WVLpNaYCPAwFiIQCGgZntVqhd1uh0wmg0KhQL1eRzabRblcpiG+QCBADbjP50MymUQikVjyAqlRpU0kEkEkEsFisUClUsFut8PpdEKr1aJer6NYLNKYOennfqtoKgMuEAig1+thsVigUCgAANFoFCdOnMDU1BRbja4gQ0ND+K//9b/SnTdx6+Xzebz22mt4++23EQgEMDMzQyebxRAKhVAqlbRJQ71ex8zMDNxuN44cOYJ33nmHrpIZ1w9JoJHJZBCJROju7obL5YLZbEZXVxdkMhmMRiPNkFWr1bRxBvFgZbNZvPPOO5ibm8PRo0dx9uxZZDIZRKNRtmBeZkgWNJn0iXgO4/qQy+W455570N/fj02bNkGn09HfczKZxNtvv41gMIizZ8/C5/MhnU4jmUyiWCzSktXrSZo1m81wOBy0flylUqGnpwc6nQ59fX3o7OxEsVikc+KhQ4cwMzNzy8XEmsqAk7gGcdUCQKFQQDgcplmBjJVBr9dj3bp1UCgUEAqF1N1dLBYxPT2N48ePI5/PI5fLLZplSW4m0qWsMc5KkkU8Hg+8Xi9raHKDEOPdqN1ssVjQ0dEBp9OJoaEhukMgu5PFlPKq1So8Hg/Gx8dx8eJFXLx4cZ5wBWP5EAqFdCzIwraxUxwJ+bHQ39URCoVwOBzo7e2FyWSiOhRE0nRmZgazs7M4c+YM3G43Vem8XogQjEqlgtVqhclkwrp166DRaLBu3Tro9XoqZFUul5FKpRCLxejcdqtLL5vKgC9GtVqlblbmTlo5isUikskkOI6DSqWijRfq9TqcTif6+/uRy+XoqjYcDqNcLkOhUEAsFsNut6OtrQ12ux133HEHDAYD5HI5UqkUzpw5g7feegsej4ctym4C0kxGp9PhrrvugsPhgMlkotn+JL6tUqmo23YxarUanXhIOIMZj1uDQCCAXC6nAiKNBjyTySCVSsHv9yMWiyGZTDL9iisgEongcrnQ29tL9fuj0ShmZmYwNzeHI0eOUHf59VYikVCUVCrF3r170dPTA6fTiY6ODsjlclrrrdPpIJFIEI1GMTU1hcnJSRw7dgyhUAh+v39Fwk8tYcDz+TyKxSIz4CsIaahAyiFI60PSjKSrqwvZbBbxeByZTIYK+pNdRV9fH3bt2gWHw4E77rgDMpmMLsbGxsbw7rvv0pwGxo0hl8vR3d2NtrY2PPnkkxgYGKCvXZ7ceTWDTGKGJFuW3We3DpKPQNq2Ngrn5HI5BINBRCIRJJNJZDIZdn9cAVL3TXT7OY5DMpnEhQsXMDMzQ13nN1LCRRqeKJVK7N27F/v27YPFYoHD4aDJ0+Rv1mo1TE1NYWRkBMPDw/jJT36CXC6HdDq9Ih6spjDgAoEAEomErkgVCgUV+0in07h48SKCwSAT+lhBIpEIzp8/j7a2NrhcLpoYBQAdHR0olUooFArI5/PIZrNob29HsViEXq+HTCZDV1cXent76c67Xq9jdHQUoVCIKhqxyenmUCqV6O/vh8vlojkjBOICJ8mCpP1ho+Y2gcTQlUolRCLRSh3+bQnJCSHSnY0Q406S24j7lrE00uk0JiYmaEXMzQioEDGWUqmETCYDmUwGlUpF+4GXy2XE43EUi0UMDw9jbGwMs7Oz1E1PjD1pKFSpVG5JMltTGHCRSAStVgu9Xg+DwUCNAI/HQzAYxLvvvksbkzBWhunpabz55pvYvHkzdu7cCblcTrMvt2/fjs2bNwO4tAotFovw+Xwol8tQq9V0MUZc7wKBANFoFG+//TaGh4cxPj5+y+oibyeMRiP2798Pp9MJg8Ew7zUiWJHL5eB2u1GpVLBu3borGnCNRkMTFhm3jsb2lZfLeZJ8ESLFuViGOuPKBINBvP/++4hGoze1AyY7cJFIREvQOI6jrUOnp6eRTqdx/vx5xGIxXLhwAZOTk3TRrNVqMTg4CKVSicnJScRisVuWjb7qBpzH40EikcBsNtPsc9KvFbiUtWm1WqFUKgFc2lkQgRaijkPq9hjLR6FQQDQahd/vx9jYGAwGA92hka5ixDgLhULo9XpUKhU6fuRBVIyISz6RSKBYLDLjvQwUCgWaR1Cr1eZpamezWdoIgzRhIAJHRqMRFotlnkJVY9kZY/kh94lWq4XL5YLVal3QACiXyyEUClF5YpaLcH2IxWJoNBqUy2VIpVIqAna9LvTGOSsUCsHtdtMs9mw2C4/Hg2w2C7/fj0QiQfOAZDIZ9Ho9jEYjOjs7oVarUavVIJfLaSb8mtJCJwbAarXi/vvvh9PpRFtbG12dchyHdevW4b//9/9OVy+VSgUXLlygiR6RSISWBrDY3fIRiUSQSqUwNzeHyclJaLVa2j+3p6cHHR0dkMlk0Gg0EIlEMJlMdJXaKLBTqVSQTCapCE8gEGAlY8vE1NQUnnvuOTo2JpOJvuZ2u3HixAmUSiV67+j1eigUCjz55JP4/Oc/TxfJxIV+eUY0Y/kgWtxbt27F5z73Oej1epp8Rbh48SJef/11zM7OUg10NqctHbvdjvvvv596A6PRKOLx+HVnn5OS2Ww2izfeeAMHDx6kIURSo1+r1ahoFQntulwu3H333XA6nXj00Ueh1Wpx/vx5+P1+HDhwAB6PZ20ZcCJqIJPJqGhI4yTCcRxtgkF22JVKha5kSKMTgUCAVCrFfuzLSLlcpl6Oer0OjUYDs9mMarUKo9GIYrFIx4lkqBMau+6QRA9ST0x+/Iybh5TLkN7sjSUrk5OTuHDhAs1i5vF4iEajtHFM40RCvGCNwi6M5YV4rrRaLex2O9Rq9YJ8g1wuh0AggEgkwu6TJUA8sCSxTC6Xw2q1ol6vw2QyzTO2ZA66nu8GcFUVQrJJIaWBOp0OLpeLPrRaLRKJBIBLC7hbkc+wqnerWCyGQqGAzWbDzp074XA4FrTW0+v12Lx5MzXO9XodXV1dyOVymJubo3rNr732GmuGcQsgjelzuRzeffddqNVqCIVCmM1m2lGskcbmARzH0fyGQqEAtVoNhUJBpXEZN0e9XqdJasPDw/MymjOZzBUNwOU1xkKhEL29vdBoNDh79uwtP+7bDR6PB61Wi7a2NpjNZtp/+vIJvVAoIB6Ps533EiiVSrhw4QJkMhnVQjcajdi5cydyuRx6enqQSqVw8uRJzM3NYWJiAiMjI8u2A+bxeDRcODg4iN7eXvT392P//v3QaDQ05HurWVUDTpI2tFoturq6YLVaF7yHZKUDvyqFaWtrA3Ap0Uqr1dKkA8byU6vVaBOAbDYLkUiEXbt2IZ/PQ6VSzXtvo/EmkLpXMmmRJgOM5aFer6NUKsHr9V73Z8n9RGQhJRLJggU04+bh8XiQy+W0Pl8ikSzq6ahUKrRklsW+r061WqWqZ3q9Hg6HA0qlEkqlErVaDW1tbSgUChAIBFCr1chkMhgdHV02rwbxOpKKm+3bt6O3t5d2a+TxeNT7dSvHclUNeK1WQ7lcRqFQQDKZpG5Y0mc6lUoB+JXiFMkMtFgsUKvV1PDH43HYbDZadsY6kC0fpP5bJpNhy5YtsFgs2Lp1K5xOJ+RyOUqlEvh8PnVThcNhqqtdLBah0WjQ09ODer0Oi8WCzs5OmiDHYKx1SHJgR0cHtm3bho6OjnkLWNIBrlAoIBQKIRKJXNV7wrhEsVjE6dOnEQ6Hae96vV6PtrY2KsTC4/HQ398Pg8EAp9OJLVu2IJPJ0DJW0qubUCqVkE6n54Vp7XY7NBoNfQ9x10ulUvT09ECv16OnpwdtbW0wGo202VMwGEQ6ncZHH32EiYkJTE5Orr1uZLVajV78eDxOJTvr9TrGx8cxMzMzL85AeoSTnQJJBEkmk3A6nQAuDSwz4MsHWWnqdDo88MAD2LBhA/r7+9HR0YFKpYJCoUDrjcvlMi5cuACPx4NUKoVUKoXOzk6YzWZ6MxC9YAZjrdOo6NXT04O9e/fCaDTOM+D1eh2xWAyxWIw2ByqVSsyFfg2KxSKOHj1KO1Zms1n09fXBarVCIpHQnI7NmzfTkFG9Xoff78d7771Hm500XudEIgGfz4dsNou5uTnUajXs3r0bnZ2d9D2kmYlKpcI999wDu90+r5qD9BYg2iXvvPMOhoeHqdDVcrOqBpyUf8ViMRw/fhxarZZebI/HQyd6sgMnKl92ux1arZYOlEQigVarpS5exs1DfqgkR8FkMsFut8NisUAul4PH46FYLCIUClEp1Vwuh5GREQQCgXku90QiAZFIRNv9KZVK2tiEuQpXDtJm8XL3ba1WQzqdRjweR6FQWKWjW3uQ+nqVSkU3GwqFYkGYqbFjHykJZPfF1eE4jrqoSalrtVql8WcSEiI19Y3zmcPhQKFQWNAy1Gg00nwdm82Ger2OgYEB2Gw2+h5ScknCgo0J14RqtYpkMknrv/P5/C3bVK6qAa9UKqhWqzh//jz+8i//cl4N6uVdkMjqhsQ5SO243W6HQqFAd3c3xGIxZmZmaOYf48aRy+XQaDSw2+248847YbfbsXPnTnR0dNAfbSwWw8mTJxEKhaiAgs/nQyKRoCveUCiErVu3wmg00laWZ86cgUQioVnpjFsPSbrRaDQLxFwqlQpt/sBCG8uHWCxGZ2cnbDYb1q1bh87OTlpmSSCGqFgsolgsolQqMU2LJcBxHEqlEkqlEk6ePIlz587BZrPh8OHDtCzZbDbDarXS37xCoYBOp8OuXbvmVck0fidZPJEQBjH+hMbdtlQqXXShRRo+zczMwO/3Ix6P3zKPyqrXjJCBiEQiS3p/o5QducgkPk7ERRg3D9EC1ul0cDgcdOcsEolook04HEYoFEIoFEIgEEA0GkU0Gp1X553NZqmUIJHIFQqFTCJyhSEG3GAwLMiA5jiOqrax5hnLB5/Ph0qlojvvKyXaVqtVeo+wnffSIdeqUCigUChAJBJR0SIip1qr1ZDL5WgDGWIfSBz7cuNMXm+spGmkse67Xq/THIfG7yGJvyQX6FbmM6y6AV8O2I9++bFardiyZQt6e3vxwAMPQKfTQS6XI5PJ4L333sOJEycQCoUwMTGBbDYLt9uNQqFwVb164rK6kQYDjJtDJBJhy5Yt2LZtG/7/9s40NrLruvP/qnq173sVq4rFfWmS6r1b3VJrt2VZ8liynYlmMrExMILAzmCCYAxjMgiifAiCAEaCAJkgMTKJt/EMkMiSEmtsLbbaaqu71Tt7YXNnkax93/flzYfOvWI12S2ym2QVyfsDCi2xXlW9erfePfece87/7Nu3r8mA1+t1KrbDQuibh0QiwcDAAIaHh2G1Wtc8huyB+/1+pFKp7T3BXUY6ncbs7CyWl5exsLAAiURCW32ubGlMasYPHjy4qgz204jFYrh27RpEIhHtAOhyuWC1Wun2b6VSQTQapduLW8mONeB3h0BY/9zNRS6Xw2w2w2azweVyQa1WI5vNolgsYnl5GePj4zTpo1Qq0ezNe0FC6iv3vdl4fcLdjSvuzpB9GEglgc1mQ3d3NwwGQ5OHQVSlmAe+uZA9cJPJtKYGPbknCoUC0uk0Kx97SKrVKq1cikQiEAgE1PEg++FE/VOtVkOtVm9Ypc3r9eLy5cvgOA7d3d2Qy+UwmUz0eSLDSkpv93w70fVAOsPE43E2AW0RpVIJly5dgt/vx8WLFzE5OYlSqYRcLkcTb+4Hz/MoFAp0v+9BNIp3KyqVCs888wwcDgf92/z8PM6fP49yufxQoVWFQoH+/n6YTCYcOHAAo6Oj0Ol0EAgEyOfz8Pv9CIVCuHTpEk1AZGw91WoVsVgM6XQa58+fx8cff7wlUpt7GTLnVCqVpu5upDqgVqttWHCFZKir1Wq6PUjmPmK8i8Ui/H4/lpeXt1ywalcY8FqthnQ6/aleIOPBKZfLuHXrFiYnJ3H9+nUsLCxs6PWkCQ0Js7NM209QKBR48skncejQIfq3X/3qV7h+/Tqt1HjQayWXy9Hf3w+n04mhoSH09fXR54rFIpaWluD1enHz5k3cunXrob8LY30Qre5oNIobN27g/PnzDzXOjLW5Xwjb7/c/8PsajUbk8/mmPW4SzSqXy4hEItuyGN4xBlwkEkGlUkGr1cJgMDS1HC2VSgiFQqxn+CYik8lgMBigUqlokk0oFMLy8vK6mpEYDAaYzWZqPGQyGebm5miyG9vy+AQSRs3lcjAajdBqtRgaGsJnPvMZpFIpLC4uolAoUJGc9SCTyaDVamG32zE2NgaXy7WqeUY2m8XExATtrsTYXkj7SfJgEamdQ71eRzweRzAYRG9vb9NzRDtDKpXSbplbxY4x4BKJBDabDWazmYrFk3KmXC6Hqakp+Hw+dhNsElqtFm63G0ajEeVyGalUCtPT07h+/fq6NOddLheOHz+O4eFhjIyMoFQq4Qc/+AEuXboEn8/HjPcKGo0G0uk0YrEYXC5Xk8JTJBLBBx98gEAggPPnz6/b0Gq1WvT396O3txfPP/883G73KpnUaDSK999/Hz6fj5WPtQBS912tVlnkcIdRrVaxuLhIa8WBT5pzkTpxlUpFM9G3irY34GTPQiaT0T66Wq0WHMehWCxSFTfWvWdzIeV65McnkUjgcDjQ29uLUCiEWCxGPfOV3jSRu9Xr9bDb7dDpdCiXy8jn88jlcsjlcmyyuotGo4FcLodUKoVqtUo79BkMBvA8D7fbDYVCgVQqBaVSSetfiQdHrj25T2QyGSwWC3p7e9HZ2QmtVkvFd2q1GnK5HLLZLILBIFKpFJPu3GSI2IdMJoNarYZWq23q1kcgSWyMnQfP82smfpKxJ+NO7tWtou0NuEgkosb7P/yH/4De3l4MDAyA4zgsLCzg4sWLuH37Nit/2WQikQhu3LiBarWKEydOQKfT4Zvf/CaSySTeffddnD17FtFoFB6Ph3oRAoEANpsNer0eBw8exJNPPgmhUIiZmRkq8sLUvlZTqVQwNzeHYrGIvr4+8DxPW+yazWY4HA5Uq1V88YtfpEk08/PzKJfLTcaXaD93dXVBrVbDbDZDoVDAYrFAIpGgVCqhUCjg2rVrOHfuHDweDxYXF5HJZNjW0yYikUhgMBjo9sWhQ4doQybG7qBeryORSEAkEtFMdqFQSBUn9+3bB6lUiitXrmzp9lTbGXCiVET+FYvFVPvc6XSis7OTZg5ms1n4fD5Eo1HmQWwypVKJNpSp1+vgOI7WO05PT9Mktng8jkqlQvvykr1vg8EAtVqNYrGIUChEpVbv1h9m3PHAM5kM5HI5stksCoUCOI6j0QyFQgGe56HX61Eul6kCVKlUQiqVotdTKBSir68PfX19UCgU0Ol0NPuWiLUUCgWEQiF4PB74/X6apcvGZPMgwlIymYzm7dwtMEV6VbPI4c6E1HuT3huNRoNGi8ViMQwGA9LpNORyOYRC4Zbl/LSVAZdIJOjp6YFWq0VnZyfsdjvK5TJyuRwtmDcajeB5HrlcDvPz8zhz5gzi8TjzIDaZYDCIUqmEYrGIAwcOwG6308XTqVOn0N/fj0gkgrm5OVoaJhKJMDAwAKvVikwmg2vXriEYDOLcuXNIJpNYWFhAoVBgE9ZdVKtVLCwsIBQKQavVIhwOo7e3F8ePH4dMJqOTgEwmg1gsprXcREN75cRAWlYS3XOS/V8oFPDzn/8cN2/exPz8PG7fvo18Po9SqcSM9yZDJnLiiNxd4w8AmUwG58+fx+LiIkKhUIvOlPGgkG0vkUiEZDKJRCIBmUwGpVIJtVqNxx57jOawZDIZ5PP5deUObZS2MuAikQgmkwl2ux2PPPIIBgYGkMlkEAqFYLFYqCQhmXii0Sjm5ua2pWB+r0G6ialUKni9XgCAw+GARCKhXl4sFkNnZycqlQptKzo0NASbzYazZ8/ixo0bmJubw7lz57bkx7tbqNfriMViEAqFuHnzJq39HhsboxmtQqEQHMdRw2wyme67ol8p1FKpVJDP53Hz5k18+OGHCIfDCAQCLJFwiyGGfC3Z4HK5jPn5eczNzVHxEcbOgUiA5/N5WkECAEqlEjKZDL29vTAYDFSCular7X4DznEcHA4Huru70d/fj8HBQZRKJXR3d0OpVEKpVKJWq+HGjRtYXFzExMQETbRik9HWkMlkcOXKFUQiEfT398NsNtPn5HI5bDYbzaYlQgbEQHi9Xra9sQF4nkc0GqUNLqrVKnQ6Hfr7+6FSqWCxWKBWqyGXy6FQKNZ8j3w+j2KxiGg0SuVtg8EgMpkMxsfHEQ6Hkcvl2P2yhZB+0vF4HB6PB0ajEWazGXq9HsViEel0GoFAAIuLi1haWtpysQ/G5kNKP+v1Oi5dugQAGB4extNPPw2O42jOw+HDh8FxHGZmZujiPJvNblrUq60MuFgsRmdnJ4aHh7Fv3z7s27ev6XmBQIBCoYDLly/j7NmzmJqa2tSLwVhNIpHA2bNn0dnZieeff77pOaVS2WRI6vU6/H4/otEo1SOOx+MsOrJOeJ5HIBBAMBjEwsICrl69CpPJhCeffBJWqxX79++Hw+GAyWRa04DzPE/bgk5MTOC9995DIpHA3Nwccrkc4vE4CoUCM95bTLVaRTKZhEgkwuzsLORyOUZHR6HX61EoFODz+bC0tIT5+XnMz8+z+WsHwvM89b7PnDmDmzdv4vOf/zyOHj1KZVqVSiVOnDiBnp4enD59mirv5fP53WnASRcZjuNgsVhgtVpRr9dRqVRQq9WQyWSQy+Xg8XgQCoWYJ7EN1Go15PN5pFIpLC0twWAwwGg0QqfT0b3ver1OFdYWFhYQjUbh9XqRSCSQy+XYBLVBiAeez+fBcRwWFxdpidny8jI0Gs0qURbyumQyiUwmg8XFRfh8PmSzWWQymTX7HzO2DrJ1MTc3R+v8ScZ/MBiE1+ulEzkbk50LCaVns1kEAgHcuHGDltByHEcrSnQ6HaxWKwQCAYLB4KY5NW1lwPP5PN5//30olUpUKhWIRCLk83nEYjFkMhncunULqVQKt2/fRjAYZKHzbaBSqSAcDqNcLuMXv/gFPB4PnnrqKRw9epRqaZPSpmw2i2vXrsHv92NhYQGzs7M0vM7YGJVKhRrjSCRCuyilUimo1epVoiwEUhtOavSJFCszFNtPNpvFv/zLv0AsFtMHUV2rVqvIZrNsTHYBuVwO+XweFy9eRKVSgdVqxVNPPQWj0QiNRgO1Wo3Ozk4cPnwYCwsLtAR0M2grA95oNJDNZlEulxEKhajEYzweRyaTgc/nQzqdRiqVor1eGVsLMQClUgnhcBhisRherxc2m412I8vn8/B6vchmswiFQgiHw8hkMqhUKnt+gvr+97//QK8jZSeNRmOV8E02m2VJgTsAMp8xdjckwpjL5RAIBGgkuVAowGg0QqlUIpPJ0JKzzZwTBfwG3m2tbMrNhpRfWK1WmEwm1Go1GkInKl7FYvGhOjQ9KO1sjLZ6bDiOo/rzer0eer2+qbMY+ZcswEql0pb3wl1Ju47N97//ffzn//yfW30aLaVdx2Y75rN2pl3HBdiZYyORSCCXyyGVSmEwGCCRSKBSqSCVSpFOp5FIJGj11Hq6N66HtvLAgU86ugQCAQQCgVafDuPfqNVqiEQiAIClpaUWnw2DwWC0F0SYBwCdK+VyOcRi8ZZJqradAWcwGAwGYzdQrVZp3sNWwAw4g8FgMBhbwIMm8DYajVXyu2vx6UcwGAwGg8HYNpLJ5LqOYwacwWAwGIw2Yr0dG5kBZzAYjD3I8vJyq0+BcQ9Ix81PY0MG/Hvf+x4A4NKlS7RO9e7Hj370IwDAyy+/jB/+8Id455138P777+OZZ56B2+2mx3k8HgDAH//xH696j9/+7d+GVCoFz/M4f/48AOBb3/oW3n///TUfmUwGPM/j9OnTAIDTp0/f8/zW+/j6178OoVAIv9+/Za3gNpO9Mjbj4+PQ6XQ4cuQIFcJo97EBdue4lMtlcByH//E//seq5/7rf/2vEAqFbd0lcDffM/V6HTKZDDabbdVzf/u3fwtg/WHaVrCbx2blY2lpCR9++CEWFxfB8zxOnDgBs9kMnU63ruu06Ulsr7/+Onp6evDGG2801fK99tprax4/Ozu76m8zMzPo6uoCAPT09AC4o5P+3HPPbfbprkm5XMZPfvITPPXUU+jo6NiWz9wOdvrYzM/P43Of+xwsFgt+9rOfrXuV2u7s1HEhOvdrZdhWq1WqCreT2aljIxQKceDAAVy6dAmVSgUSiYQ+R8pzVzYm2ons1LFZSWdnJzo7OwEAqVQKV65cwZe//OV1v37TQ+gikQhAcyH6hQsX6Ormbt566y34/X76/xcvXsSFCxfwwgsvAAAsFgueeuopfPe730UwGFz1+mg0et/zKRQKmJqaQiwWW/d3+NnPfoZUKoXf+q3fWvdrdgI7eWxCoRA++9nPQigU4t13393xk89Kduq4WCwW6HQ6vPnmm7T+FbijSPXTn/4UQ0NDkMvl932Pdmenjg0A/OZv/ibq9Tp+8IMf0L+VSiX8+Mc/xr59+3a8c7KTx2Yt/vAP/xC1Wg1/8Ad/sO7XPJAH/o//+I945513Vv3993//9/HSSy/hjTfewCuvvIIXX3wRHo8Hf/d3f4d9+/bRnqkr6evrw+OPP45vfOMbKJfL+Ku/+isYjUZ8+9vfpsf8zd/8DR5//HGMjY3hd37nd9DT04NwOIzz58/D5/Ph+vXr9zzXixcv4umnn8Zrr72GP/mTP1nX9/vxj38MqVS6oZVQu7Bbx+Zzn/scFhYW8O1vfxsfffQRPvroI/qc1WrFZz7zmXVcndaxG8dFJBLhW9/6Fv7oj/4Ijz76KL761a+iXq/jH/7hH+Dz+fC///f/3thFahG7cWwA4Hd/93fxv/7X/8Lv/d7vYWZmBp2dnfjRj36EpaUl/PSnP13/BWohu3Vs/vzP/xy3bt3C8ePHwXEc3nrrLbz33nv40z/9Uxw9enT9F4jfAN/73vd4APd8eL1evtFo8H/2Z3/Gu91uXiqV8gcPHuTffvtt/mtf+xrvdrvpe3k8Hh4A/53vfIf/i7/4C97lcvFSqZQ/deoUf/369VWfPT8/z3/1q1/lbTYbLxaLeYfDwb/00kv866+/To85ffo0D4A/ffr0qr+99tpr6/qO6XSal8lk/Je+9KWNXJqWs9vH5n7f7cknn3yIK7e17PZx4Xme//GPf8wfO3aM1+l0vFwu548fP970Ge3KXhibcDjMf+1rX+MNBgMvlUr548eP8++8886DXrJtY7ePzdtvv80fO3aMV6vVvEKh4B999FH+n/7pnzZ8nTakhc5gMBgMBqM9YGVkDAaDwWDsQJgBZzAYDAZjB8IMOIPBYDAYOxBmwBkMBoPB2IEwA85gMBgMxg6EGXAGg8FgMHYgzIAzGAwGg7ED2ZAS20q92b1IO5fMs7FhY9OutOvYsHFpz3EB2Nisd2yYB85gMBgMxg6EGXAGg8FgMHYgzIAzGAwGg7EDYQacwWAwGIwdCDPgDAaDwWDsQJgBZzAYDAZjB8IMOIPBYDAYO5AN1YHvNmQyGRQKBYRCISQSCQAgn8+jVquhUqmgWq22+Ax3BgKBABzHQSgUQq1WQyqVgud58DwPgUAAkUgEAKhWq6jX66hWq6hWq6jVaiiXyy0++90Fx3EQiUQQCoUQiUT0+guFQkilUnAch1qthmq1ikqlglwuh0aj0erTZjAYD8CeNOBCoRBCoRD79+/HyZMnYTAYMDg4iHK5jA8++AA+nw8zMzNYWlpq9anuCCQSCex2O7RaLb74xS9i//79qFQqqFQqkMlk0Ov1qNfrWFpaQiaTwfz8PHw+H/x+P27duoVardbqr7ArEAgEsFgsMJlMUKvVMJlMkMlkMBgMUCqVGBsbg8PhgM/ng8fjwfz8PP7lX/4F2Wy21afOYDAegD1pwImXYrFYsG/fPthsNhw+fBilUglerxeNRgOBQKDVp9n2CAQCCAQCSCQSaLVamM1mPPLIIzh16hTK5TKKxSLkcjlsNhvq9TomJycRj8fBcRwEAgFKpRKEQraLsxkIBAIIhUKoVCqYzWbo9Xo4HA4oFArY7Xao1WqcOHECvb29mJ2dhUajAQAoFAoUi0X6Po1Gg3nkjD0PUYITCoVNqnA8z6Ner7fqtFaxpww4CSOeOnUK/f392L9/P44ePQqVSgW1Wg2RSAS73Y5SqYTp6elWn25bIxAIYLPZMDQ0BJPJhGPHjsFsNmN4eBhKpRJSqRRyuZyGbHmeh9VqhVarhUKhQF9fH86fP4/p6Wnkcjlks9m2ujF2AiKRCFKpFBKJBDabDRqNBidPnsSBAwegVCqh1+shFouhVCohkUhgNBoBACaTCSMjI9DpdOA4DtlsFul0GqVSCdevX8fs7Czb3mDsWSQSCSwWC5RKJQ4ePAin04lisYhCoYBgMIiPP/6YbrW2Wo52TxrwY8eO4amnnkJnZyd6enroCovneZhMJpRKJajV6hafbftCPG+z2YyjR4/C4XDgs5/9LMxmMzUWwCfeXKVSAQAYjUb6ulqthlwuR/9WKBSYAd8gQqEQMpkMSqUSvb29sFgsOHXqFJ566ilIJBLI5XIIBIKmSYbneeh0Ouh0OnR0dKCrqwvFYhF+vx/pdBqVSgV+vx+lUgmVSqXlExSDsd2IxWJYrVaYTCZ89rOfxZEjR5BMJpFIJHDz5k1MTEygUqmg0Wi0fM7aMwZcqVSiv78fZrMZvb29sFqtUKvVdIKr1+s0qSeVSjWFFRnNmEwmGAwGDA8PY2xsDBaLBRqNBlKplIbEA4EA/H4/stks/H4/RCIRhoeHYTAYoFKpoFQqYTabsX//fgSDQWQyGWroGfdHIpFAKpXCYDBgaGgIOp0Oo6OjsFgscLlckEgkdJsCwCojTuA4DkqlEmKxGPV6HWq1Gg6HAx0dHUgkEsjn88yAM/YcIpEIarWaLnR1Oh3EYjEUCgXC4TAkEglNzG01e8aA63Q6vPDCC+ju7sbx48fR19dHjQ3xEguFAsLhMPx+P3K5XIvPuD0RCoXo6enB6Ogojh49ihdffJGGyolnzvM8ZmZm8POf/xzLy8s4c+YMZDIZvv71r2NsbAz79u2DyWRCb28vXnrpJczNzWFqagrpdLrVX6/tEQgEUCgUMBgMGBgYwG/8xm/AZrNhdHQUBoMBYrEYHLe+21osFsNoNILneVgsFlQqFUxNTcHn82FxcRHBYJDthzP2HGKxGGazGR0dHejo6IDdbgfP82g0GkilUpDL5RCLxRAKhcwD3ypIprlUKoVarYbdbqeDoVKpIBKJ6KDk83kEAgHE43EEg0FEIhHmgd+DuxOlZDIZLRtrNBqIRqPIZrNYWlqCz+dDKBRCOp1GtVpFuVymoSfgjgcol8shk8lYMtunIBAIIJfLIZFI4HQ60dXVhe7ubtjtdhiNRqhUKsjl8qbj1/Oe5F9y/Y1GI9xuN91GKpVKKJfLzJAz9hTEfpAHcMczJyWZ7cKuNeAymQwymQw9PT04evQoOjs78dxzz8FqtUKhUAAADZvPz8/jxz/+MUKhEK5evYpoNIpCodDib9B+cBwHsViMrq4uek2Jt8fzPIrFIt5++21cuXIF09PTuHnzJsrlMsrlMhQKBer1Omq1GjUGpH78bu+dsRqO49DT0wO73Y5nnnkGL730EhQKBU1Ek8lkD/0ZIpEIhw8fRnd3N86dO4dYLIZYLIalpSW2oGUw2pBda8DFYjHkcjn0ej26urrQ0dEBk8kEnU5Hk6tKpRLy+Tzi8TgNGYbDYaRSqVaffttBysWkUik0Gg31+gQCARqNBsrlMgqFAgKBAGZnZ+H1ehGPx6lB5nmeeuEk7CQUCmlJHxEdYQa8GZKoJpVKaVivq6sLAwMDW7IPp9FoIJPJYLfbYbVaAQB+v3/TP6fduPu3R7wssrBc+bi7tIjwoN4ZET0iUayVf6/X603/svuDsZJdacAFAgHcbjd6e3tx4MABPPfcc9BqtVCpVOB5nibo3Lp1C+Pj41heXsaNGzeQyWSY530PZDIZHnnkEdjtdhw8eBADAwOQy+UQCoXIZrO4cuUKwuEwLl++jNu3byOXyzVNNpVKBZOTk8jn83A6nRgZGYFCoYDT6UQul6OlZ5VKpeX7Su2EyWTCiy++iI6ODoyOjsLhcMDhcGxZGI8kwO3fvx/f/OY3MTMzA5/Pt6vFXmQyGcRiMVUIFIvFUKvVEIvFdEGjVCqhUCigUqlgtVqb6oTFYjFEIhGsVivsdvuGFlb1eh25XA7lchnJZBKxWIwa8Wq1SsWPgsEgYrEYVYlkMIBdasCFQiEMBgN6enrQ19eH4eFhGmJsNBooFApIJpOYn5/H2bNnEY1G4fP5WN3rfeA4Di6XC93d3ejs7GyaxEqlEjweD5aXl7G8vIxgMLjq9bVaDeFwGI1GgyarSaVS6PV6aLVaajiYfG0zKpUKR44cQX9/P4aHh2G32zf8Hhvx2kg0hCwUdDodVCrVhj9zp7AyskSiSRKJBGq1GjKZDBaLBXK5HEajEWq1GhaLBd3d3U1GmiRx9vb2Yt++fU3e+coS1bVoNBqIx+PI5XIIBAJYXFykzxWLRYjFYsRiMZRKJWQyGQB3DDvzxLefdrzmLTXgFosFHR0dNCxbrVYxMzODVCqFSqWyYYlNsVgMl8sFjUaDRx99FCdPnoTT6aT7tAKBALVaDdPT05iZmcH169exuLjIREQ+BTLJORwO9Pb2QqfTAbhjuLPZLILBIObn57G4uHjPTHIyNl1dXVRQhAgjhMNhFAqFpgS3vYpMJoNEIoFGo4HFYoHb7UZXVxfsdjvN3VgP1WoVpVIJ1WoVuVwOtVqNhnjVajW0Wu0Wfoudg1gsxvHjxzEwMEATliQSCVQqFcRiMbRaLRUlkslkUKlUMBgMqNfrSKfTdI5qNBrw+/0IhUL0vdcKs69lBBqNBg2f63Q6Oh82Gg0YjUbkcjn4/X5EIhF4vV6Mj4+jUCggFouxBe8WwHEc9Ho9jEYjpFIpAFBlyWw2S21TOxj0lhrwjo4OPProo9BqtVTtplQqUZGPjRpwiUSCwcFBdHZ24qmnnsKzzz5L96vIzVSv13Hz5k2cOXMG8/PzmJmZYXtL94Hs+UmlUrjdbgwNDcFgMAC4Y4BDoRCWl5cxOTmJxcVF6iXcjVgsRl9fH0ZHR2EymejrfT4fgsEg8vk8i4Dgjjen0Wjgcrlw4MABuFwu9Pf3w2az0clkPVQqFaTTaRQKBfj9fpTLZdrMxOFwQKPRrCtTfbcjkUjw9NNP43Of+xxUKhU0Gg1EIhHEYjFduBJve2XYvFAoYHFxEfl8HsViEZVKBQsLC5ienm6aSz7NAyfRQqVSiY6ODuqAuN1uSCQSulWSTqeRzWbx8ccfo1QqIRqNIpPJMAO+BZDySovFQiO3xWIRiUSCih3tWQMuEAigUqkglUrR2dmJoaEhqFQqWCwW5PN5dHV1odFoYHl5ed370SQUq9Pp0N/fT728lWGuarWKVCqFdDqNcDiMSCRC92nbYSDaFWK8ZTIZtFotDAYD/VHn83laLkakOIkwyEoEAgEMBgNttEE8yZXJO2wMPrk3zGYznE4nBgYGYLPZoFQqm4RZ1oJcw1QqhUwmg2QyCb/fj3w+Tw24SqWiBsntdm/jN2tfeJ5HNpule88ymQz1eh3FYhE8z9Pkylwu1zQflUolOkeVy2VUq1UEAgF4vd4N/ZYFAgHi8ThkMhmi0SjC4TDUajWCwSD1xCUSCZRKJVQqFUwmE3p6eiCTyTA3N4d8Pr8Vl2VPIxKJoNFoaDREIBCgUqkgk8lQCVWSVNhqtt2Ai0QidHZ2wmKx4Omnn8arr74KsVgMsViMbDaLYrGIxcVFvPfee03hqPuh1+tx/PhxdHR04JVXXkFfXx9t1kDI5XK4evUqgsEgrl27RuXw2mEQ2hmO46jh7urqQl9fH92S8Pv9eO+99xAOh7G0tIRCoYCuri7qYQOf1FPq9XocOHAA/f390Ov1rfo6bY1AIIDD4cDo6CgOHTqEl19+GQqFgiYL3s+ANxoN1Go1TExM4MaNG1heXsbVq1fpFkelUoHZbIZKpcJ//I//EceOHdvGb9a+1Ot1zM3NQaFQoKurC7VaDel0GjMzMyiVSqjX66jX65iamoLH46GvI14wqargeR61Wm3DntnKrHaSf6BQKKgs8cDAAIxGI55//nk89thjGBwcRKPRwMzMDK5cuYJEIrEVl2VPI5VK0dXVhf7+fhqpymazWF5eRigUott97WA7tt2AC4VCaLVa2Gw2WtZFvAue52EwGJDNZptEKe4FCY1LpVKYTCZYLBYYDAbodDpIJJKmC1ytVhGNRhEMBpFKpejNybg/pGEGyRInOufAnWuazWZRKBSo1KDVakVHRwc9hhhwnU5HG5kQBaNSqURDg3t975tAxG2USiXUavV97wNSXrQykzkYDMLn8zVtTUSjUboHXqlUqHfJAFXXCoVCkEgkUCgUSKVS8Pl8dDuvXq/Ta0quW7VaRTweR7Va3fTfrlQqRbVahUKhgFqtRrVaRbFYpKF9sh/P2BoEAgHEYnHTFka9Xke1WqU6Fu1y/2yrASeT05EjR3DixAkMDg7SEBV53mazQSAQrCvJRiwWQyqVwm634/jx43A4HDCbzU2a3IR0Oo0PPvgAMzMz8Hq9bbOH0e7I5XJ0dXXB5XKtMibECIvFYhw9ehQKhQLPPfcchoaGmo4jNwTJ6C2VSojH47h9+zbeeOMNRKNRJqP6AJRKJSQSCWQyGXz88ccIBoO4dOkSJiYmUCwWkclk6MTD9rvXplqt4tq1a5iZmYFcLodCoUCtVkM+n2+qvc7n800h9EajsSXGm5xTPB5HNpsFx3FIJBKIx+MA7ox5OBxGLBZjDsgWsxO297bNgJMQEen00tPTQztRAZ8keUilUigUilX7qGtBVqMajQY2mw1WqxVyubxp75tkdxYKBXi9Xpp13s6D0k5wHAe1Wg2VStW0D7vy+kkkEtjtduh0OgwODmJkZGRVD13yLxF0yeVyiEaj8Hg8SCaTLIHt3yCh2Hq9TgWHViZhklInch0zmQzi8TitApidncXCwsKq992ItCrwSSe53W4keJ6ninPtAhFGqtfryGazEAqFtPa7Wq3SxQSLWm0PxIaQRzuxbQZcJpOhu7sbZrMZPT09cDqdUCqVtJUkUUA7c+YMvF7vmpMQ8EkzB4lEgv379+PYsWNwu90YGBiAVquloSVy0UnZhcfjoclWTAhh/UgkEro9cXcWdGdnJ774xS9CIBDAYrFAoVDAZrOteg+SBELKzG7cuIHp6WnMzc0hEAiw7Yx/g+f5psxmnU5H9QyIkIhEIkEgEMDS0hLtTZxKpTA5OYlUKoVIJPKpn3P3goB89kr8fj/tDX6vygLG9pNKpXD79m14vV6USqVWn86uguQjkHJLoVBI7UgwGMSFCxewuLjYVvZj2ww4yTq32+1wOp1NE32pVILf70c4HMaFCxcwPz8Pn893z/cie4Sjo6N4+eWXqVzqSq+deA+BQAC/+tWv4PP5aOY5Y/1IJBIYjUba6WoldrsdTz/9NEQiEbRa7X27YFWrVSwvL8Pv9+ODDz7A2bNnkc/nkUgkWDTk3+B5Hn6/H36/H7VaDRaLBXa7HWq1mgqMSCQSRKNRjI+PY35+Hj/96U+RSqWQzWY3XFK00pCTzyeEw2H8+te/hs/nY5nObUQ2m8Xc3BzC4TCLWm0yxIDf3cSE53lEIhFMTEwgEom0VenethlwuVyOgYEBuN1uKgSSz+eRzWbh9Xpx7tw5RCIRLC0tUeWhNU+Y4+j7kLaUSqWyac+b53mEw2HE43FMT09jdnYWsViM/eAfgFqthmw2i0wms6ouXyQS0Qxpcv2r1Srq9TpisRhCoRCKxSJVmpqamkI8HofH42F1359CKpXCxMQEgsEgqtUqNBoNlfWcnZ3FxMQEQqEQstnsurqFiUQiOJ1OuFwummeyFiRMWC6XWai2xYjFYrjdbro9VavVUCwWaakgi1ptLlKpFDqdDiaTCVqtFkqlErVajW5ThUIhpFKptrru22bA1Wo1Hn/8cQwNDdEs5UQigYWFBdy4cQM/+MEPaOLG/VL0JRIJnnzySTz99NPo7OxEV1fXqhIbnuexsLCAa9eu4erVqzh79iyKxWJbXfidQqVSQSwWg1wuX2VwV/aeJlUEpVIJpVIJ169fx4cffohQKIQrV66gUCggk8mgUqnQbE6gPeUJ24FAIIBIJAKJRIIzZ87QhE2S1BSNRmki1XquIcdxOHDgAI4dO4bBwUE6XitfS/bfyT5rMplkhqKFyGQyHDp0CMPDw3A4HKhUKrQsMJlMbljoinF/VCoVVT602WwwGo2IRCK0hGx6enrLEhcflC034KTjFGkIsFLoI5vN0tB5NptFPp+/p5wmUcfR6XSw2Wy0TnJl2HZlOQ3RFSahJvZjfzDEYjEMBgMMBsOq0jzgk6QqMnaBQACpVAoLCwvwer2IRqNIJpMoFosoFoss+3+dNBoN2tglnU7TVq4ikQjZbHbd+59CoRByuZxKgJL75l5kMhm6l55MJpnMcAsgcq5KpRJmsxk2mw0KhYLW+pdKJSY7vAWIxWKoVCqaCE2iiisTOtvtmm+5ASctPS0WS1PjCoFAgNnZWbz11ls0NFEqle55gYxGI7785S/D7XbjscceQ19fH30fANT7u379OoLBIP71X/8Vp0+fRrFYbKs9i52G2WzGs88+C5fL1STQspJCoYBbt24hHo/j3Xffxe3btxGJRBAKhWgNK7kJmPHeGCQTeWU7y40YVLlcjv7+flgsFoyNjWFsbAxarXbNEHqj0cCNGzdw+fJl3Lp1C1euXKELL8b2IZfLaa7Q8ePHcfDgQQiFQhpCT6fTLDKyBSgUCnR0dMBms62rCqod2HIDznEcVCoVlU/lOI6GUROJBPx+P5LJ5D1XlKT8TKFQNDXDIBnswCcqVOVyGdFoFH6/H8FgcN1KbozVEGMhk8lgNpthMplWLZjIf9dqNSQSCYRCISwsLGBmZobumzMengeZqFdq2JvNZipypNFo7qmpzvM8bV1JtLbbKeN2r0DmO6VSCa1WC71e36S5TvJMGJsLx3FQKpU0r2cnsOUG3Gw247HHHkNXVxc0Gg3q9To++OADzM7O4tKlS1hcXLxvGZFOp4PD4UBfXx+OHDmC7u5u2s2KGBFSWhGJRPDTn/6U1sQyHhy1Wg2dToeOjg5YLBYYjcYmFbaVZLNZnDt3Dh6PBwsLC0gmkyzq0WLUajVsNhtcLhf+/b//93C5XBgcHIRKpbpvv2qyx5pKpdouXLhXINsepE2pQCCA1+vF/Pw85ufnmfHeIlQqFXp7e+F0OqkGeruz5QZcq9XSum+ZTIZGo4GpqSn8+te/hsfjoU0E7jVZkLCGw+GA2+1GZ2cnTcAh/5LOQH6/Hzdu3MDs7CzzHB4SmUxGG8So1Woolcp7TvzlchkejwdTU1OIRqPrbkLD2DrkcjksFgtcLhcOHz6Mrq4uyOXye4YGSUIbUXArFApsu6NFCIVC2h+CeIKJRAKLi4uIRqNsXLYIEm1cWTLb7kZ8yw040Spf+WMkesz305UloYyBgQE899xzTS0QyUXN5/PI5/Pw+XyYmpqiCVQsxPTgkJKw7u5unDx5EgMDA1AoFE2St3ej0+nwwgsv4JFHHsH777+PyclJWiLI2B5IgpvVaoVer8fAwAAee+wxWK1WKi+81hgKBAKUy2VMT08jFovhxo0b8Hg8yGQyzFC0CI1GgwMHDsDhcIDjOGQyGUxPT+Ojjz7C0tISS8jdQu7ukNju98C27IFLpdImffJyuUzb8K11kQQCAdRqNYxGI0ZHR/Hv/t2/g1arhU6na5qA8vk8gsEgPB4PxsfHEQ6H6X4648HgOI7W2r/00kswmUxQq9Vret8kCmIwGPCVr3wFmUyGNicJhUK0XStjayF9qyUSCXp6erBv3z4cOXIEX/nKVyCXy++58CKL4XK5jMuXL2N2dhZXr17F7OwsG7cWotPpcOzYMdjtdojFYiSTSdy8eRO/+MUvaC9qBgPYBgNeLBYRDochlUpRq9Vo57DOzk7IZLKmhJpGo0HLVpxOJ9xuN1wuF5RKJWQy2aqJiIiEkAfxvhkPDpESVCgUMBgMUKvVEAqFaDQaNKmJlIIpFArodDpaKkgiJrlcDnNzcxAKhSiVSkgmk21ZgrHTEQgEdIHc09MDnU6H0dFRDAwMwOFw0N7f9/IkyuUy0uk04vE4lpaW4PF4kEqlmPFuERKJBDKZDDqdjpbMkrp8UjrGjPf2QbZnSYVUO94XW27AY7EYLl26hHQ6jWeeeQYGgwH79++HWq1GOp1u6mdbKpUwOTmJTCaDJ598EkePHoXT6YTJZALHcasyA2OxGCYmJjA1NYX5+XmkUin2A38IVhoEk8lES/U4jkOpVKJh1mKxiFKphJ6eHhw9epQKuiiVSrzyyit44YUX8Otf/xrvvfceAoEALl68eN8af8aDQa650WjEyy+/jMHBQQwODqK7u5uO2/3CgMlkEleuXIHf78f777+P6elplr/QQrRaLex2O3p7ezE4OAitVguPx4NEItHWRmS30mg0EI1Gsbi42LZ917fcgBNBCqIYRfqBk9aSK0UlSqUScrkcMpkMOjo6YLVaodPpmorqyXHVahXJZBLRaJT+uJn3/fDIZDKoVCraOIMkc9TrdaRSKUSjUSq6I5fLkUgkIJPJqBFXqVTQaDSw2+1wuVw06lIul9n4PCArlQZJWSURRzIajTCbzbDb7ejo6IDZbG6q817ZCe5uSD/xarWKarVKlfaI184WW9uLWCym5WMkb4go4jHjvf0QbZF8Pt+2jWO23IBLpVIaDiKZlX19fXA6najVak0ec6PRwLPPPotarUZrVkkYkFCtVnH9+nUsLy/j0qVL+PWvf00NOOPh4DgOfX19NNt/5aKpUqngxo0btBNSOBxGf38/FhcXYTAYMDw8DI1GA4fDAZ1Oh5GREVitVly9ehUTExNUiIIlF24MkUgEtVpNcxNEIhFV57JarTh8+DD0ej0OHToEs9m8qmf7/VAqlejt7YVSqURPTw+drEiEhSWybS9qtRoulwtmsxkymQzVahWXL1/GxMQEK4ttAfV6nbY8btetpW2RUiXeGQnnaTQaaLXaDb0PuXgrwxpLS0tYXl5m7Sg3CYFAAI1GA4vFApVK1ZRzUK/XEY/H4ff74fF44Pf70Wg0YDAYqMperVaDyWQCz/PQ6/XQ6/WIxWJQqVSQyWT3rT9mrA3JRyBJahzHwWw2w+Vywel04pFHHqFaCSqVakPvLRaLodVqUS6XYTAYoNfr6UKBtPlluQvbh1QqhUajoc2ZqtUqwuEwlpaWmChSiyBlle2qRrjlBjwej+PatWuIx+Po7++Hy+XC0NDQmn2jgXv3J240GsjlcsjlcpiZmcHVq1exuLiIbDbL9LU3kXK5jFwut6qhzN3Xt16vIxKJ4NKlS9BoNAgGgzAYDHjhhRewb98+qNVqqNVqyOVy2uq1Wq1CJBLRhBzGvZHJZHQx9fzzz8NsNkOtVtMtDtItqbOzk1Z5bBSO46DVaiEWi/GlL30JTzzxBPXAQ6EQZmZmkEgkcOXKFWZAtgGj0YiRkRE4nU5ahknay7KSzK2DJO4qlUoYDAa6bbsT2HIDnk6n6ePy5cuIRqM0BHg3Kz0+ItJCII1KUqkUFhcXcfv2bcRiMdareBPheb4p4/VeCVBEeCeRSCCRSEAikcDr9UKv16O3txdWqxUAqMFxOBwQCAS0ly7Ji2DcG6lUCr1eD7fbjc9//vPo6emBwWC4byOSjbJS5vjZZ58FcMfjIMI8Fy9exOLiIqamppgB3wYMBgP6+vqg1Wpp29BYLIZwOMy2CLeQlTklWq0WKpWKSaneTbFYxPT0NJLJJMxmMwqFAqRSKZUMtFgsEIvFTUItK8lms7hw4QICgQDNOG/XsMZOhed5pFIpSKVS5HI5AJ8sqjiOQ0dHB9LpNO1WRRKgSDcyoVBIu8CJRCLY7XZotVrs378fNpuNqkhVq1X6/oxm5HI5pFIpent7cezYMbhcLlgsFtrFbyPKUGvp1t/rGALp7mcwGDAwMAC9Xk+Nyd2LuUwmA4/Hg0KhgFAoxDLYHwCBQEDH3Gw2o6OjAxKJhHYdI8mfbItw6yBlsAqFAnq9HhqNBhzH0RwtUj3TjlHebTPg6XQaZ86cgUwmQyaTQX9/P82g7ejowMmTJ2kZ093eNwBEo1H8+Mc/xvT0NILBINLpdFte0J1MvV5HIBCg2f0rkUqlGBkZgVarRTwep7kH5EdOWoZOTU1BJBJBJpNheHgYVqsVX/jCFxCLxRAKhSASiVAoFFa9P+POZK7VamEymXDixAl84xvfgEajgclkeiDjvfL4te6VtY4he+2dnZ1wOBxoNBp46aWX1lSmmp2dxT//8z/D5/Phww8/ZAb8ASBCSDqdjorwFItFuj1YKBTYdd1iVnrfTqeTal8Q410oFNo2YrhtBpzneRqWjUQikMvlaDQaUCgU1BCQVc7K0pdSqYRsNkv7E6fTaZTLZZZYs0UQ4YhisYhcLkfFdkQiEXQ6HarVKlwuF3p7e2lZGQAaTSFawiQbmnTEIv3gFQpFUw93RjNisRgymQwymYz2AF9ZRrZdEEndlUb+bgNOOgxKJJIdE3JsN4jUNBGrEovFdBuDzIuMrWVlq17yuwfu/N5zuRx1TtqRbZ1Jyd7nrVu3MDs7i8OHD0Or1dI9H6LnvHIy8Pl8OHv2LDweD3w+H5LJJK1XZWwuK+uCo9EopqamYDAY0NXVBYlEguHhYfT29sJkMuHkyZNYWFjAxx9/DI7j4HK5oNVqcfLkSaoKtnIcRSIRDAYDbDbbhrOl9xJKpZJ63IlEAo1GA0qlcsOLnvslIG72MYwHRygUwmg0wul00sqcSqWCSCSCaDTKtBNaSK1Ww8zMDM6fP09lv9uNbXeFyKomn883hYbuXt2T/8/n8/D7/QiHwygUCnTPlbE1NBoN1Ot15PN5RKNRCAQCdHR0QCgUQqFQ0O5wYrEYjUYDXq8XHMehu7ubKkmZzWbaepS8H5mIdkp2ZysRCASoVCpIpVIQCASwWq00g/9uT5zcM2t56OQ+udfEc698k7uPWcuIk8XeTmn60M6QqAsRTarVaigWiygWi2zvuwWQOatcLiOfzyOdTrf6lO5JS2KZpC7cZrNh3759sNls0Gg0kMvldIIvlUoolUpUsCUSiSCXy7GSsS2GJMxcv34d9XodPT09VFjH5XJBpVJBp9NBLpdDp9Ohq6uLNp8Ri8VQqVRNBpskOkWjUVy+fBnz8/OIRCIt/pbtCc/zCAaDyGazNFnTbrfji1/8Imw2G5xOJ3Q6HT2+Xq/T0N7KewcA3QIhxnWtRS/Z+xMKhRuq0y+XyygWi3SPtlQqsUX1JlIul2n5GMs+334KhQImJycRDofbPlenJQZcJBJBLBZDrVbDbrfDaDRCLpdTrw0ATR4gfXCZnOD2QErE/H4/CoUCstksHnnkEdRqNVoeplQqac1kV1cXfS2JmFQqFdTrdQgEAuRyOXi9XoRCISwtLcHr9bKSpPtASi5jsRh8Ph/cbjceeeQRAKCKhgTiJQCgeQqEWq2GXC5HveS1IOJKHMetUjy8H9VqlRpuImHMDPjmUa1WkUqlWG+HbWRlZKlSqcDn88Hv97d9mfK2GnCRSASJRILDhw/D7Xbj+PHj6OjooMpDZI+8Vqvh6tWruHXrFm7cuEHrvVk4afsgnaqWl5fxq1/9CkajEclkElarFSaTCXq9niZ9FAoF+Hw+5PN5RCIRaqB5nkcymcT09DTS6TQ8Hg9diDHuD/Guo9EoPvzwQ0xOTiIQCMDpdNJjCoUCgsEgAMBut0OhUNDnQqEQJicn72vAiZa6TCajuQkrt0CUSiUNsdfrdSwsLCAajSIQCGBpaQnBYBA3btxAOp1u+4muXSH3iEQioaFajuOoEBLbctp6SLZ5JpNBKBRCo9FAuVzeESqE22bASRMGuVyOxx9/HI899hh6enqaNLcbjQaKxSIKhQLOnTuHN998E4lEAuFwmBnvbYZ4V6QGWKvVIplMwuFwYGRkBH19fVQAIR6P48MPP0QkEqEhctIgI5fLwe/302Y2LIKyPkh5Xrlcxs9//nMoFAosLS01RTySySRmZmYAAIODg03e+fT0NM6cOXPf8heVSgWn0wmVSoXBwUEYDAYcO3YMo6Oj0Ol0TQuCWq2GiYkJ+rhy5QotB2TbWg9Oo9FALBZDqVRCKpUC8InEbSaTofvijK2D/H7T6TR8Ph8EAgGKxeKO+F1vexZ6tVrF0tISNBoNIpEIvF4vXeU3Gg0UCgWUy2XMzs4ikUggn8+3/SpoN0M8QaFQiKWlJZpIGI1GqQRhJpPBzMwMUqkUQqEQUqkU3QdfWSLI2DgkpAfc8apXkslkkEgkIBAI4PP5qAEAgEgk8qn9o8vlMjKZDKrVKnw+H9LpNGQyGbLZLN0iIYlulUoF4+PjWFxchN/vRy6Xoz0I2n2Sa3fIluJKR6ZcLrNy2W2i0WhAIBDQ+QoA8vk88vl8229hbGsdeK1WQz6fx9tvv4133nkHyWQSGo0GarW66ThiyIvFYtsq4OwVyH4cMRbE6yaCO8AnE06j0WjqOEY8bhY9eXAajQZV5Lp27RoqlQpt80pU7QA0LYSBTzz4+1EqleDz+dBoNLC4uAixWIwzZ85Q0ZiV4VuiyUB0AoimAzMwDwepAyc19cCd/J9EIoFbt25hYWGhxWe4+yF5P8ViEclkEo1GA4FAgDqQ7cy2J7E1Go0mYf5MJsOSmtoc8gNv99VoO/L973//od+DLISIJ16pVFaFxh+kXnjllgbx+BjbD8dxTUmIK0v09iKbcc88CKVSCdFoFPV6HYlEAqlUqu3r8JkkFoPBYLQIoVAInU4Hq9XaFIlkbD/T09P4+7//ewCg4fOV21LtCDPgDAaD0UKkUint+c5oHaSEcyfBBIwZDAajRVSrVZw7dw4///nPceXKlVafDmOHwQw4Y1ewMq+CwdgpCIVCdHd3g+M4/OVf/iXeeuutVp8SYwfBDDhjVxAIBFp9CgzGhiE9BsbGxtDV1YX//t//O9LpNO2+yGDcD2bAGbuCdm33x2Dcj1qtBr/fj6mpKXR0dGB+fh4TExOYnZ2lKnsMxr1gSWyMXQHpP85g7DSSySSSyST8fj8AIJFI7OkyMsb62ZAH/r3vfQ8AcOnSpaY2gisfP/rRjwAAL7/8Mn74wx/inXfewfvvv49nnnkGbrebHufxeAAAf/zHf7zqPX77t38bUqkUPM/j/PnzAIBvfetbeP/999d8ZDIZ8DyP06dPAwBOnz59z/O71+Mf/uEfIBaLae9x8gCAr371q/B4PE3tT9uN3Tw29XodUqkU3/jGN1Y990d/9EcAgI6Oju2+5BtiN47LysfS0hI+/PBDLC4ugud5nDhxAmazua2NULveM7/7u7+LDz/8kH72w44Nz/P4zne+AwDweDxN9f/tTLuNy2bfM/cal42MzaZ74K+//jp6enrwxhtvNClDvfbaa2sePzs7u+pvMzMzVPO5p6cHwB194Oeee26zT5eyvLyMarWKxx57bNVzP/zhD/HDH/4Qb775Jl5++eUtO4etZqeOjVAoxNjYGC5fvrzquQsXLqCnp2dH19Du1HFZSWdnJzo7OwEAqVQKV65cwZe//OVt+eytZDeMzW6EjcsdNn0PfKWaEOHChQt0hXM3b731Fg0dAcDFixdx4cIFvPDCCwAAi8WCp556Ct/97nfX3BP6tH6thUIBU1NTiMVi9z3u1VdfxZtvvrnqAQCf//zn8eabb+L48eP3fY92Z6eODQB85StfwaVLl5qM+PT0ND744AP8xm/8xqe+vp3ZyeOyFn/4h3+IWq2GP/iDP3ig17cTu21sdgtsXO7wQB74P/7jP+Kdd95Z9fff//3fx0svvYQ33ngDr7zyCl588UV4PB783d/9Hfbt24dcLrfqNX19fXj88cfxjW98A+VyGX/1V38Fo9GIb3/72/SYv/mbv8Hjjz+OsbEx/M7v/A56enoQDodx/vx5+Hw+XL9+/Z7nevHiRTz99NN47bXX8Cd/8if3PG5oaAhDQ0NrPtfd3b1jPO/dODYA8M1vfhN///d/jxdffBHf+ta3IBaL8Zd/+ZewWq34b//tv63/ArWI3Touf/7nf45bt27h+PHj4DgOb731Ft577z386Z/+KY4ePbr+C9RCduvYpNNp/PVf/zUA4OzZswCA//k//yd0Oh10Oh3+y3/5L+u5PC2Djcs64DfA9773PR7APR9er5dvNBr8n/3Zn/Fut5uXSqX8wYMH+bfffpv/2te+xrvdbvpeHo+HB8B/5zvf4f/iL/6Cd7lcvFQq5U+dOsVfv3591WfPz8/zX/3qV3mbzcaLxWLe4XDwL730Ev/666/TY06fPs0D4E+fPr3qb6+99tpGvioFAP97v/d7D/Ta7WQvjI3X6+W/8pWv8BqNhlepVPxLL73Ez87OPugl2xZ2+7i8/fbb/LFjx3i1Ws0rFAr+0Ucf5f/pn/7pYS7ZtrHbx4ac01qPlefebrBxWT8Cnt8B2QwMBoPBYDCaYHXgDAaDwWDsQJgBZzAYDAZjB8IMOIPBYDAYOxBmwBkMBoPB2IEwA85gMBgMxg6EGXAGg8FgMHYgGxJyWSlZtxdp54o7NjZsbNqVdh0bNi7tOS4AG5v1jg3zwBkMBoPB2IEwA85gMBgMxg6EGXAGg8FgMHYgzIAzGAwGg7EDYQacwWAwGIwdCDPgDAaDwWDsQJgBZzAYDAZjB7KhOvB2QSqVguM41Go11Go18DyPRqPR6tNiMBgMBmPb2HEGnOM4DA4OwmazIRwOIxAIoFwuI5fLMSPOYDAYjD1D2xhwoVC46v/XUuORSCQwGo1wOBxoNBpIp9MQCoUoFArgeb6t1YV2OmQ8BAJB02O9NBoNFi1pE8jY3X2fkXuIjBWDsVu51z2wFuS+aLf5qy0MuFarRV9fH6RSKYRCIUQiEXp6euB0OiESiSAWi+kFFolEcDqd0Ov1SKVSiMfjWF5exrvvvotUKoVIJIJSqdTib7T7EIlEsFgsUCgUsNlsMJlM0Ov1cDgckEgkUCgUEIlE95z08/k8zp8/j3A4DJ/Ph0gkss3fgEEwGAywWCywWq04cuQIlEollEolRCIRZmZmsLS0BL/fj6mpKdRqtVafLoOxaQiFQojFYkilUnR1dUGj0aC/vx9dXV3gOA5SqbTJUSFG2+v1YmFhAdFoFDdu3ECpVGoLQ94WBlytVmN4eBhqtRpisRgikQhPPPEEDh06BIlEArlc3nRRJRIJRCIR6vU6qtUqxsfHMTc3B7/fj0wmwwz4FsBxHIxGI4xGI/bt24f+/n643W4cPHgQCoUCBoMBYrH4ngY8Ho+D4zjcvHkThUKBGfAWotFo0NXVheHhYfyn//SfYDKZYDKZIBaL8Ytf/AIff/wxrl+/jtnZWWbAGbsKoVAIiUQCpVKJgYEBOBwOPPPMM3j00UchlUqhUqmoVw584nlfunQJv/71rzEzM4PZ2VlUKpW2iPhuuwEXCARQKBQ0FG4ymeByuXD8+HEolUpwHEe9bIVCAY7jmjxw4JPwOllNabVaDA0NQa1Ww+/3I5VKbffX2rUoFApYLBbodDqcOHECHR0dcDqdsFqtMJlMUKlUkEgkEAqF4HkeqVQKpVIJCoWC3gwCgQAcx8FisaCzsxPz8/Ot/lp7GovFgtHRUfT09ECr1VLvGwDEYjHkcjkkEkmLz3LnQBwKvV4Ps9mMer2OUqmEarWKaDSKYrG46RO9SCSi8yiJSKrVamg0GuRyOfj9fhQKBSwvLyOXy23qZ+8UBAIBdDod5HI5NBoNDAYDpFIpnbfGxsZgMpngdDohlUohFovpa3meb7I5RqMRQ0NDUCgUiEQiiMfjmJ6eRiqVQr1eb5k33hIDbjQaodPpcODAATz66KNwOBw4ceIE5HI5Nc4ikYhOKvfanyDHWK1WPP300/B6vRgfH4ff79/Or7Sr0el0ePTRR+F0OvHqq6+iv78fQqEQHMfR7Q5ipGu1Gvx+PyKRCBwOB7q6uugxHMehv78fSqUSExMTrf5ae5ru7m48//zzMJlMsNlsNGzYaDQgk8mg0Wiaol6Me0McEplMhtHRURw9ehTlchnxeByZTAbnz5+nEcHNNOJkQazVavHiiy9idHQUXV1d6O/vp1uKfr8f//qv/7pnDbhIJILD4UBHRwcGBgYwMjICvV6Pnp4eyOVy2Gw2uvVH5jECGSvyr9vthsPhQDQahcPhgN/vxw9+8ANMTk6iXC6jUqm05Du2zAPX6XQwm83o6OiA2WyGUqmEVCpFrVZDo9FAsVhEtVqlr+N5HtVqFY1Ggz40Gg10Oh1EIhFkMhlkMtmqZDjGgyMQCOgP3Waz0QhJIpFANpule0Y8z9Mf8czMDCKRCCqVCjiOg0KhgMlkgkgkglKphE6ng0qlglwuR61WaxpjxvZQLpeRTqehUCgANC+QybZUvV5v1entKAQCAcRiMWQyGQwGA1wuF6rVKrRaLdLpNBYXF1Gv15HNZlEoFDbl84RCIWQyGUwmEwwGA70/TSYTNBoNjYqt9Cj3IkKhkCY8kwfxxOVyOZRK5X0jTSuN+MrQu8ViQbVapZHHVi50t92Ai0QiuFwu9PX14fDhwzh58iSkUimkUikajQYSiQSKxSICgUDTPmmlUkE0GkW5XEY2m0W5XMbjjz+Oz33uc/QmEovFzIBvEsRrtlqteOKJJ2C32yESiRCJRPDTn/4Up0+fhsFgQGdnJyqVCg3VLS8vI5lMYmRkBIcPH0Zvby++8IUvQKFQ0FBfd3c3Ojs7kU6nEYlE2iIZZC/h8Xjw7rvvYmRkBMPDw5DJZADuTFTZbBbhcBjpdJqNyzoQCATQaDQwGo3Yv38/XnzxRQiFQtTrdaRSKWg0GszNzeHKlSubEnniOA5yuRxWqxWnTp2C0+nE8ePH0dvbS41Ro9FApVJBuVxu+R5tK+E4DkeOHMETTzxBI4Icx0EikdBtvY2iVqsxMjICrVYLvV4PsVjcMu8baJEHLpPJoFaroVQq6T5puVxGtVpFKpVCLpdDKBRCMBikP8BKpYJQKIRyuYx8Po9qtYpsNtu0Smq3FP+dDNnGIHtGBoMBAOjianJyEhaLBbVaDeVyGQsLC8jlcggEAkin01CpVDCZTFCr1ajVatSbB+4kUanValSrVRambQGlUgmpVArZbJbeL/V6HbVaDcViEdlsliWCrgOyPUQiSyS7n2z9KRQKmM1mJBIJukh6UMg2lUwmo59ls9lgt9thMBjofVYsFumjXTKlW4FIJIJEIoHBYIDdbqd5PGS+edCFDRlvlUoFjuNaPn+1NAs9HA7jypUrKJVKCIVCyOVyuHHjBhKJBBKJBNLpND2WJIYIBAIMDw/DbrfTfbpSqYRgMIhgMNjS1dBuguyFGo1G+uP3eDyIx+Pwer0IBoNIJpPw+/2o1+vI5XKoVqs0YScajeLmzZuQyWTI5XKQy+U0QjIwMIBTp05hcnISPp+PhWu3Gb1ej76+PrhcLnAch2q1SiMnly5dwpkzZ5BMJlkG+n3gOI4mjn32s5/FwYMHMTQ0RJM5ycI2nU4jmUw+9IJIpVJBqVTiwIED+MIXvkCTqsg9CgAzMzMYHx+Hx+PBe++9R+fRvYZcLofb7YbJZEJvby9cLhcUCkXLje1W0BIDTlY/mUwGy8vLSKfTmJ+fRyKRwMWLFxEKhZDP59f80UulUvT09DTtX9RqNaTTaaTTaTbpbBIkVKdQKKDRaKBQKJDP5xGPx+niKp1OIxwOr/n6bDaLYDCIWCyGUqmEWq1GjbjFYkFPTw8SiQTb8mgBSqUSVquV5o/U63XEYjGEw2EsLi5ifn6e5psw1oZ4Ynq9Hvv27cOJEyeg0WhoMiDJ7ygWiygUCg+d60EW1L29vXj++eeh1Wqh0Wiotw/ccYiuXr2KpaUl3L59G7lcbk8ujskcs1KvYuV1+jTuzkBvZ7bdgNfrdXi9XjqhKxQKFAoFxONxFAoFxGIxFIvFVYaYJG4olUr09PTQEoBisYhYLIaJiQn4/X5ks9nt/kp7Ao7jYLPZaFhKLBaj0Wjcc4KoVCrIZrNIJpMIhUIQCoU0amIwGNDV1YXFxUVmwLcJkntiNBpx8OBB7N+/n9Z+l0olzM3NYWFhgUax9uLEv16EQiE0Gg2OHDmCjo4OuN1uaDQaSKVSAHdEi5aWlhAOhzE1NYXp6Wkkk8kH+iySIe12uzE4OIiBgQGoVCrIZDK6WAiFQkin05iYmMD169eRSCRQLpf3rJpeo9FAoVBAoVBApVJBo9Foqu2+m2g0Suep6elpNBoNmpf16KOPoru7e5u/wfppiQFfWFiA3+9HuVxGqVRqkm681w9OKBTSFe/g4CAOHz4MqVSKQqGAUCiEq1evIhAIIJPJbPM32huIxWI4nU4YjUaYzWZIpdL7ZitXKhVUKhXEYjF4vV4Ad0K3JCud4zhMTU0xA75NiEQiagBOnDiB48ePg+M4cByHTCaDqakpjI+Pw+fzoVwut/p02xZiCLRaLR577DH09PSgt7cXOp2OHpPL5TA1NQWv14ubN29icnLygQwpSbSSSCTo7+/HE088gb6+Pmg0Ghp9rFarWFpawvLyMq5evYrLly/Te28vGm/gjo0pFArIZrN0MXqveYbneYRCIczOzuLy5ct4/fXXUavVaIWTwWBgBnwlpBwMuDPJE0/7foab4ziq1mY2m2G1WqFSqVAoFJBOpxGLxZBOp/dsyGgrWKn9W6/XwfM8nUyMRiPNIg8Gg/cNtdZqNSQSCSgUCjruZMHGQrTbAymBsVqt6OnpgdFopLK3+XwemUwGsVgM0Wh0U0qddjMajQZWqxVutxt2ux1Wq5UmZ1arVVQqFSSTSXi9Xvh8Ptqj4UEgnr5SqYTNZoPT6YTBYKBZ7sTL9Hg8mJycRDAYpIvqvWq8gTseeC6XA8dxGB8fh0AggEqlglqtBsdxNHpB7M/NmzcxOztLE3EbjQYkEkmT93437RJib8keeLlcpqv8T/uhSSQSaDQadHZ24rd+67eoBKTJZMLt27dx7do1TExMwOv1IplMMgO+SRDDXavVUK/XUa/XIZFIwHEcRkZG8NnPfhaTk5OIRqP3TRzMZrOYmppCPp/H2NgYraEsl8u0FSxj6xAKhVQi8tChQ3j++edhMBjAcRwKhQICgQB8Ph8mJydx8+ZNVpd/D0gWuMvlwvPPP4/Ozk4cO3YMVquV1lvncjnEYjHMzMzgV7/6FUKh0AOHzoE7US+32w2z2YyjR4/i8ccfh1gspmO3tLSEaDSKt99+Gx999BHy+fyeLx0D7jiGgUAAoVAIf/3Xfw2pVNq01eFyuSAUCpFIJFAoFHDr1i3MzMygUqmgWCxSNUJiwO/HRhs6bTYtTWJbD2KxmBbfE8ECqVRKxRGCwSDi8Tg1CIzNodFooFqtUmNbqVSoUI5Op4PD4UAymYRer6eewMo9N/LDvjt5hOd5Wsb0MN4JY30IhUKoVCpoNBpotVrodLomDyQSiSAcDiOXy9HtLMZqSOhcpVLB4XDAZrNBpVJR/QqSSOv3++mclEqlHmhBRKKOcrkcZrMZDocDer0eSqWSHtNoNJDP55HNZpFKpZBIJNiC+N8gVQDAnR4M5G8ikQgajQYAqAEvFouIRCJIpVJ0jCUSCfR6PYxGI81raFfaopnJ/bBYLDSRoL+/HzabDclkEpFIBOfOncNPfvITpFIpFIvFVp/qroIo4YVCISwuLqJSqaCzsxMqlQqjo6NwuVwYGBiAVqtFKBTCmTNn6CRCkkCIjvq+ffvgcDio+trExATOnTuHiYkJtujaYtRqNY4fP46Ojg709PTQzHMA8Pl8+D//5//A5/PB7/ezyf8+kK6Ibrcbzz33HC0h43ke6XQa+Xwev/zlL/GTn/wEiUQC8/PzTZHGjUBEj2w2G770pS9haGgIbre76ZhyuYzl5WX4fD4kEok9vee9Hoix5jgOk5OTdAFLHBUih0sWaF/5ylfgdDoxODh43/dtdUOTtjXgxIMjF9Rut0On00GpVCIUCiGRSCAQCNAbhRmCzYWEzYvFIpLJJBQKBex2O4A77SgNBgNKpRJ8Ph/kcjmuXr2KQqFAx0IqlUKpVEKr1dJSDqFQSBPbPB4PotEo2wffYiQSCex2O5xOJ7RaLfUY6/U6MpkMZmZm4PP59qxe9nohBpyEYNVqNYA7nnCpVEI2m6WJZKVSCfl8/oEndlLpYTab0dvbi8HBQbrPTj6zUqlQz3svC7asl/stppRKJRQKBZRKJRXjGR4eRldXF3Q63T3HkahVtjIRt60MOFHrEovFVD1n//79ePrpp2ktXyaTwfj4OCYnJzE1NUVrjNnqc2sIh8P42c9+BofDAalUit7eXqo7b7PZcOLECQwODsJgMCCVSiEWiyGXy8HpdKKzsxMmkwn79u2DUCjE+Pg4UqkUPvroI9y4cYPJdW4hRCVMq9VibGwMfX19VPBjeXkZs7OzuHnzJrxeL2KxGBNAug8CgQBOpxNutxvd3d1N20K1Wg0zMzOYn5/H3Nwcrfl+kPlIKBRCKBTCbDbj1KlT1HFZ2R0uHo/D4/HA5/Phww8/hM/nY615HwDSllosFuP48eMYGxuDwWCgcs8DAwNNpYF3o1Ao8Oyzz8LlcuHSpUu4ffs2yuXytm8LtpUBFwqFkMvlkMvl6O7uxtDQEA4cOIBjx45BLBZTHfSpqSmcP3+ehnaZ8d46EokEzpw5A4fDgaNHj8JsNtPmMcQTL5fL6O7uRqFQwOLiIpLJJIaGhjA2NgaBQEAlct99913MzMzg6tWrmJ6ebvVX29WQ/gAqlQoDAwMYGhqipU7BYBAXL17E7OwsQqEQK738FAQCAWw2G0ZHR+F0OpsMeL1ex+LiIq5du4alpaWH8obJ3rder8ehQ4doo6eV3ncqlcLk5CQ8Hg8uXboEv9/PEg8fAIFAAKlUCplMhkceeQQvvvgirFYrBgYG7tmZbCVyuZxq0OfzeRrF2u4qjrYy4BzHweFwwGg04pFHHsHo6CjcbjfEYjHq9ToCgUCTlOdKLXTG1lCr1ZDP5xGNRnHhwgXEYjGMjo6iv7+f7nOLRCLaAcnpdNKtjlwuh0Qigdu3byMWi9FWrw+TmctYH1qtFp2dnejr66MJUCSBkAiMsMl//YjF4lU9o1c+J5PJHriHOtkudDgc6OnpwdDQEDo6OmCxWJrUJqvVKiKRCKampuDz+ZDP5/d8ydiDIhQKoVaroVaraWmyVqtdd1Y56f/OcRx6e3sxNjYGr9eLVCq1rdu5bWXApVIpRkZG0NfXh2eeeQbHjh2je0/xeBy3b9+Gz+fDxMQEpqam2A93GyB71ul0Gv/3//5fqNVqvPrqq5BKpbQtH8dxNDxrtVrRaDSQSqUQiURw48YN/O3f/i3C4TBCoRAKhQIr9dsGbDYbnnzySXR1ddFFFan5npmZwenTp2nOAuP+EG9NrVbTDH4CiRreL9z6aZDQ+ejoKL70pS/B4XBgdHSULo6BOw1oisUiFhcXcfr06YfKcmfcMcDEcPf09GBgYOC+am13Q5xN0nVOrVbj4sWLmJqa2nsGnPS21el06OzspPsQMpkMhUIBkUgE0WgUy8vLVG2NGYHtgwivkMQ2AKv64JISMiL9KBaL6R4T+RvpNc3YWgQCARQKBaxWK5VLFQgESCaTSCaTtP6V1Qyvn3t5ZQKBADqdDjabDd3d3di3b9+GrykJ2fb29sJut8NoNEIikUAkEtH7LhqNIhQKwe/306x3Ngc+HETrgnS4JPXfKykUCqjVak0Ja6QNKZkDibiPXq+HXC6nme3bcW+1hQF3Op149dVXYbfbqTiCVqsFAMzNzeFnP/sZQqEQzp8/j0QiwZI2WgBZsZJVq9FopB1+arUaUqkU6vU6tFotbRcrl8uRy+Vw9OhRGl5i2c5bC1ks2e12nDx5EmazGUqlEpVKBRcvXsT4+DiuXr3Kwq+bhFgspp3ITp48id/8zd/c0DUlIVuBQACTyYSOjg66+CWa3qVSCb/4xS/wy1/+El6vF16vF9VqlVXePASNRoN2u5yfn8eNGzdgsVjQ29tLDXS9Xsfy8jKi0SiVVSXtXImnLhAI0NnZSZN4HQ4HjT5uR2Joyww40fkVi8W0uUVHRwdsNhuMRiNNfkqn01hcXEQoFKK9plnYb3sRCoUQi8XQarUwGo1Qq9V0b45k3ZIacI7jIBKJmhTATCYTSqUSZDIZ9SqY4dgaiNytSqWC0WikHauq1SpisRjNQWCVGxuDeMKkTSiZv4gHplarodFoYLFYNvzexICT5k6ElW1JI5EI5ubmaGtSVr3xcPA8T5XXSCMTgUAAo9FIx7VarSIajVKJWlLWrFKpaFQLAN1GJEJJ1Wp120rLWmLASanY6OgoDh8+DLfbjaNHj9J9JNLfOxaL4eLFi7hy5QptX1kul1noaBuRy+WwWCwwm8149dVXMTw8DJfLBaVSifn5eZw9e5YmqtVqNTz33HMYHh6Gw+FAV1cXjEYjnnzySQQCAczOzqJeryOZTLLM5y1AIBDA7XbD7XZj3759sFgskMvlKJfLyGQymJubw/j4OKLRKDPeG4DnecRiMczOzqJWq0EsFsNoNGJsbAwqlYr2AJdKpeA4bkPGdaUnd7dqIdH0TqfTCIVCWF5eZtsemwSZh3K5HN5//31cv34dcrkcWq2WjgnP81QkTKfTwWg0or+/H1/60peg1+tpUxkib9vZ2YlTp07B5/MhHo8/dA/49bDtBpyUt8hkMrjdbpw8eRI2mw1ut5sa75ViHx6PB8vLy1Tnl608txciK9jR0YEjR47g8OHD1INOJpO4cOECLUuqVCowGo2QyWSQy+Xo6uqCUqlEf38/1Go1bDYbgsEgisUiM+BbgEAgoN2TOjo6aPOGTCaDUqmEWCwGn8+3LRPLboLneWSzWUQiERqJymaz6O/vh0qlAvBJRJGEXzfC3fvrxECTqoFCoUBFWxibA8/zKBaLdC6ampq67/EajQZ6vR6pVApPPvkkJBIJFAoFpFIpNfikUybHcdsmwbrtBlwkEqGnpwcOhwP79+/H4OAgXcnUajUsLCwgkUjg448/xq1bt+D1eml/8M1aeZKVMklMIPA8TxcJ92uVuRewWq1wOBzo6OjAiRMnYLVaYTabUavVMD4+jqmpKdy+fRu3b99GKpWikREi0FKtVuFyuegKValU0v2hXC7H8hi2ANJzfWRkBE6nE0KhEPl8HleuXEEwGKTGmyUSbgyyWAWAZDKJaDQKjUaD2dnZJn1yiUTSNHETI7HWPKJQKGhr0L6+PpjNZvoaEt7NZDI4f/48dWQYrYNEsfx+P86ePQuHw4FTp07R/hBCoRAymQwWiwWpVOqBFnIPwrYbcI7jMDQ0hP379+PIkSMYHR2l2XzZbBYzMzPweDx4//33cfbsWZoBvVmQvSaZTEYL+QmkQQoRh9nLBtzhcODkyZMYHBzEl7/8ZWi1WohEIpTLZZw7dw6vv/46otEo5ufnm5KhLl68iMuXL0OlUuHo0aPQarWw2+1Qq9Vwu92oVCq0Pzhj8yAlME6nE4cOHYLFYqHKhWfPnsXCwgKWlpZYu9AHJBqNIhaLAWhOPFvpPZN9cEKj0UAikVgzZ8diseCll15CZ2cn1Gp10955vV6nEZMPPvgAV65cQTgc3sJvx/g0iBSrx+PBL37xC7hcLgwNDcFqtQL4xK50dHQgnU6vqRewFWybAScKQ6Q1aHd3N4xGI4RCIc22TCaTCAQCWF5eRjqdXtOAkhIZkqij0Wg21M5NKBTCaDRCpVJBLpc3raDr9ToVvc9mszR0FQgE9kzonhgCrVYLt9sNm81GFzk+nw/pdBo+nw+xWAzZbHZVQlq9Xqd7d4lEAkKhsOlHvpFaS8b6EAgEUCqVNENWp9NBoVCA53maCJpMJplc6kOyMrS9FsVisam8kpQorYx4rDT8xOCTyX6l9x2JRBCJRGi+CBu79oCMjVQqRaFQQKVSgUQiacpl2M72ottmwEk/Yrvdjueeew5Hjx6FVCqFQCBAoVCg6mrnzp3DrVu3EI1G13wfEiY0mUwYHh7GI488siGDIBQKaXIVyRokF7xarWJ5eRm5XA7RaBTJZBKXL1/GP//zP++ZfUOSkNHb24tnnnmGGoN8Po8PPvgAMzMz+Oijj7CwsECN9UrIJJRIJDA9PY1isYiurq7WfJk9AsdxsNlsMBgM6O3tRX9/P4A7BiSXy2FxcRHz8/PIZrMtPtPdDcnfIawVPSTtKjUaDfr7+9Hb2wuNRtPU1SqVSuHatWvw+XzweDwIhUJ7OhrYTmSzWUxOTiKTySAajdKI1naFzO9m2z5VLBbDZrPB6XTCYDBArVZTw7nyX5lMBqVSec9SMTJZkfcizdnXi0gkov11NRoN1YcGQGsrc7kcTVKYn59vacP27YQkGJLohlarhUKhQKPRQLlcRjgcht/vRyqV+lSPoFKpUG3gvRK9aBWkTzUZL4lEQvu4kwYLRJCCsXWsZ9uN9BAgwlXEAyfGnnQ3i8ViiMViNP+H0R6Q0j7SNrnVFQHbZsCNRiNeeeUVDA8Pw2q1NhlFuVxOO1d97WtfQzweh8/nW3Pfh+M4jIyMwOFw0JZ7GzGwKzue3b1q4jgOFosFRqMRJpMJxWIRS0tLq8o7diukE5LBYIDD4YDZbEaj0aC1kFevXsXVq1fXpWWeyWTg9XqhVCqZ97DFSCQSDA4Ooru7m25X5PN5+P1+GtmKRqMsDNsGDA4O4pVXXkFHRwcOHTpEBZEA0K07j8eDDz/8EH6/n2WetxlisRg6nQ4mkwkKhYIqTbaKbTPgUqkUXV1d6OvrW/UcaYYhk8kwPDyMfD4Pg8EAv9+/6lixWIyxsTE4HA7IZLIm4YP1sjJcdbdxIVmkRIiEqI3tBUh+gVarpTkCpIwlk8kgEokgGAyuyyCTrE3SXm+vXMNWQBorWK1WmtNRrVaRzWbpGBSLxRafJQO402RmZGSEVnWQvuLAnTHL5/NIJpPw+/0IhUJMtKrNEIlEUCgUUCgUVFq1lXPbthnwYrGImZkZ2lt3ZeiaIBKJoNFoaP9bu92+6hgiOUgM7UazaklXs1QqRSe4tcIgpFnAtWvX9oznQhLO+vr6YDKZANzZ1/P7/fD7/VR+cz0h8VKphHg8TpMRW7VHtBfgOA5WqxVut5tmQUciEVy4cAEej4dlnrcRd2vUr5x7PB4Pzp8/j7m5OTpHsZK/9kKn0+HIkSNwuVwwm82QSqV7wwMnKfjESK9lwIlUHXCnKP5eECNSqVQ2rExUrVaxuLgIr9eLSCSCQCCw6vU8z8Pn8yEUCiEej++Zm0goFMJkMqGzs5Mm95G973A4jFKptO5weKlUQiKRoJnqjK1DJBLBZDLBbrdTDzyZTOL69etUOIfRHkgkEhiNRuj1+qaJn8w5586dQyAQQDQaZePWhqjVaoyOjsLlckGv10MikewNDzyfz+PatWsIhUK0ry3QrEIkFAqhVCrpPsPK8FK9XkexWESlUqF6zslkktZmrpdarYbl5WXE43FkMhmkUqk1DUwikUA6nUYul2t5osJ2QULoOp2Olo7JZDI4HA40Gg2akV6pVFhiTRsgFouhVqthMpnog3RTIrXETHqzPdBoNFAqlTCZTJDJZE1a2sQhSaVSdG5ieSPtARkjjUYDrVaLnp4eDA4Orlosk9yFS5cuYXl5edsqPrbNgMfjcbz55puQy+VYWFhAf3//qpo5iUQCl8sFjUaDoaEh9PT00OfK5TJCoRDS6TR++ctfYmpqCgsLCxvuC74y2/N+IjHkuZX75budla0RSX29Wq3G2NgYDAYD7HY7vF4vXdgwWotcLofL5YLL5aIa6KQio1qtIpfLIZfLMWPQYgQCASwWC1wuFzo7O2nNPhkr0mbX7/fj6tWrqFQqbMzaBLLHbbfbqQDZE088QZueAIDf78fc3BzOnTuHH/zgB1T2ezvYNgO+UqY0HA5TT2GlAReLxahWq1CpVOB5vslIVCoVRKNR5PN5LC0tIRQKUTGRvWJgt4NGo0FLJIBP6lYVCgVsNhst2yOTzL0mGolEArVaDYVCQZs9kNImNjltDkS+kXh0pK80eY6UBLIEwtZB2rsaDAa4XC4YDIam5Ccyz+XzeeRyuT0v4bzZEJlT8iBlYOQ5MgYrbQhpLEPKM6VSKTo7O9HX1weHw0GrmMj8l0gkaNVULpfbVs2Qbc8sqtVquHXrFmZnZ1c9JxAIqKqNTCajLSuBZsOSzWZpjSsz3psHz/NUoMDpdILneYhEIshkMhiNRnzuc5/Dvn378MEHH+DChQsoFApIp9NrjoHJZMLIyAi6urrowiwWiyEUCiGfz7fg2+0+JBIJDAYDDAYDFUUikAVXo9HYNllHRjPEAMjlchw9ehSf+cxnaPUM2f+u1WqYnp7G8vIyFhcX2Xy2iZCSYalUSlU3K5UKUqkUeJ6HXC6HSCRa5YiIRCIaJRkdHaUlfydPnoRCoYBSqaTqocViEVeuXMH/+3//j7Yd3U623YCTFScLwbYfpPlCNptFqVSiK1PihVutVpqpTpJwSGIb0UMnq1qlUkkFe4BPZCXz+fyeSQrcalaWtNydCUvGjPQ2Zmw/xCEh4i0dHR0wGAx0rIhTkkqlEAqFWDRxEyDbshKJpKmqiZTHEu+Y53kolUpwHIdKpdK0lcpxHH1dR0cH3G43Ojs7m0TDSE5WLpdDPB5HIBC4Z0XTVsJqexgUEh2JRqNQKpXYv38/rbUXi8Xo6emB3W6HSCTC8PAw/H4/bt68iXQ6jbm5ORSLRVgsFqjVahw4cADHjh2DVqtFsVhEPB7H4uIiZmdn1yUEw/h01Go1RkZG4HK5aPUGgXgamUyGhWRbBMdx6OjogNlsRnd3N7q6umikpFarIZlMIp1O49y5c7h48SKWlpaYAX8ISBtPo9GIxx57DCaTiebz6HQ6GAwGFAoFBINB8DwPjUYDjuNQrVabnAqxWEyjWgaDAQqFgvbtIBSLRVy4cAFerxfj4+PU+97uihtmwBmUlTXygUAAxWKRhqFEIhGMRiP1su12O2ZnZ1GpVBAOhxEKhQDcUdwzGo1wOp3o6uqCSCRCtVpFoVBAPB5v0g9mPBykQqCjo6Opqx5wZzFG+h2zMr7WIBKJoNPpYDabYTKZYDAY6HNEpz6VSmFxcRE3b96kokeMB4NEnXQ6HQ4ePEiTOw0GA/R6PUwmE/L5PHw+H+r1OnQ6HaRSKcrlcpPWh0QigdlsbmoyczekHHlycpJ6362AGXAGZWWi4c2bN/HGG2/A5XLh8ccfb8qcJd4eCVVls1mMjY2hUqlAr9dDqVTSXsfxeBzj4+MIBAK0JpmVoG0OMpkMdrsddrudChuRCot8Pk+7WbHr3RpIV7+V5X0E0ro4nU7Tzod7RTBqqyCVQ6TjpNVqpU2ryPUn3jXP85DJZOA4jiakEUgC21pkMhksLCwgEong2rVrmJ2dvWfjre2AGXBGE8Rru3r1KoLBIA4ePIjBwUGa1SwUCqHRaKDRaGC32zEyMnJPr0EgEMDr9eKXv/wlfD4f/H4/8743EdJDwOl0NtV/12o1ZLNZhEIhJBIJZsBbBMk+t1gsTW2LgTsRknQ6jUQigUwmw3KCNgHSTIb0dHA4HHRLj+SBSCQSqjJ5P+6VN5JMJnHhwgX4fD58/PHHtCtjq2AGnLEmpVIJqVQKwWAQ165dg9lsRmdnJ9RqNdVKX9nbmJDP51Gr1ZDJZJBOpzE7Owu/349oNMp0nTeZer2OSqVCFQkbjQY1CJFIhHp1LIS+vZDFrkKhgMVigcPhWGXAeZ6nfQbYAmtzICViuVwOExMTyGazsNvt0Gq1kMlkkMvltESM4zjodLqmSidCvV6n40JaxBYKBeRyOfj9fkxPTyMSiVCNhVZuezADzliTbDaLfD6PbDaLaDQKg8GAp59+Gi6XC4cOHcLQ0NCq19TrdUSjUSQSCUxOTmJ8fBzLy8u4dOkSstksyz7fZMhCKZVKoVarQSQS4ebNm5iZmcHVq1cRj8fZHngLIGFau92OgwcP4uDBg6u8vmq1ikQigVgsxha2mwRRtFteXsZ3v/tdKJVKdHV1wWg0oqOjAy6XCxKJBEqlEiqVCkePHoXZbF71PuVyGYuLi8hms/D5fEgmk/B4PJienkYymcTc3BxKpRLy+XzL7y1mwBlrsnIvleige71e2kzmbo8CuHMDeb1eJJNJeL1eeL1ehMPhbVUm2kuUSiUEg0E0Gg0olUqIRCJ4vV74fD6q4d/qCWavc3eEinh1mUwGiUQCiUSC3RubCBGMisfjyGazEIvFKBQKNMy90oBbrdY1RVdyuRy8Xi814KlUis5n2WyWNplph+oOAb8B/3+v15O2c4boVo3Nynpio9EImUwGtVrdpFO/kmKxSGU8M5kMyuXyPfXmN5O9ODYajQY9PT1U/EgoFNL2oblcDtFotC2uSzucw1ps1biIRCJazvT1r38dIyMjGB0dRX9/Py078vl8eOuttxAMBuH3+5FOp7fkXO5Hu44L8PBjQ/QPiGoaEXMRCATgOI5WCKwnhE6qaMj2YKlUapv5jHngjPvSaDToKpUl2rQXmUwG4+PjrT4Nxl00Gg2Uy2WqfxAKheB2u1GpVJBOp7G8vIzl5WXMz88jEokwD3wLIHkFu33bjnngG2A3r1h3Omxs2pd2HZutHBeBQACZTIbBwUEYjUbY7XaYTCYkEgkaip2dnUU+n29ZIlS7jgvA7pn1jg0z4BuA/eDbFzY27Uu7jg0bl/YcF4CNzXrHZu1qdQaDwWAwGG0NM+AMBoPBYOxAmAFnMBgMBmMHsqE9cAaDwWAwGO0B88AZDAaDwdiBMAPOYDAYDMYOhBlwBoPBYDB2IMyAMxgMBoOxA2EGnMFgMBiMHQgz4AwGg8Fg7ECYAWcwGAwGYwfCDDiDwWAwGDsQZsAZDAaDwdiB/H9brD/+z2XdxwAAAABJRU5ErkJggg==\n"
+          },
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "! pip install tensorflow-addons"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "TvaZ-jfOi6r_",
+        "outputId": "065442b9-fcf2-40e6-8585-77ef26e8e333"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Collecting tensorflow-addons\n",
+            "  Downloading tensorflow_addons-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (611 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m611.8/611.8 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow-addons) (24.0)\n",
+            "Collecting typeguard<3.0.0,>=2.7 (from tensorflow-addons)\n",
+            "  Downloading typeguard-2.13.3-py3-none-any.whl (17 kB)\n",
+            "Installing collected packages: typeguard, tensorflow-addons\n",
+            "Successfully installed tensorflow-addons-0.23.0 typeguard-2.13.3\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import tensorflow as tf\n",
+        "import tensorflow_addons as tfa"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "-wFNGe2JjDDn",
+        "outputId": "33890c61-62dd-40f0-e43c-a472be69c34b"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/usr/local/lib/python3.10/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning: \n",
+            "\n",
+            "TensorFlow Addons (TFA) has ended development and introduction of new features.\n",
+            "TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.\n",
+            "Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP). \n",
+            "\n",
+            "For more information see: https://github.com/tensorflow/addons/issues/2807 \n",
+            "\n",
+            "  warnings.warn(\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from utils import CustomViT as myViT"
+      ],
+      "metadata": {
+        "id": "kssAyeSPjEk1"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "vit_model = myViT(\n",
+        "    num_classes=10,\n",
+        "    input_shape=(28, 28, 1),\n",
+        "    num_heads=8,\n",
+        "    image_size=12,\n",
+        "    patch_size=4,\n",
+        "    transformer_layers=8,\n",
+        "    num_epochs=30\n",
+        ")"
+      ],
+      "metadata": {
+        "id": "oQxEncJqjXQk"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "model = vit_model.create_vit_classifier()"
+      ],
+      "metadata": {
+        "id": "jnbkX8mBjfZw"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "tf.test.gpu_device_name()"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 36
+        },
+        "id": "SdQlD4yhj5bT",
+        "outputId": "27d2d94f-1383-4adf-daf2-24801d99ca43"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "'/device:GPU:0'"
+            ],
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            }
+          },
+          "metadata": {},
+          "execution_count": 10
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "%%time\n",
+        "\n",
+        "with tf.device('/device:GPU:0'):\n",
+        "    history = vit_model.run_experiment(model, x_train, y_train, x_test, y_test)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "4lvYGbJGjg85",
+        "outputId": "e76dfccd-924a-4bed-a15c-377f5861b7cb"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Epoch 1/30\n",
+            "211/211 [==============================] - 27s 60ms/step - loss: 1.2322 - accuracy: 0.5791 - top-5-accuracy: 0.8981 - val_loss: 0.4628 - val_accuracy: 0.8520 - val_top-5-accuracy: 0.9940\n",
+            "Epoch 2/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.5396 - accuracy: 0.8252 - top-5-accuracy: 0.9908 - val_loss: 0.2593 - val_accuracy: 0.9168 - val_top-5-accuracy: 0.9970\n",
+            "Epoch 3/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.4178 - accuracy: 0.8674 - top-5-accuracy: 0.9939 - val_loss: 0.2068 - val_accuracy: 0.9340 - val_top-5-accuracy: 0.9978\n",
+            "Epoch 4/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.3659 - accuracy: 0.8868 - top-5-accuracy: 0.9951 - val_loss: 0.1956 - val_accuracy: 0.9355 - val_top-5-accuracy: 0.9978\n",
+            "Epoch 5/30\n",
+            "211/211 [==============================] - 12s 57ms/step - loss: 0.3335 - accuracy: 0.8967 - top-5-accuracy: 0.9956 - val_loss: 0.1647 - val_accuracy: 0.9500 - val_top-5-accuracy: 0.9980\n",
+            "Epoch 6/30\n",
+            "211/211 [==============================] - 12s 56ms/step - loss: 0.3077 - accuracy: 0.9051 - top-5-accuracy: 0.9959 - val_loss: 0.1743 - val_accuracy: 0.9458 - val_top-5-accuracy: 0.9978\n",
+            "Epoch 7/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.2870 - accuracy: 0.9116 - top-5-accuracy: 0.9967 - val_loss: 0.1572 - val_accuracy: 0.9525 - val_top-5-accuracy: 0.9983\n",
+            "Epoch 8/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.2706 - accuracy: 0.9150 - top-5-accuracy: 0.9971 - val_loss: 0.1371 - val_accuracy: 0.9600 - val_top-5-accuracy: 0.9982\n",
+            "Epoch 9/30\n",
+            "211/211 [==============================] - 11s 50ms/step - loss: 0.2555 - accuracy: 0.9215 - top-5-accuracy: 0.9974 - val_loss: 0.1360 - val_accuracy: 0.9588 - val_top-5-accuracy: 0.9983\n",
+            "Epoch 10/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.2451 - accuracy: 0.9243 - top-5-accuracy: 0.9969 - val_loss: 0.1205 - val_accuracy: 0.9643 - val_top-5-accuracy: 0.9988\n",
+            "Epoch 11/30\n",
+            "211/211 [==============================] - 11s 51ms/step - loss: 0.2382 - accuracy: 0.9260 - top-5-accuracy: 0.9973 - val_loss: 0.1490 - val_accuracy: 0.9560 - val_top-5-accuracy: 0.9985\n",
+            "Epoch 12/30\n",
+            "211/211 [==============================] - 10s 50ms/step - loss: 0.2322 - accuracy: 0.9281 - top-5-accuracy: 0.9975 - val_loss: 0.1260 - val_accuracy: 0.9632 - val_top-5-accuracy: 0.9985\n",
+            "Epoch 13/30\n",
+            "211/211 [==============================] - 11s 51ms/step - loss: 0.2266 - accuracy: 0.9299 - top-5-accuracy: 0.9978 - val_loss: 0.1142 - val_accuracy: 0.9615 - val_top-5-accuracy: 0.9987\n",
+            "Epoch 14/30\n",
+            "211/211 [==============================] - 11s 51ms/step - loss: 0.2155 - accuracy: 0.9336 - top-5-accuracy: 0.9979 - val_loss: 0.1417 - val_accuracy: 0.9603 - val_top-5-accuracy: 0.9985\n",
+            "Epoch 15/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.2123 - accuracy: 0.9346 - top-5-accuracy: 0.9981 - val_loss: 0.1070 - val_accuracy: 0.9663 - val_top-5-accuracy: 0.9985\n",
+            "Epoch 16/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.2028 - accuracy: 0.9364 - top-5-accuracy: 0.9980 - val_loss: 0.1231 - val_accuracy: 0.9617 - val_top-5-accuracy: 0.9987\n",
+            "Epoch 17/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.2021 - accuracy: 0.9379 - top-5-accuracy: 0.9980 - val_loss: 0.1218 - val_accuracy: 0.9637 - val_top-5-accuracy: 0.9987\n",
+            "Epoch 18/30\n",
+            "211/211 [==============================] - 12s 57ms/step - loss: 0.1951 - accuracy: 0.9390 - top-5-accuracy: 0.9982 - val_loss: 0.1090 - val_accuracy: 0.9683 - val_top-5-accuracy: 0.9987\n",
+            "Epoch 19/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.1964 - accuracy: 0.9390 - top-5-accuracy: 0.9982 - val_loss: 0.1087 - val_accuracy: 0.9693 - val_top-5-accuracy: 0.9985\n",
+            "Epoch 20/30\n",
+            "211/211 [==============================] - 11s 51ms/step - loss: 0.1950 - accuracy: 0.9388 - top-5-accuracy: 0.9984 - val_loss: 0.1067 - val_accuracy: 0.9670 - val_top-5-accuracy: 0.9988\n",
+            "Epoch 21/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.1863 - accuracy: 0.9425 - top-5-accuracy: 0.9986 - val_loss: 0.1132 - val_accuracy: 0.9662 - val_top-5-accuracy: 0.9987\n",
+            "Epoch 22/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.1860 - accuracy: 0.9442 - top-5-accuracy: 0.9984 - val_loss: 0.1084 - val_accuracy: 0.9673 - val_top-5-accuracy: 0.9978\n",
+            "Epoch 23/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.1811 - accuracy: 0.9439 - top-5-accuracy: 0.9984 - val_loss: 0.1085 - val_accuracy: 0.9672 - val_top-5-accuracy: 0.9983\n",
+            "Epoch 24/30\n",
+            "211/211 [==============================] - 11s 50ms/step - loss: 0.1799 - accuracy: 0.9446 - top-5-accuracy: 0.9984 - val_loss: 0.1018 - val_accuracy: 0.9683 - val_top-5-accuracy: 0.9983\n",
+            "Epoch 25/30\n",
+            "211/211 [==============================] - 11s 50ms/step - loss: 0.1790 - accuracy: 0.9453 - top-5-accuracy: 0.9985 - val_loss: 0.1028 - val_accuracy: 0.9690 - val_top-5-accuracy: 0.9993\n",
+            "Epoch 26/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.1773 - accuracy: 0.9448 - top-5-accuracy: 0.9985 - val_loss: 0.0991 - val_accuracy: 0.9712 - val_top-5-accuracy: 0.9988\n",
+            "Epoch 27/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.1755 - accuracy: 0.9460 - top-5-accuracy: 0.9985 - val_loss: 0.1067 - val_accuracy: 0.9678 - val_top-5-accuracy: 0.9985\n",
+            "Epoch 28/30\n",
+            "211/211 [==============================] - 11s 52ms/step - loss: 0.1720 - accuracy: 0.9461 - top-5-accuracy: 0.9986 - val_loss: 0.0946 - val_accuracy: 0.9705 - val_top-5-accuracy: 0.9993\n",
+            "Epoch 29/30\n",
+            "211/211 [==============================] - 11s 53ms/step - loss: 0.1684 - accuracy: 0.9482 - top-5-accuracy: 0.9988 - val_loss: 0.0892 - val_accuracy: 0.9733 - val_top-5-accuracy: 0.9993\n",
+            "Epoch 30/30\n",
+            "211/211 [==============================] - 11s 51ms/step - loss: 0.1703 - accuracy: 0.9473 - top-5-accuracy: 0.9984 - val_loss: 0.0931 - val_accuracy: 0.9732 - val_top-5-accuracy: 0.9987\n",
+            "313/313 [==============================] - 4s 14ms/step - loss: 0.0981 - accuracy: 0.9673 - top-5-accuracy: 0.9994\n",
+            "Test accuracy: 96.73%\n",
+            "Test top 5 accuracy: 99.94%\n",
+            "CPU times: user 5min 36s, sys: 20 s, total: 5min 56s\n",
+            "Wall time: 6min 38s\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "GqpUbHHVj7o3"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

docs/notebooks/ex06a - autoencoder.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"ex6 - autoencoder.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["## Autoencoder\n","\n","<p align='center'>\n","  <img src=\"https://blog.keras.io/img/ae/autoencoder_schema.jpg\" width=600></img>\n","</p>\n","\n","\"Autoencoding\" is a data compression algorithm where the compression and decompression functions are 1) data-specific, 2) lossy, and 3) learned automatically from examples rather than engineered by a human. Additionally, in almost all contexts where the term \"autoencoder\" is used, the compression and decompression functions are implemented with neural networks.\n","\n","1) Autoencoders are data-specific, which means that they will only be able to compress data similar to what they have been trained on. This is different from, say, the MPEG-2 Audio Layer III (MP3) compression algorithm, which only holds assumptions about \"sound\" in general, but not about specific types of sounds. An autoencoder trained on pictures of faces would do a rather poor job of compressing pictures of trees, because the features it would learn would be face-specific.\n","\n","2) Autoencoders are lossy, which means that the decompressed outputs will be degraded compared to the original inputs (similar to MP3 or JPEG compression). This differs from lossless arithmetic compression.\n","\n","3) Autoencoders are learned automatically from data examples, which is a useful property: it means that it is easy to train specialized instances of the algorithm that will perform well on a specific type of input. It doesn't require any new engineering, just appropriate training data.\n","\n","To build an autoencoder, you need three things: an encoding function, a decoding function, and a distance function between the amount of information loss between the compressed representation of your data and the decompressed representation (i.e. a \"loss\" function). The encoder and decoder will be chosen to be parametric functions (typically neural networks), and to be differentiable with respect to the distance function, so the parameters of the encoding/decoding functions can be optimize to minimize the reconstruction loss, using Stochastic Gradient Descent. It's simple! And you don't even need to understand any of these words to start using autoencoders in practice.\n","\n","A more detailed illustration is below\n","\n","<p align='center'>\n","  <img src='https://miro.medium.com/max/1400/0*_K4yRm8ILDZ02vPq.png' width=600></img>\n","</p>\n","\n","The loss function to train such a neural network is the following\n","$$\\mathcal{L}(\\theta, \\phi) = \\frac{1}{n} \\sum_{i=1}^n (x_i - f_{\\theta}(g_{\\phi}(x_i)))^2$$\n","\n","As seen above, the loss function depends on $\\theta$ and $\\phi$ which are the parameters that define the encoder and the decoder. From the autoencoder image above, the encoder is represented by $G_{\\phi}$, while the decoder is represented by $f_{\\theta}$ and they simply mean the weights and bias of the neural network. So in the equation, we are summing up the difference between the original image, $x$, and the reconstructed image $f_{\\theta}(g_{\\phi}(x'))$.\n","\n","\n","The original paper is [here](https://arxiv.org/abs/1312.6114).\n","\n","\n"],"metadata":{"id":"iGhCxXl5o-ja"}},{"cell_type":"markdown","source":["### Setup\n","\n","We'll start simple, with a single fully-connected neural layer as encoder and as decoder:"],"metadata":{"id":"0BJO-ehH00As"}},{"cell_type":"markdown","source":["### Encoder"],"metadata":{"id":"P3RSyGCFmRD0"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"w7hyJnu60USA"},"outputs":[],"source":["# import\n","import keras\n","from keras import layers\n","\n","# This is the size of our encoded representations\n","encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats\n","\n","# This is our input image\n","input_img = keras.Input(shape=(784,))\n","# \"encoded\" is the encoded representation of the input\n","encoded = layers.Dense(encoding_dim, activation='relu')(input_img)\n","# \"decoded\" is the lossy reconstruction of the input\n","decoded = layers.Dense(784, activation='sigmoid')(encoded)\n","\n","# This model maps an input to its reconstruction\n","autoencoder = keras.Model(input_img, decoded)"]},{"cell_type":"markdown","source":["Let's also create a separate encoder model:"],"metadata":{"id":"w4yjtNJq01uQ"}},{"cell_type":"code","source":["# This model maps an input to its encoded representation\n","encoder = keras.Model(input_img, encoded)"],"metadata":{"id":"7VaRBrMF0jqL"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Decoder\n","\n","As well as the decoder model:"],"metadata":{"id":"BzxN6brP03Q2"}},{"cell_type":"code","source":["# This is our encoded (32-dimensional) input\n","encoded_input = keras.Input(shape=(encoding_dim,))\n","# Retrieve the last layer of the autoencoder model\n","decoder_layer = autoencoder.layers[-1]\n","# Create the decoder model\n","decoder = keras.Model(encoded_input, decoder_layer(encoded_input))"],"metadata":{"id":"sdBJPGHI0kIj"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Now let's train our autoencoder to reconstruct MNIST digits.\n","\n","First, we'll configure our model to use a per-pixel binary crossentropy loss, and the Adam optimizer:"],"metadata":{"id":"YiOx-Qtr05Fj"}},{"cell_type":"code","source":["# configure\n","autoencoder.compile(optimizer='adam', loss='binary_crossentropy')"],"metadata":{"id":"O6hXTm7q0lZr"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Let's prepare our input data. We're using MNIST digits, and we're discarding the labels (since we're only interested in encoding/decoding the input images)."],"metadata":{"id":"V2EjeX1v06r4"}},{"cell_type":"code","source":["# data\n","from keras.datasets import mnist\n","import numpy as np\n","(x_train, _), (x_test, _) = mnist.load_data()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"chD4cZ6q0mzq","executionInfo":{"status":"ok","timestamp":1655991591052,"user_tz":240,"elapsed":1177,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"89c94647-d3c7-40b3-8bc1-092044332202"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n","11493376/11490434 [==============================] - 0s 0us/step\n","11501568/11490434 [==============================] - 0s 0us/step\n"]}]},{"cell_type":"markdown","source":["### Rescale\n","\n","We will normalize all values between 0 and 1 and we will flatten the 28x28 images into vectors of size 784."],"metadata":{"id":"WSoOnASB08RV"}},{"cell_type":"code","source":["# process and normalize\n","x_train = x_train.astype('float32') / 255.\n","x_test = x_test.astype('float32') / 255.\n","x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))\n","x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))\n","print(x_train.shape)\n","print(x_test.shape)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"zZUHQNHE0oPS","executionInfo":{"status":"ok","timestamp":1655991591052,"user_tz":240,"elapsed":5,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"5ccc534e-76f6-4b37-a08f-6c93e6c8307e"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["(60000, 784)\n","(10000, 784)\n"]}]},{"cell_type":"markdown","source":["### Train"],"metadata":{"id":"sl5auPekmbf0"}},{"cell_type":"code","source":["# model\n","autoencoder.fit(x_train, x_train,\n","                epochs=50,\n","                batch_size=256,\n","                shuffle=True,\n","                validation_data=(x_test, x_test))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"rpKoyIGA0qPe","executionInfo":{"status":"ok","timestamp":1655991694931,"user_tz":240,"elapsed":103882,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"0e8f0886-787f-4d93-e414-e8f2f209eb9d"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/50\n","235/235 [==============================] - 4s 15ms/step - loss: 0.2766 - val_loss: 0.1889\n","Epoch 2/50\n","235/235 [==============================] - 3s 14ms/step - loss: 0.1703 - val_loss: 0.1534\n","Epoch 3/50\n","235/235 [==============================] - 3s 13ms/step - loss: 0.1441 - val_loss: 0.1334\n","Epoch 4/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.1282 - val_loss: 0.1210\n","Epoch 5/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.1182 - val_loss: 0.1128\n","Epoch 6/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.1112 - val_loss: 0.1071\n","Epoch 7/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.1061 - val_loss: 0.1026\n","Epoch 8/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.1023 - val_loss: 0.0995\n","Epoch 9/50\n","235/235 [==============================] - 2s 10ms/step - loss: 0.0995 - val_loss: 0.0971\n","Epoch 10/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0976 - val_loss: 0.0955\n","Epoch 11/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0962 - val_loss: 0.0945\n","Epoch 12/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0954 - val_loss: 0.0939\n","Epoch 13/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0948 - val_loss: 0.0933\n","Epoch 14/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0944 - val_loss: 0.0931\n","Epoch 15/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0942 - val_loss: 0.0928\n","Epoch 16/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0940 - val_loss: 0.0927\n","Epoch 17/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0938 - val_loss: 0.0924\n","Epoch 18/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0937 - val_loss: 0.0923\n","Epoch 19/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0935 - val_loss: 0.0923\n","Epoch 20/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0934 - val_loss: 0.0922\n","Epoch 21/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0934 - val_loss: 0.0921\n","Epoch 22/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0933 - val_loss: 0.0921\n","Epoch 23/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0932 - val_loss: 0.0920\n","Epoch 24/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0932 - val_loss: 0.0920\n","Epoch 25/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0931 - val_loss: 0.0919\n","Epoch 26/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0931 - val_loss: 0.0920\n","Epoch 27/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0931 - val_loss: 0.0919\n","Epoch 28/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0930 - val_loss: 0.0919\n","Epoch 29/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0930 - val_loss: 0.0919\n","Epoch 30/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0930 - val_loss: 0.0918\n","Epoch 31/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0929 - val_loss: 0.0917\n","Epoch 32/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0929 - val_loss: 0.0917\n","Epoch 33/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0929 - val_loss: 0.0917\n","Epoch 34/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0929 - val_loss: 0.0917\n","Epoch 35/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0928 - val_loss: 0.0918\n","Epoch 36/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0928 - val_loss: 0.0917\n","Epoch 37/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0928 - val_loss: 0.0917\n","Epoch 38/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0928 - val_loss: 0.0917\n","Epoch 39/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0928 - val_loss: 0.0917\n","Epoch 40/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0927 - val_loss: 0.0916\n","Epoch 41/50\n","235/235 [==============================] - 2s 10ms/step - loss: 0.0927 - val_loss: 0.0917\n","Epoch 42/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0927 - val_loss: 0.0916\n","Epoch 43/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0927 - val_loss: 0.0916\n","Epoch 44/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0927 - val_loss: 0.0916\n","Epoch 45/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0927 - val_loss: 0.0916\n","Epoch 46/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0927 - val_loss: 0.0916\n","Epoch 47/50\n","235/235 [==============================] - 2s 9ms/step - loss: 0.0926 - val_loss: 0.0916\n","Epoch 48/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0926 - val_loss: 0.0915\n","Epoch 49/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0926 - val_loss: 0.0915\n","Epoch 50/50\n","235/235 [==============================] - 2s 8ms/step - loss: 0.0926 - val_loss: 0.0916\n"]},{"output_type":"execute_result","data":{"text/plain":["<keras.callbacks.History at 0x7f34e2bfca90>"]},"metadata":{},"execution_count":7}]},{"cell_type":"code","source":["# Encode and decode some digits\n","# Note that we take them from the *test* set\n","encoded_imgs = encoder.predict(x_test)\n","decoded_imgs = decoder.predict(encoded_imgs)"],"metadata":{"id":"TUhxIQ0n0seA"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Use Matplotlib (don't ask)\n","import matplotlib.pyplot as plt\n","\n","n = 10  # How many digits we will display\n","plt.figure(figsize=(20, 4))\n","for i in range(n):\n","    # Display original\n","    ax = plt.subplot(2, n, i + 1)\n","    plt.imshow(x_test[i].reshape(28, 28))\n","    plt.gray()\n","    ax.get_xaxis().set_visible(False)\n","    ax.get_yaxis().set_visible(False)\n","\n","    # Display reconstruction\n","    ax = plt.subplot(2, n, i + 1 + n)\n","    plt.imshow(decoded_imgs[i].reshape(28, 28))\n","    plt.gray()\n","    ax.get_xaxis().set_visible(False)\n","    ax.get_yaxis().set_visible(False)\n","plt.show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"822iBWbp0wEs","executionInfo":{"status":"ok","timestamp":1655991697069,"user_tz":240,"elapsed":13,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"463d581d-d245-4af8-9a74-faa93088fe24"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 1440x288 with 20 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAABG0AAADnCAYAAACkCqtqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dZ7wURdbH8WJVFBMIAio5mBFRUFHRFcUEmHFFMWEOa3oQc467sLJrQMKaMwZUUNBVFF2zAoIg4IIECZJFUcR0nxf78ey/DneGucPM3L4zv++r01bdmWZ6qrunrVOnWllZWQAAAAAAAECy/KGydwAAAAAAAACr46ENAAAAAABAAvHQBgAAAAAAIIF4aAMAAAAAAJBAPLQBAAAAAABIIB7aAAAAAAAAJNC6FelcrVo16oNXkrKysmq5eB2OYaVaXFZWVjcXL8RxrDyMxaLAWCwCjMWiwFgsAozFosBYLAKMxaJQ7lhkpg1QOLMqewcAhBAYi0BSMBaBZGAsAslQ7ljkoQ0AAAAAAEAC8dAGAAAAAAAggXhoAwAAAAAAkEA8tAEAAAAAAEggHtoAAAAAAAAkEA9tAAAAAAAAEoiHNgAAAAAAAAnEQxsAAAAAAIAEWreydwCl6dJLL7W4Ro0aUVvr1q0t7tatW8rXGDBggMXvv/9+1Pboo4+u7S4CAAAAAFCpmGkDAAAAAACQQDy0AQAAAAAASCAe2gAAAAAAACQQa9qgYIYMGWJxurVq1G+//Zay7eyzz7a4U6dOUdtbb71l8ezZszPdRVSybbbZJtqeMmWKxRdddJHFd999d8H2qZRttNFGFvft29diHXshhDBmzBiLjz322Kht1qxZedo7AACAyrHZZptZ3Lhx44z+xt8TXXLJJRZPnDjR4i+++CLqN378+Gx2EUWEmTYAAAAAAAAJxEMbAAAAAACABCI9Cnmj6VAhZJ4SpSkxr776qsXNmzeP+h122GEWt2jRImrr0aOHxbfffntG74vKt8suu0Tbmh43Z86cQu9Oydtyyy0tPvPMMy32aYtt27a1uGvXrlFb//7987R3ULvuuqvFQ4cOjdqaNm2at/c96KCDou3Jkydb/NVXX+XtfbFmeo0MIYRhw4ZZ/Oc//9nigQMHRv1+/fXX/O5YEapXr57FTz/9tMXvvfde1G/w4MEWz5w5M+/79buaNWtG2/vuu6/Fr7zyisU///xzwfYJqAq6dOli8eGHHx617bfffha3bNkyo9fzaU9NmjSxeP3110/5d+uss05Gr4/ixUwbAAAAAACABOKhDQAAAAAAQAKRHoWcateuncVHHXVUyn6TJk2y2E83XLx4scUrVqywuHr16lG/Dz74wOKdd945aqtTp06Ge4wkadOmTbT9/fffW/z8888XendKTt26daPthx9+uJL2BBV18MEHW5xuinWu+RSc0047zeLu3bsXbD/wX3rtu/fee1P2u+eeeyx+4IEHoraVK1fmfseKjFaNCSG+p9FUpAULFkT9KislSiv8hRCf6zW9ddq0afnfsSpm0003jbY15b5Vq1YW+yqmpJolmy6rcP7551usqeAhhFCjRg2Lq1Wrttbv66ukAplipg0AAAAAAEAC8dAGAAAAAAAggXhoAwAAAAAAkECVuqaNLwGteYTz5s2L2n788UeLH3/8cYu//vrrqB/5uJVLSwT73E/N+db1F+bPn5/Ra/fq1Sva3mGHHVL2ffnllzN6TVQ+zQnXMrQhhPDoo48WendKzoUXXmjxkUceGbXtvvvuFX49LSUbQgh/+MP//t/A+PHjLX777bcr/NqIrbvu/y7hnTt3rpR98Gtl/N///Z/FG220UdSma1QhP3T8NWzYMGW/J5980mK9v0Jqm2++ucVDhgyJ2mrXrm2xriV0wQUX5H/HUrjmmmssbtasWdR29tlnW8x98+p69Ohh8a233hq1NWrUqNy/8WvfLFmyJPc7hpzR8+NFF12U1/eaMmWKxfpbCLmjJdf1XB1CvMaqlmkPIYTffvvN4oEDB1r87rvvRv2ScJ5kpg0AAAAAAEAC8dAGAAAAAAAggSo1PapPnz7RdtOmTTP6O53W+d1330VthZx2NmfOHIv9v+WTTz4p2H4kyfDhwy3WqWohxMdq6dKlFX5tXz52vfXWq/BrIHm22247i306hZ+Cjtz7+9//brFOE83W0UcfnXJ71qxZFh933HFRP59mgzXr2LGjxXvuuafF/nqUT770saatbrjhhlEb6VG558u7X3311Rn9naaelpWV5XSfitWuu+5qsZ9ir2666aYC7M3qdtxxx2hbU8qff/75qI1r6+o0XeYf//iHxXXq1In6pRovd999d7St6d7Z3PMiMz4VRlOdNMXllVdeifqtWrXK4uXLl1vsr1N6X/qvf/0raps4caLFH374ocXjxo2L+q1cuTLl6yNzupxCCPEY03tN/53I1B577GHxL7/8ErVNnTrV4nfeeSdq0+/cTz/9lNV7Z4KZNgAAAAAAAAnEQxsAAAAAAIAE4qENAAAAAABAAlXqmjZa4juEEFq3bm3x5MmTo7btt9/e4nR5xe3bt7f4q6++sjhVib7yaB7bokWLLNZy1t7s2bOj7VJd00bp+hXZ6t27t8XbbLNNyn6aS1reNpLrsssus9h/ZxhH+TFixAiLtSR3trS06YoVK6K2Jk2aWKxlZz/66KOo3zrrrLPW+1HsfD63lm2ePn26xbfddlvB9umII44o2HthdTvttFO03bZt25R99d5m5MiRedunYlGvXr1o+5hjjknZ9/TTT7dY7xvzTdexef3111P282va+PUgEcKll15qsZZwz5Rfp+2QQw6x2JcN1/Vv8rkGRrFKt87MzjvvbLGWevY++OADi/V35cyZM6N+jRs3tljXMg0hN+sAYnX6POD888+32I+xTTfdtNy/nzt3brT973//2+IZM2ZEbfobRNdW3H333aN+ek7o3Llz1DZ+/HiLtWx4rjHTBgAAAAAAIIF4aAMAAAAAAJBAlZoeNWrUqLTbypdq+50vN9qmTRuLdZrTbrvtlvF+/fjjjxZ/8cUXFvuULZ0qpVPTsXa6du1qsZbOrF69etRv4cKFFl955ZVR2w8//JCnvcPaatq0abTdrl07i3W8hUBpxFz54x//GG1vu+22Fuv03kyn+vrpnzo9WUtnhhDC/vvvb3G6csTnnnuuxQMGDMhoP0rNNddcE23rFHGdiu9T1HJNr33+u8V08cJKl7Lj+TQCpHfHHXdE2yeeeKLFen8ZQgjPPPNMQfbJ22effSyuX79+1PbQQw9Z/NhjjxVql6oMTd0NIYSePXuW22/ChAnR9oIFCyzu1KlTytevWbOmxZp6FUIIjz/+uMVff/31mne2xPn7/yeeeMJiTYcKIU4PTpcyqHxKlPLLXyD3Bg0aFG1rWlu68t363OCzzz6z+Kqrror66e96b6+99rJY70MfeOCBqJ8+X9BzQAgh9O/f3+LnnnvO4lynyjLTBgAAAAAAIIF4aAMAAAAAAJBAlZoelQvLli2Ltt98881y+6VLvUpHpx77VCydijVkyJCsXh+r03QZPyVS6Wf+1ltv5XWfkDs+nUIVsupGsdM0tKeeeipqSzfdVGk1L53yeeONN0b90qUj6mucddZZFtetWzfq16dPH4s32GCDqO2ee+6x+Oeff17TbheVbt26WewrFkybNs3iQlZa0zQ3nw41evRoi7/55ptC7VLJ2nfffVO2+ao06dITsbqysrJoW7/r8+bNi9ryWQGoRo0a0bZO/T/vvPMs9vt72mmn5W2fioGmO4QQwiabbGKxVpvx9yx6fTr++OMt9ikZLVq0sHiLLbaI2l588UWLDz30UIuXLl2a0b6Xgo033thivwSCLqOwePHiqO1vf/ubxSyVkBz+vk6rNp1xxhlRW7Vq1SzW3wU+db5v374WZ7ucQp06dSzWKqY33HBD1E+XafGplYXCTBsAAAAAAIAE4qENAAAAAABAAvHQBgAAAAAAIIGq/Jo2+VCvXj2L7733Xov/8If4GZeWoyYPNXsvvPBCtH3QQQeV2++RRx6Jtn35W1QNO+20U8o2XdcEa2fddf93es90DRu/NlT37t0t9nnjmdI1bW6//XaL+/XrF/XbcMMNLfbfg2HDhlk8ffr0rPajqjr22GMt1s8ohPj6lG+6RlKPHj0s/vXXX6N+t9xyi8Wltv5QoWiJUo09n+P/6aef5m2fSk2XLl2ibS2nrms5+TUYMqXrqOy3335RW/v27cv9m2effTar9ypV66+/frStawL9/e9/T/l3Wj74wQcftFjP1SGE0Lx585SvoWut5HM9pKrsyCOPtPiKK66I2rQMt5a9DyGE5cuX53fHkBV/Huvdu7fFuoZNCCHMnTvXYl1b9qOPPsrqvXWtmkaNGkVt+ttyxIgRFvt1bJXf30cffdTifK7lx0wbAAAAAACABOKhDQAAAAAAQAKRHlWO888/32ItS+vLi0+dOrVg+1RsttxyS4v99G6dsqopGTrtPoQQVqxYkae9Q67pdO6ePXtGbePGjbP4tddeK9g+4b+0VLQvEZttSlQqmuakKTYhhLDbbrvl9L2qqpo1a0bbqVIhQsg+9SIbWq5d0+0mT54c9XvzzTcLtk+lKtOxUsjvRzG68847o+2OHTtavNVWW0VtWnpdp84ffvjhWb23voYv5a2+/PJLi33JaaSn5bo9TX/zKfyptGvXLuP3/uCDDyzmXrZ86VI/9b5xzpw5hdgdrCVNUQph9dRq9csvv1i8xx57WNytW7eo33bbbVfu369cuTLa3n777cuNQ4jvc+vXr59yn9SCBQui7UKlhTPTBgAAAAAAIIF4aAMAAAAAAJBApEeFEPbee+9o269S/jtdyTyEECZOnJi3fSp2zz33nMV16tRJ2e+xxx6zuNSqxhSTTp06WVy7du2o7ZVXXrFYqzIgd3zlO6VTT/NNp/z7fUq3jzfccIPFJ510Us73K0l8RZMGDRpY/OSTTxZ6d0yLFi3K/e9cBwsvXRpGLioX4b/GjBkTbbdu3driNm3aRG2HHHKIxVoVZdGiRVG/hx9+OKP31mok48ePT9nvvffes5h7pIrx51NNZdMURJ+CoRUwjzrqKIt9tRkdi77tzDPPtFiP9eeff57RvpcCnwqjdLxdf/31UduLL75oMRXzkuONN96ItjWVWn8jhBBC48aNLb7rrrssTpcqqulWPhUrnVQpUb/99lu0/fzzz1t84YUXRm3z58/P+P3WBjNtAAAAAAAAEoiHNgAAAAAAAAnEQxsAAAAAAIAEYk2bEELnzp2j7fXWW8/iUaNGWfz+++8XbJ+KkeYL77rrrin7jR492mKfq4qqaeedd7bY56Q+++yzhd6dknDOOedY7HNzK8thhx1m8S677BK16T76/dU1bYrdd999F21rTr6uqRFCvD7U0qVLc7of9erVi7ZTrS/wzjvv5PR9Ub4OHTpYfMIJJ6Tst3z5cosphZtby5Yts9iXttftyy+/fK3fq3nz5hbrWmAhxOeESy+9dK3fq1S9/vrr0baOHV23xq8zk2pdDf96559/vsUvvfRS1Lb11ltbrOtj6HW71NWtW9dif0+ga79dd911Uds111xj8cCBAy3WMushxOumTJs2zeJJkyal3Kcdd9wx2tbfhZxv0/NluHU9qFq1akVturasrju7ZMmSqN/s2bMt1u+E/uYIIYTdd9+9wvs7ePDgaPuqq66yWNerKiRm2gAAAAAAACQQD20AAAAAAAASqGTTo2rUqGGxlo4LIYSffvrJYk3P+fnnn/O/Y0XEl/LWqWWagubp1N8VK1bkfsdQEFtssYXF++yzj8VTp06N+mkZPeSOpiIVkk5pDiGEHXbYwWI9B6Tjy+SW0rnXTyHWMr7HHHNM1Pbyyy9b3K9fvwq/V6tWraJtTclo2rRp1JYqJSApqXfFTq+nf/hD6v/f9tprrxVid5BnmvLhx56mX/lzJTLnU0r/9Kc/Waxp2zVr1kz5GnfffbfFPi3uxx9/tHjo0KFRm6Z/HHzwwRa3aNEi6lfKZdz/9re/Wfx///d/Gf+dnh/PO++8cuNc0fGnSzt079495+9VzHy6kY6PbDzyyCPRdrr0KE1J1+/ZQw89FPXTkuKVhZk2AAAAAAAACcRDGwAAAAAAgATioQ0AAAAAAEACleyaNr1797bYl5595ZVXLH7vvfcKtk/FplevXtH2brvtVm6/F154IdqmzHdxOPXUUy3W8sEjR46shL1BoVx99dXRtpY9TWfmzJkWn3LKKVGblnUsNXo+9KV/u3TpYvGTTz5Z4ddevHhxtK1rZ2y++eYZvYbP+0Z+pCq57tcCGDRoUCF2Bzl27LHHRtsnn3yyxbrmQgirl71FbmjJbh1vJ5xwQtRPx5yuPaRr2Hg333xztL399ttbfPjhh5f7eiGsfi0sJbquyZAhQ6K2J554wuJ1141/yjZq1MjidOt/5YKu4affGS07HkIIt9xyS173AyFcdtllFldkTaFzzjnH4mzuowqJmTYAAAAAAAAJxEMbAAAAAACABCqZ9CidRh5CCNdee63F3377bdR20003FWSfil2mJfr+/Oc/R9uU+S4OTZo0Kfe/L1u2rMB7gnwbMWKExdtuu21Wr/H5559b/M4776z1PhWLKVOmWKwlaUMIoU2bNha3bNmywq+tZW29hx9+ONru0aNHuf18iXLkRsOGDaNtn6Lxuzlz5kTbn3zySd72Cflz6KGHpmx76aWXou2xY8fme3dKnqZKaZwtf57UdB9Nj+rYsWPUr3bt2hb7EuXFTkss+/PaNttsk/LvDjjgAIvXW289i2+44YaoX6olG7Kl6ctt27bN6WujfGeccYbFmpLmU+bUpEmTou2hQ4fmfsfyhJk2AAAAAAAACcRDGwAAAAAAgAQq6vSoOnXqWHzXXXdFbeuss47FOrU/hBA++OCD/O4YIjr9M4QQfv755wq/xvLly1O+hk6PrFmzZsrXqFWrVrSdaXqXTuG8/PLLo7Yffvgho9coRl27di33vw8fPrzAe1KadKpuugoK6ablDx482OKtttoqZT99/d9++y3TXYwcdthhWf1dKfv000/LjXPhyy+/zKhfq1atou2JEyfmdD9K1V577RVtpxrDvvoiqiZ/Hv7+++8tvuOOOwq9O8izp59+2mJNjzruuOOifrp8AEs3ZGbUqFHl/ndNJw4hTo/65ZdfLH7wwQejfv/85z8tvvjii6O2VGmryI/dd9892tZz48Ybb5zy73TZDa0WFUIIq1atytHe5R8zbQAAAAAAABKIhzYAAAAAAAAJxEMbAAAAAACABCq6NW10rZpXXnnF4mbNmkX9pk+fbrGW/0bhTZgwYa1f45lnnom258+fb3H9+vUt9vnCufb1119H27feemte3y9JOnToEG1vscUWlbQnCCGEAQMGWNynT5+U/bScbLr1aDJdqybTfgMHDsyoHyqHrolU3vbvWMMmP3RNPm/x4sUW33nnnYXYHeSBrq2g9ykhhLBw4UKLKfFdfPQ6qdfnI444Iup3/fXXW/zUU09FbV988UWe9q44/etf/4q29f5cS0SfeeaZUb+WLVtavN9++2X0XnPmzMliD7Emfu3DTTbZpNx+uiZYCPG6Ue+++27ud6xAmGkDAAAAAACQQDy0AQAAAAAASKCiS49q0aKFxW3btk3ZT8s5a6oUcseXUvfTPnPp2GOPzervtMxfurSOYcOGWfzJJ5+k7Pfvf/87q/0oBkcddVS0ramK48aNs/jtt98u2D6VsqFDh1rcu3fvqK1u3bp5e99FixZF25MnT7b4rLPOslhTGJE8ZWVlabeRXwcffHDKttmzZ1u8fPnyQuwO8kDTo/z4evnll1P+naYEbLbZZhbr9wJVx6effmrxddddF7X17dvX4ttuuy1qO+mkkyxeuXJlnvaueOi9SAhx2fU//elPKf+uY8eOKdt+/fVXi3XMXnHFFdnsIsqh57vLLrsso795/PHHo+3Ro0fncpcqDTNtAAAAAAAAEoiHNgAAAAAAAAnEQxsAAAAAAIAEqvJr2jRp0iTa9iXdfufXdNAyt8iPo48+OtrWXMT11lsvo9fYcccdLa5Iue4HHnjA4pkzZ6bs99xzz1k8ZcqUjF8f/7Xhhhta3Llz55T9nn32WYs1Bxj5M2vWLIu7d+8etR155JEWX3TRRTl9X1/mvn///jl9fRTGBhtskLKN9RPyQ6+Luj6f9+OPP1r8888/53WfUDn0OtmjR4+o7ZJLLrF40qRJFp9yyin53zHk1SOPPBJtn3322Rb7e+qbbrrJ4gkTJuR3x4qAv25dfPHFFm+88cYWt2vXLupXr149i/3viUcffdTiG264IQd7iRDi4/H5559bnO63o44BPbbFhJk2AAAAAAAACcRDGwAAAAAAgASq8ulRWkI2hBAaN25cbr+33nor2qZ8aeH16dNnrf7+hBNOyNGeIFd0av6yZcuiNi2TfueddxZsn7A6X2ZdtzWl1J9PDzvsMIv1eA4ePDjqV61aNYt1Kiuqrp49e0bb33zzjcU333xzoXenJPz2228Wf/LJJ1Fbq1atLJ42bVrB9gmV44wzzrD49NNPj9ruv/9+ixmLxWXRokXRdqdOnSz2qTmXX365xT6FDmu2YMECi/VeR0uphxBC+/btLb7xxhujtoULF+Zp70rb/vvvb3HDhg0tTvfbXdNGNYW4mDDTBgAAAAAAIIF4aAMAAAAAAJBA1SqSJlStWrVE5BR16NDB4hEjRkRtuuK02n333aNtP/U46crKyqqtudeaJeUYlqgxZWVl7dbcbc04jpWHsVgUGItrMHz48Gi7X79+Fr/55puF3p1yFfNY3GqrraLtW265xeIxY8ZYXATV2Up2LOq9rFYCCiFOYR0wYEDUpqnIP/30U572rmKKeSwmha+Ou+eee1q8xx57WLwWKcolOxaLSTGMxfHjx1u80047pezXt29fizVdsAiUOxaZaQMAAAAAAJBAPLQBAAAAAABIIB7aAAAAAAAAJFCVLPm9zz77WJxqDZsQQpg+fbrFK1asyOs+AQBQLLQEKgpv3rx50fZpp51WSXuCfHnnnXcs1hK3QHm6desWbeu6Hy1btrR4Lda0ARKhdu3aFler9r8lenyJ9X/84x8F26ckYKYNAAAAAABAAvHQBgAAAAAAIIGqZHpUOjpd8IADDrB46dKllbE7AAAAAJC1b7/9Ntpu1qxZJe0JkF/9+vUrN7755pujfvPnzy/YPiUBM20AAAAAAAASiIc2AAAAAAAACcRDGwAAAAAAgASqVlZWlnnnatUy74ycKisrq7bmXmvGMaxUY8rKytrl4oU4jpWHsVgUGItFgLFYFBiLRYCxWBQYi0WAsVgUyh2LzLQBAAAAAABIIB7aAAAAAAAAJFBFS34vDiHMyseOIK0mOXwtjmHl4ThWfRzD4sBxrPo4hsWB41j1cQyLA8ex6uMYFodyj2OF1rQBAAAAAABAYZAeBQAAAAAAkEA8tAEAAAAAAEggHtoAAAAAAAAkEA9tAAAAAAAAEoiHNgAAAAAAAAnEQxsAAAAAAIAE4qENAAAAAABAAvHQBgAAAAAAIIF4aAMAAAAAAJBAPLQBAAAAAABIIB7aAAAAAAAAJBAPbQAAAAAAABKIhzYAAAAAAAAJxEMbAAAAAACABOKhDQAAAAAAQALx0AYAAAAAACCBeGgDAAAAAACQQDy0AQAAAAAASCAe2gAAAAAAACQQD20AAAAAAAASiIc2AAAAAAAACcRDGwAAAAAAgARatyKdq1WrVpavHUF6ZWVl1XLxOhzDSrW4rKysbi5eiONYeRiLRYGxWAQYi0WBsVgEGItFgbFYBBiLRaHcschMG6BwZlX2DgAIITAWgaRgLALJwFgEkqHcschDGwAAAAAAgATioQ0AAAAAAEAC8dAGAAAAAAAggXhoAwAAAAAAkEAVqh4F5EO1aqkXOi8rY/FyAAAAAEBpYqYNAAAAAABAAvHQBgAAAAAAIIFIj0JOaapT9erVo7bDDjvM4ltuucXiBg0aRP3WWWcdi5cvX27x5MmTo37Dhg2zePTo0VHbkiVLLP71118trlevXtRv2bJlFi9atChqW7lypcWkaVWOP/yh/OfK/nhwfPJPj4U/LjrGOBYAAKCUrLfeehb7ZR9+/vlni7lHQraYaQMAAAAAAJBAPLQBAAAAAABIIB7aAAAAAAAAJBBr2iCn1l9/fYsPP/zwqO3222+3uFGjRhavu278NdRc0Bo1alhcv379qN8+++xjsV+P5rrrrrP4mWeesXjixIlRP9biKAw9prpmUQghbLbZZha3atUqatt4440tnjp1qsVz586N+un6Q7/99tva7WyJSbcO1dZbb23xhRdeaPGWW24Z9ZsyZYrFAwYMiNpmzJhhMWMst9KtM5SKHgM/VjI9Pvqd8bn76V6P459/6cazXmt/+ukni3/55ZeoH8dpzfz3Xj9bXdvCf7a6zbUKnv9e6f2Sv1dW+r3y37liVJHrTi75+1ddk3PPPfe0uH379lG/6dOnW/zmm29GbV9++aXFq1atspjzAzxm2gAAAAAAACQQD20AAAAAAAASiPQo5JROC27SpEnU9t1331msUwDTTR/WKdzpzJkzJ9rWNCh9X6YbVo5001Vr1qxpcYcOHaK2LbbYwmL9zsybNy/j10fmNtxww2i7Z8+eFh955JEWb7LJJlG/tm3bWvzRRx9FbbNmzbJY0xFRcT7dpVatWhZvtdVWFvtUqW+++cbiBQsWWPzjjz9G/dKdH1OlOOo537f583emqSGM58z5NAE9n3br1i1q22233SwePny4xaNGjYr6abop/kfHVe3ataM2HX8bbLCBxYsXL476LV261OLvv//eYn9uzOZexX8XNF190003Tfl3en7Q0sQhMBZDSJ+Ko3LxWaV7DX/+13OtHjc/fovxupvrY6Jj24+VAw880OJzzz03att1110t1nHvr8F6fPw54b333rP4/vvvt3jSpElRP71281umNDHTBgAAAAAAIIF4aAMAAAAAAJBAlZoe5aeP6TRrrRoTQrxquk7p/uGHH6J+fmpnofipeqUypdT/uzfaaFAyT3QAACAASURBVCOLfdrT+PHjLR45cqTFL730UtTv888/L/c1dtppp6jf5ZdfbvHmm28etel3pFSORVXhp3XqMdZKUiHEx3H27Nnl/vcQOMa54sdY9+7dLdZj48/dmqZzzDHHRG0jRoywWNMBkBk9x+r06xDitLS99trLYn9dfPXVVy3W1MJsq0elSxPRqeWaChJCCMuWLbM4XYUT3a9iG9v+mplqmn+m09/932+77bYW33jjjVGbpjXqcXr77bejfqRH/Zc/zzVt2tRiraYXQggNGza0ePLkyRb7+xufGvE7X5VGj2u6car9fHqrVrBp165d1KbVGEePHm3x8uXLy92/UqPH3v8e0eqJOlZ8FdNc3IdqapP/Huj1QF/fp6wW4/lU/03pUqW0zY9n/b3SpUsXi2+77baon6bp+wpeqao2pqsC5qtvdu7c2WKtQPXpp59G/S666CKLNe0ca6bHyacZ6pjQ8ebTCpMwdphpAwAAAAAAkEA8tAEAAAAAAEggHtoAAAAAAAAkUF7WtPG5fJpLpjnV2223XdRv7733tlhzh0OISxfOnz/f4jFjxkT9vvjiC4u11LPPRdNcUJ8HrLn2mofo19vQMom+BPG3335rsebFJSEnLpd87mzjxo0t1s8nhBAefPBBi7UssF9/IRX/Ge+8884W16tXL2o79dRTLe7Vq5fF+S59WAprG6XKH87235qqbHEIIYwbN87iuXPnWpxuPQxUTP369S0eOHBg1Ka53Klyt0OIc4S1PGYIIVxyySUW33vvvRb78wMlLMun482Pj+OOO87ibbbZxmJdRyiEeOxoGe5cnJ/8Pu24444W6/gNIT7m+t5+P4rtvKnHMN39US7KPB9//PEW6/j1ffW+Z9WqVRV+32Kln1GdOnWitmuuucbiAw44IGrT9Us+/PBDi/WeNIQQVqxYscb39TIdD36tjBNPPNFiv+6flhPWdVmKbexVhI5FvZe9/vrro376W0XPaQ8//HDU79lnn7XYr2WU6b2oHg89d6frV2pl29OdU/U3iv7+DCGEI444wuJbb73VYr9Om/LHQH/r6fnbr5uix0R/m4YQ38/WrFnTYn/91N++pSTd9VOPr57jOnToEPU755xzLN5ll12iNv389Xw9ZMiQqN+LL75osV+/Ss+h+byXZaYNAAAAAABAAvHQBgAAAAAAIIEKkh6l08R0+lKzZs2iflqqslWrVlFbjRo1LNZ0mjZt2qR8b02ZadSoUdRPp89pGdIQQvjqq68s1qlXdevWjfp9+eWXFvft2zdq++CDDywutqmJ+tn5ErQ6ddCnrk2ZMqXcfuno8fzrX/8aten0Vc+n3hVKsR3rENJPPU2X4pCKT6nTEod+WqpOVcw0jS6dXExBLwZ6DO655x6Lt95666hfqpSodOkseq4OIYQLLrjA4oMPPtjiQYMGRf2GDh1qcS6OdbHQ76xeI0OIp/nqOdWXcF6yZInF6abu6nulGw/6vdC0rBDi8TxjxoyoTachl1I6nH6WfkylOydlwo83TU/076UpGcOGDbM40+uxl+n3pSrRlHh/PtTUP18Gevbs2RY/9dRTFvs00FSfU7bfA/07LfEdQgh77bWXxXPmzInasrkfK3aaqv3oo49a7D9XHVd6rerWrVvK13711VejbT03Zpoq5fvpcUtXqrgY6fdel88IIU79TLc8hV6PNGXJ36PqWLnllluiNk0z1Pfy+5QuzU1Lj+tvqv/85z9Rv2Iep+lKpOu5tnnz5lG/o446ymK99vnnC/qd8MdGvwd6ztelNUII4eSTT7b4448/jtr69etnsT4byPVSDsy0AQAAAAAASCAe2gAAAAAAACQQD20AAAAAAAASKC9r2vhcdS11prn1mgsYQggNGzZM+ZqaQ6ql8zTvLYQQWrRoYbHmCXqav6jlp0MIYfr06RZrLquWxg0hzpHzpTWLOV9fczO1zFkIIcyaNcvi77//PmrLJh9Ty3ofeeSRUZvmQPo1MC699FKLSyG/N59yUaJWX8OvW6N54H6dAF1fKtsxlapcYLGswZAJv7ZFz549Le7SpYvF666b+pKgn7+Wty1vW+l5Utcqu+2226J+Wq727rvvzvj1i53mX2uJ0hDi687YsWMt9rnweux0DKRbXyXdeVPz7nWdohBCaNCggcW+tCnn4tVls5aJ/o2/b0q31tvy5cst1vKlFTkXpiu/qqrq+VXHhJbfDSFe19CPncmTJ1uspXkz/Rz8Z5np3+n19Lzzzova9Hr61ltvRW26Bk9VPVZry1/vbrzxRot1PSB/rPV8qmsFjR8/Pup3yCGHWNyxY8eoTddGmTBhQrmv7fk2PW7F/JujPHpd9KXu9Tyn1yD/PX/hhRcs1t99fizqb1W/Bmqm6zqma+O+NF5PKoQQOnXqZHHnzp0t9uW6de3aVatWWezLo+tzCC3THkI8hvW3pL+30fX7mjRpErXpb+Frr73W4hUrVoRcYqYNAAAAAABAAvHQBgAAAAAAIIHykh7l6ZRonZak09FCCOGJJ56w2Kc96dQjTbPxpbt0urhOm/K0lLCf7rbJJptYrNPu9t5776ifTj310/eLeYqb/tt0ylkIcXnLbD8DPab9+/e32E9l1e9Ejx49ojafeof8yHTKp06D1BSZEOISfgsXLozadDvb71OqKfzFPEZDiP/dO+ywQ9T2l7/8xWJ/DlVarlCnfvvypfPmzbN4t912i9q0BLSmG2y66aZRvxNPPNHi4cOHR21Tp061uNiPm6fjw3+2eoxfeeUVi/01TelY9OdUPZ/7z1n/btddd7X4gAMOiPrpdGBNhy7vNZFd+oweNz+FW8ezT5nQMqW+FHWmij3dVP8d1atXj9q0vHq6e9RsUlX86+m511/D9N7zvvvus9hfW6dNm2bx4MGDo7ZiLh+cKU3lDCG+j0yXBj5u3DiLTz/9dIt96vc999xjsU9j1M9fX8MvOaD8GCuWMZcJPwbatGljsf5mCyGEd955x+J0Kbl6Dnzvvfcs9mMx3XVRZXs8SuU4+jRDHRM33XRT1KaphXoe9r85NS1VUw4/++yzqJ/+RvfHV9O9dUmOrl27puzny8JvvvnmFvt/Zy4x0wYAAAAAACCBeGgDAAAAAACQQAVJj0q1wrmfBqjTl9JNL9XX81Pmli5davGUKVNSvl6mVSx0arpPI9BVyX21jlKZ7uZl8+/2x1CrRG277bYW+xS0fv36WTxs2LC13g+Uz3+WOnYy/Zx1KqFOewwhTpOZOHFi1JbNFP5irGiSDZ3KqdNGQ4jTlPTz8mNs6NChFl955ZUW+xXx9fyqfxNCCPvtt5/FF1xwgcU+Zatp06YW+3RHnTrrp8cWGz91V9PL9JiGEFdX05Qyn/qQ6nuvKRghpL/u6hjW74JOCw4hrtzh06OQvgJMunsb3daKbCeccELUT1On/PF98MEHU+5Hpoo93VSvb74CZrrUQk1N0nQNX6lE6Xj2qTU6jnwl1EGDBlms6XF+f6+66iqLFyxYkHI/Skm6VG09x2klGq0yFEII5557rsX6mfvrllaz9d+XrbbayuJsKsiVmkaNGkXbvXr1stina7/55psZvWYuqm8Vy3kvl1Kl0PrqsDpefHVgPYfq/YyvgnfhhRdaPHfuXIv9cUlXNVO3Dz/8cIt9xWgdw/4Zglagyud3gpk2AAAAAAAACcRDGwAAAAAAgATioQ0AAAAAAEACFWRNG5Uu1yubnMJ0621kuvaGzydt3LixxVpWzufBffrppxbPnj07wz1GCPG6Db6M7XXXXVfu33z00UfR9p133mlxuu+OHjd/rPXvyE0tX7ZlJlPlsvrjrWuU+Nxkv8ZKRd+3lOmaMb4ss35Gep784IMPon66Bk26MtL6ndC1GEII4aWXXrJY17fZZZddon46Tvfff/+oTcf64sWLU+5HMfDrV+jn5POon3jiCYt1fZtM14SriJYtW1q8xx57WOyvi1pu9YcffsjqvYpZpudTfx7Tz7lZs2YWb7PNNinfa9GiRdH2u+++a3Gm91ultkaY/pv82l06/nzJ15122sliXfPJr8Gg41vXNfHjXtdx8OsWtW7d2mI9Pu+//37U74033rC4GI9VNvTz8usITZ8+3eKPP/7Y4t69e0f9dD1LXR/u1ltvjfrpOpj+3J3p+bqU6TlP12cKIb5H8J/tQw89VOH3yvS3AONozVKtaVOvXr2o3zHHHGOxL9uuf6frSz399NNRv1T3g35tQD1f+3X4Tj755HJjf45Xfv2wCRMmWJzPdReZaQMAAAAAAJBAPLQBAAAAAABIoIKnR+mUp3xMM8vmNf30bp2KqlO2fHpA//79Lc4mjaPUVK9e3WItY3vXXXdF/bbcckuLtYS7pkiEkL4ctB5TLY/qS6DqFGSmPeaWjvUGDRpYXKtWraifTkkeMWJE1OaPVybvVapTW/15TFOR/FRR/Rx0jJ1//vlRPz3nZfrZ+X6aYrBw4UKL/ZRwnYqq54AQ4u9MMaZH6fFp2LBh1LbFFltYPGPGjKht5MiRFmc6VjLl02L0ulijRg2L9fsTQgj33HOPxX7aOjLnx5F+Rw466CCL9ViEEH8Phg4dGrXp+MtWKaVyaAnZEEKYNGmSxZoiGEJcvvukk04qNw4h9fnQpxJquWhN2Q8hLj2rf+fTc/I5Tb+q0vPa1KlTo7YHH3zQYk3VXrlyZdRPx9y9995rsS8RrPxr6PIK/tqN/9Jr37HHHhu1aVqaT7XWsehTHDORrlw01kw/P72v8+NDl01IlzasY0d/S4QQnyfr1KljcfPmzaN+7du3t7hdu3ZRmy4loN8dT6+to0ePjtrGjBljcT6vkZwpAAAAAAAAEoiHNgAAAAAAAAlU8PSoXEi1MnUI8ZSqTCtG+XQNXZVcV4h+8skno346PaqY0y6y5Y/N9ttvb/H9999vsZ/6qyuFjxs3zmJf2Sbd1Ht9b52q5v+mlKZ6e7muCuJfT9Ph9tprL4t9lYwXXnjBYj8dPRfHp1TGpk6ZDyFOs/GfgZ7XdEr4l19+GfXLxWenr6FTwv2x1X7p0hjznWJbGfTYNWnSJGrT6cV+in0+qzP5tJuuXbtarMdDx28IIUyePDlv+5RkmU6hz6b6XghxqrYeC5/6qCmNAwYMiNqySZcpljGWKf33LliwIGq7+uqrLb7ooouitq233tpinWLv7zmmTZtmsd7TNG3aNOqn0/49PT9qhajPPvss6ldqx66i/Plz3rx5Fuvn79MpevToYbGmIXt6L6vHPYT4O6KpHP/5z39Svkap2WeffSz21YU0paxRo0ZR23bbbWfx2LFjLU53P6nnW38vpePIj+dU5/1S/m2hn5d+Dr6yqJ6vfEq8Hl+NzznnnKifnof1nkUrt4UQ30f5qlD6+rrvftmTUaNGWeyrHGsF6XymhTPTBgAAAAAAIIF4aAMAAAAAAJBAPLQBAAAAAABIoIKvaZNtjq3mDWo+ms/n1vxPfS+fd6j5bmeeeWbUtvnmm1v88ccfW9ynT5+oH2W+0/N5g5deeqnFut6GL3f47bffWjxw4ECLFy1aFPVL913SPEpdB4Ic7//J9RoM/jhuttlmFh911FEW61o3IYQwYcIEi3NRorRUj7E/F2rOvK5hE0IIX3/9tcX//Oc/Lfaff6pc34p8xrpfO++8s8Xp8sZ9Hn+x5/XruVLLYIYQH0efp926dWuLv/nmG4v92jd6PtRj6r8zun3EEUdEbbrmhq5p89JLL0X9cl16PMn0HJqubG826xv412vZsqXFO+20U7n7EEKcW69xCKV7bsyWPx/q+hjnnntu1KbrKejx9q+h66josfNr2my11VYW67U0hBAWL15s8TXXXGMx96Rrpp+5L+97yCGHWKxrW9arVy/qp+drPd/59Wg+//xzi/2x0fc6+OCDLX744Yejfo8//njK1yhGet7bY489LPbnOd3295Q6Jq644gqLlyxZEvXTdXKOOeYYi/0aRkOHDrVYfxP6/fjuu+8s1jEaQumucaPru/jx0atXL4sPPfTQqE2PgY5TXzZc1y/Sc7C/v9Rtf23VY6PrWl177bVRv+eff95if0+dz3VsFDNtAAAAAAAAEoiHNgAAAAAAAAlUZUp+67TtVFMTQ0g9/den6nTv3t3iU089NWrT6at9+/a1eOHChZnvMKLp3CGEsO+++1qsx1PToUII4eyzz7ZYy6pXZPpZPqeB57pUdiHpvueidLK+hk+10BKo2267bcr30jLT2ZbDxepThDXNxqcXzZw50+J0U65TpdL4866OTT/1VFOizjrrLIt9qo8eey1bvKZ9LAY6PXfGjBlRm5bQ3nHHHaO2448/3uLOnTtb7D8vPcdqatPcuXOjfvp35513XtS20UYbWaxTg3VKeAjJPwfmi46PbKfCp5vyf/jhh1vsU+iUTt/3aXKoGP9d1vOev2/x2xXly4trWpWms4YQl3KfMmWKxZmWNE6nlMbvhhtuGG3vsssuFmt6mr+30XPe7bffbvGTTz6Z8vWPPvroqK1jx44W16xZ02JdOiCEED788EOLJ02aFLUV47HS7+n8+fMt9tc0/Wz9b4MddtjB4v79+1usS1+EEEKzZs0s1uubp6lyb7/9dtSm41TH4p133hn18/c0pULPSf4+VNN3Bw8eHLXdd999Fuu9oqZAhRCXCj/hhBMsPuWUU6J+mlblvy96P6z3UdOnT4/6FSoFKh1m2gAAAAAAACQQD20AAAAAAAASiIc2AAAAAAAACVRl1rTRXDJdcybTnE7NTw0hhN69e1vsy/lpCdNPPvmk3H1A+TT38Oqrr47aateubbHmp2rOaQghjBw50uJMP3Ofr52LXF9dm0NLzvkykbpugM+7rSo5x34/M91v7efXMmnVqpXFmn/sy/7NmjUro/dNV14323LUxcSvEdOkSROL/ZpeSsdYpusepCsVrWsZhRDCs88+a7Gea/17ae7zvffeG7XpGgLFeHx1nZmpU6dGbU888YTFuuZCCHHuvq53o3neIYSwYsUKi8ePH2/xxIkTo36a19+gQYOoLdW6cv58mIu1sqoi/bf6tUWy+Ry0HG0IcVlgPRa+9KiWCM50bR3Op5VPr5chhLDbbrtZPGfOnKjtqaeestivE5Epxmm8nl4I8b2njomvvvoq6nfRRRdZrOtheLoulV8LRdff0POuL1Ws11Nd3yyE4vxNouesl19+2eI2bdpE/XR8LFq0KGrTss1aEtrfm+jxSTce9P61U6dOUZte//T+31/HdcyW0njLlP9MUv3m1ziEEL755huLdV2crl27Rv3q1q1rsf+ddvHFF1usv0+SeJyYaQMAAAAAAJBAPLQBAAAAAABIoCqTHpXNNCWd3njQQQdFbTpVypdT1HQdPxUL6bVv397iLl26RG1aqk2ntPkyiammdPt0Cp3a6P9GS3Pqd8e/hk7z92lyOq1Sp2YuWbIk6qdlc996662oTcsBZlsGNl90f3IxDdBP6919990t1rHoPyNfMjiVdOlR+m/JR6pcVeDLAOv32Zez1GnCOs341VdfjfrpFFX9HH26VYcOHSweNGhQ1NaoUSOL06VfafnH1157LWpL2tjJNf2cfbrL2LFjLfYlX2vVqmWxllb308B1OrCm//oyw3Xq1LF48eLFUVvjxo0t1vQc/ZtSpscwF99XPbYhxONIx6Kfhv/ZZ5+V28/TsZhuXJbK+TMTmaaPqnSfn97D+HRyvUd98cUXo7alS5fmdD9K6RjrOPX3/lq++69//avF/ndAquui/35o2qsutRBCCGeccYbFV155pcU+9Vj5EuWa9losx1D/HZqqoktahBBC8+bNLdbPIYQ4ZalHjx4W65gqb/t3etxCiI+rT1vVNr0vOuecc6J+zzzzjMX6+wRrR5cFOPHEEy1u2LBh1E+/V++//37UNmrUqHL7JREzbQAAAAAAABKIhzYAAAAAAAAJVGXSo7JRv359i88+++yoTac3Pvzww1GbTi8u9mn5a8tPB9WV1X1Khqa06DRPn0blV+r/XYsWLaJtTV/64osvojad9q/vqykE/r333XffqE2nWC5fvtxiv4K/pkf5fbzvvvssTsJ3KddT//T4+zSJPfbYo9y/mTBhQrSd6VTRpE9brGyachhCnJrn05k01UWn5fvvqJ4LmzZtarFO7Q4hhMMOO8xiP31Yx58eQx1TIYTQuXNni7OZ/l+VpavWo1O19ZiGEE/b13QmP/1XU6602ox/L30NX1lKK1fpMfXHu5TkusqSnk/9tUTTi/V7oBVWQlg9VaCi7xsC59rf+c9Fv/c+jSVV9bB01309D7dt2zble7/55psp3ytbpXqM9d/t7z1yMXaUHntf5WvMmDEW33DDDRbvs88+UT/dX3+u1bTXVEsCVGV6nps/f37Upvf4/hjo9rRp0yz2x/e0006z2KeeKU3BSXe8U6VKhVA8x6Sy+THw2GOPWXzggQdarKmnIcT3x5qOGMLq1aSSjJk2AAAAAAAACcRDGwAAAAAAgATioQ0AAAAAAEACFd2aNprHdt1111ns88N1DZQhQ4ZEbStXrszT3hUfn9+payzoukEhxCWhdb0YX+ryzDPPLLffpptuGvXTHN5vv/02atM1HGrXrp3yNTQvPd2/RdetmTlzZtRP13LR8rwhrL4GRbHRz+zoo4+O2po0aWKx5nP7ssXZrPXj/ybXa0tURb5UtK5rouvRhBDnb+taCo8//njUT3N9taS4jssQVi/BrvQ8oCU8TznllKifnpNL9Riuif9c9LPVY+VztDP9PNOVHk/1XnPmzMnqvbA6PZ8eeuihUZuukaD3KFpKNoTVr7uppFqDBf9TkTLZqdax8f30XLn99ttbrOfXEFKPN1S+VGW5KzKO9P51xowZFm+55ZZRv5o1a1q8+eabR226Rksxlv9W6a596SxZssTiAQMGRG2tW7e2uF27dhb7sZju/ibVPg0fPjxq4xybPR1vzz77bNS2//77l9vPl23v06ePxePHj4/aqtJ4YaYNAAAAAABAAvHQBgAAAAAAIIGqfHqUn6Z44YUXWnz88cdb7Kem9e/f32JfYroqTZWqbP6z0vSKI444ImrTNAw9bj5lSaeDpnsvPaa+DJz21SnnPgVKpzMuW7YsatP0Ei0L79N7ND1q3Lhx5e57sdJppD169IjaNIVm0aJFFs+bNy+r92I6f3q+pOjNN99ssU4DDiGeCpxuLPrtbPZj8ODBFmvKqi/5zXl37eTi89PvgpaYDiGefq8pUdOnT1/r98V/bbTRRhb/8Y9/jNp0ir5eqzR1N1uMvczo/UKm6Rmepokfc8wxFqcrEdyyZcuoTUuA+7LVKt29TzGnFKdLZ8nm3+1/Z+gyDPr5Z/ud0L/z9zZ6Da5bt27UpvdV2ZQrLwV6jH3Z8JEjR1q8yy67WOyPgS5z4I+xpi6OGjXK4gceeCDlfqBidOkFTYcKIT6f6nG7//77o3533HGHxdmO0yRgpg0AAAAAAEAC8dAGAAAAAAAggXhoAwAAAAAAkEBVck0bzc3VdVJCCKFXr14Wa1nbd955J+r3/PPPW1yV89tyST/XTPMvfT9dH2i//faL2g444ACLTz31VIv9ehv169e3WPO8fflszev362hoTrPG/lhPnDjR4rvvvjtqGzNmjMVa/tbnu2oJ8FL4Lun3RHOs/b9dc6xHjx5tsS8lnA3yg9dMx2LXrl2jNi1HqedQv65CKn4M6DpFPXv2jNp0/YVSGB9VmV4z/Tn166+/tnjKlCkW+/Uj0p1vsTo9n2q53/XXXz/qp2tn6NhOt6ZJpjifFo6uA9e+fXuL/TjSdVR23nnnqE3XOdHraapS1CGsXgI3XVnyqkjHka5z4T9X/Xf7saOvoX/nP1f9vLI9x+l76XdC739DCKFp06YW+2Oo955YM3+89b70yCOPtNiv5ae/NT7//POo7YUXXrBY17TMxX1uKdPy9ro+ox/POhYnTJhg8cUXXxz1K5Z7EWbaAAAAAAAAJBAPbQAAAAAAABKoSqZH6RTu008/PWrTaYYrV660eODAgVG/XJTHyyadKMly8W/Q19BSeCGE8PLLL5cbZ8qXrNRtP5VcvyM6VdZPj/zhhx8s9ukAmZaVLpZpd9nQaaMDBgyI2po3b27x0KFDLfZTfJF/ixcvjrY7dOhgccOGDS32aVR77723xTVr1rT4pptuivqNHTvWYp/GiLXjz3upZHP+9lONN9hgA4tnzZoVtc2YMcPihQsXWly7du2on6bYlfK5MVOp0jr0807X5lM3iu2+pNjovUm6VEI9j/q0VS0Nr/e5+h0JIX1561T3N+nON0n+PqX6t/rPRKVLJ9PPxx8bbcv0PjHde+vx1O9HCPG9rb/P1WOlr+ePU5KPW2XSa5wurbF8+fKon/6W8WlP+huC6132qlevHm0feOCBFm+yySYW+98Pem+rqfn5uA9NwrmRmTYAAAAAAAAJxEMbAAAAAACABEpsepSfhqRTp/bdd1+LDzrooKifThGcM2eOxX7F72ymMqVLz/GYjph76aZ86hTh8raRO/q5a3rU448/HvXT8aFTFRkblU+n8eoU4f79+0f9/DYKT8eLv+as7Vjyf//NN99Y/OKLL0Ztc+fOtVin8C9dujTql4tqRqVE0yumT59u8TXXXBP108pSWiXDT9fn/Jpsem8ybNgwi4844oion96/vvHGG1GbXnc1XcCPvXQpPqkUw/dH7zf8vztVOmIIqT+vfKQbpbqPev3116N+b7/9tsU+nXXBggVrfG38j/9cvvvuO4vHjx9f6N0peToWa9WqFbVtu+22FmsKmk9HfOihhyz2v/NzLQnjipk2AAAAAAAACcRDGwAAAAAAgATioQ0AAAAAAEACVeqaNunWhPElDhs0aGDxqaeeanGjRo1SvqaWafO5n9nkpvm/8fmla/v6QFWj33NKPQP5levrin+9VatWWTxlypSoPtcpHgAAAhdJREFUTddb0eusX0eDNW2yp/csY8eOjdq4pygOugbUX/7yF4vvueeelP38Og66jpF+Lyg5vLp0Jbl9WyHHmB4rXUtM4xDic60vB677n+73CJAE/je/fmc33XTTqE1Lfm+xxRYWL1myJOp37733WuzLgRcjRjkAAAAAAEAC8dAGAAAAAAAggQqeHpVuCl+6dCktiadTArXEdwjxlMNBgwZZ7KdU5QJTUQEAxcinDmjqFPKPdKjipMd18eLF5cYojKSMsXT7oedhTYsLIS5ZXgqpIaja/Pdcf0PPmjUraps2bZrFrVq1srh///5Rv/nz5+dyFxOPmTYAAAAAAAAJxEMbAAAAAACABOKhDQAAAAAAQAJVq0hOZ7Vq1RKRAKpr36y//vpRm5YdLqY1Z8rKylIv+FMBSTmGJWpMWVlZu1y8EMex8jAWiwJjsQgwFosCY7EIMBaLAmOxCDAWi0K5Y5GZNgAAAAAAAAnEQxsAAAAAAIAEqmjJ78UhhFlr7JVnmtL1448/VuKeFEyTHL5WIo5hieI4Vn0cw+LAcaz6OIbFgeNY9XEMiwPHserjGBaHco9jhda0AQAAAAAAQGGQHgUAAAAAAJBAPLQBAAAAAABIIB7aAAAAAAAAJBAPbQAAAAAAABKIhzYAAAAAAAAJxEMbAAAAAACABOKhDQAAAAAAQALx0AYAAAAAACCBeGgDAAAAAACQQP8PNg453Ycua7EAAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["## Deep autoencoder\n","\n","We do not have to limit ourselves to a single layer as encoder or decoder, we could instead use a stack of layers, such as:"],"metadata":{"id":"vfdgxg8rpiLh"}},{"cell_type":"code","source":["model = tf.keras.models.Sequential()\n","model.add()"],"metadata":{"id":"cO2BHVdDqgtZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# functional form\n","output = name_of_function(input_arg)"],"metadata":{"id":"KE1VZK1Pq41R"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# model\n","input_img = keras.Input(shape=(784,))\n","\n","# encode\n","encoded = layers.Dense(128, activation='relu')(input_img)\n","encoded = layers.Dense(64, activation='relu')(encoded)\n","encoded = layers.Dense(32, activation='relu')(encoded)\n","\n","# decode\n","decoded = layers.Dense(64, activation='relu')(encoded)\n","decoded = layers.Dense(128, activation='relu')(decoded)\n","decoded = layers.Dense(784, activation='sigmoid')(decoded)"],"metadata":{"id":"hFfQ3o8n0x9Z"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# build model\n","autoencoder = keras.Model(input_img, decoded)"],"metadata":{"id":"7uf1vqItps8i"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# compile\n","autoencoder.compile(optimizer='adam', loss='binary_crossentropy')"],"metadata":{"id":"ZSNdxBQopwtx"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# train\n","autoencoder.fit(x=x_train, y=x_train,\n","                epochs=100,\n","                batch_size=256,\n","                shuffle=True,\n","                validation_data=(x_test, x_test))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Qg0uFw9op8Kh","executionInfo":{"status":"ok","timestamp":1655992021948,"user_tz":240,"elapsed":324889,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"0fb4578a-95b4-430e-934f-870b96061ec8"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/100\n","235/235 [==============================] - 4s 14ms/step - loss: 0.2522 - val_loss: 0.1726\n","Epoch 2/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.1554 - val_loss: 0.1388\n","Epoch 3/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.1327 - val_loss: 0.1257\n","Epoch 4/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.1231 - val_loss: 0.1180\n","Epoch 5/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.1171 - val_loss: 0.1133\n","Epoch 6/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.1128 - val_loss: 0.1092\n","Epoch 7/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.1089 - val_loss: 0.1062\n","Epoch 8/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.1061 - val_loss: 0.1038\n","Epoch 9/100\n","235/235 [==============================] - 3s 12ms/step - loss: 0.1039 - val_loss: 0.1017\n","Epoch 10/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.1019 - val_loss: 0.0998\n","Epoch 11/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.1002 - val_loss: 0.0987\n","Epoch 12/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0989 - val_loss: 0.0971\n","Epoch 13/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0977 - val_loss: 0.0965\n","Epoch 14/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0968 - val_loss: 0.0955\n","Epoch 15/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0959 - val_loss: 0.0945\n","Epoch 16/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0951 - val_loss: 0.0940\n","Epoch 17/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0944 - val_loss: 0.0930\n","Epoch 18/100\n","235/235 [==============================] - 3s 15ms/step - loss: 0.0937 - val_loss: 0.0923\n","Epoch 19/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0932 - val_loss: 0.0919\n","Epoch 20/100\n","235/235 [==============================] - 4s 15ms/step - loss: 0.0925 - val_loss: 0.0915\n","Epoch 21/100\n","235/235 [==============================] - 4s 16ms/step - loss: 0.0920 - val_loss: 0.0909\n","Epoch 22/100\n","235/235 [==============================] - 3s 15ms/step - loss: 0.0916 - val_loss: 0.0905\n","Epoch 23/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0911 - val_loss: 0.0899\n","Epoch 24/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0906 - val_loss: 0.0899\n","Epoch 25/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0903 - val_loss: 0.0893\n","Epoch 26/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0898 - val_loss: 0.0887\n","Epoch 27/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0895 - val_loss: 0.0886\n","Epoch 28/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0891 - val_loss: 0.0883\n","Epoch 29/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0888 - val_loss: 0.0879\n","Epoch 30/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0885 - val_loss: 0.0877\n","Epoch 31/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0882 - val_loss: 0.0875\n","Epoch 32/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0880 - val_loss: 0.0869\n","Epoch 33/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0877 - val_loss: 0.0869\n","Epoch 34/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0874 - val_loss: 0.0868\n","Epoch 35/100\n","235/235 [==============================] - 4s 17ms/step - loss: 0.0872 - val_loss: 0.0867\n","Epoch 36/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0870 - val_loss: 0.0864\n","Epoch 37/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0868 - val_loss: 0.0864\n","Epoch 38/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0865 - val_loss: 0.0862\n","Epoch 39/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0863 - val_loss: 0.0857\n","Epoch 40/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0861 - val_loss: 0.0858\n","Epoch 41/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0860 - val_loss: 0.0854\n","Epoch 42/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0858 - val_loss: 0.0852\n","Epoch 43/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0856 - val_loss: 0.0848\n","Epoch 44/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0854 - val_loss: 0.0847\n","Epoch 45/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0853 - val_loss: 0.0847\n","Epoch 46/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0851 - val_loss: 0.0847\n","Epoch 47/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0850 - val_loss: 0.0844\n","Epoch 48/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0849 - val_loss: 0.0847\n","Epoch 49/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0848 - val_loss: 0.0843\n","Epoch 50/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0847 - val_loss: 0.0841\n","Epoch 51/100\n","235/235 [==============================] - 4s 15ms/step - loss: 0.0846 - val_loss: 0.0841\n","Epoch 52/100\n","235/235 [==============================] - 3s 15ms/step - loss: 0.0844 - val_loss: 0.0841\n","Epoch 53/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0844 - val_loss: 0.0837\n","Epoch 54/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0843 - val_loss: 0.0838\n","Epoch 55/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0841 - val_loss: 0.0837\n","Epoch 56/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0840 - val_loss: 0.0838\n","Epoch 57/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0840 - val_loss: 0.0836\n","Epoch 58/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0839 - val_loss: 0.0836\n","Epoch 59/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0838 - val_loss: 0.0834\n","Epoch 60/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0837 - val_loss: 0.0833\n","Epoch 61/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0836 - val_loss: 0.0832\n","Epoch 62/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0836 - val_loss: 0.0834\n","Epoch 63/100\n","235/235 [==============================] - 3s 15ms/step - loss: 0.0835 - val_loss: 0.0830\n","Epoch 64/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0835 - val_loss: 0.0829\n","Epoch 65/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0834 - val_loss: 0.0830\n","Epoch 66/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0834 - val_loss: 0.0829\n","Epoch 67/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0833 - val_loss: 0.0827\n","Epoch 68/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0832 - val_loss: 0.0828\n","Epoch 69/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0832 - val_loss: 0.0831\n","Epoch 70/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0831 - val_loss: 0.0827\n","Epoch 71/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0830 - val_loss: 0.0829\n","Epoch 72/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0830 - val_loss: 0.0827\n","Epoch 73/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0830 - val_loss: 0.0827\n","Epoch 74/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0829 - val_loss: 0.0825\n","Epoch 75/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0829 - val_loss: 0.0828\n","Epoch 76/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0828 - val_loss: 0.0824\n","Epoch 77/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0827 - val_loss: 0.0823\n","Epoch 78/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0827 - val_loss: 0.0822\n","Epoch 79/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0827 - val_loss: 0.0827\n","Epoch 80/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0827 - val_loss: 0.0828\n","Epoch 81/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0826 - val_loss: 0.0823\n","Epoch 82/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0826 - val_loss: 0.0824\n","Epoch 83/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0825 - val_loss: 0.0822\n","Epoch 84/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0825 - val_loss: 0.0820\n","Epoch 85/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0824 - val_loss: 0.0822\n","Epoch 86/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0824 - val_loss: 0.0822\n","Epoch 87/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0823 - val_loss: 0.0825\n","Epoch 88/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0823 - val_loss: 0.0818\n","Epoch 89/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0823 - val_loss: 0.0822\n","Epoch 90/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0823 - val_loss: 0.0821\n","Epoch 91/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0823 - val_loss: 0.0822\n","Epoch 92/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0822 - val_loss: 0.0820\n","Epoch 93/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0822 - val_loss: 0.0820\n","Epoch 94/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0821 - val_loss: 0.0817\n","Epoch 95/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0821 - val_loss: 0.0821\n","Epoch 96/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0820 - val_loss: 0.0817\n","Epoch 97/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0820 - val_loss: 0.0818\n","Epoch 98/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0820 - val_loss: 0.0816\n","Epoch 99/100\n","235/235 [==============================] - 3s 14ms/step - loss: 0.0820 - val_loss: 0.0818\n","Epoch 100/100\n","235/235 [==============================] - 3s 13ms/step - loss: 0.0819 - val_loss: 0.0819\n"]},{"output_type":"execute_result","data":{"text/plain":["<keras.callbacks.History at 0x7f34deb9b110>"]},"metadata":{},"execution_count":13}]},{"cell_type":"code","source":["# predict\n","decoded_imgs = autoencoder.predict(x_test)\n","decoded_imgs.shape"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YRK1UsZ5qdJw","executionInfo":{"status":"ok","timestamp":1655992022783,"user_tz":240,"elapsed":839,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"1f4c7147-78dd-42f8-adda-525bafa910a4"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(10000, 784)"]},"metadata":{},"execution_count":14}]},{"cell_type":"code","source":["# Use Matplotlib (don't ask)\n","n = 10  # How many digits we will display\n","plt.figure(figsize=(20, 4))\n","for i in range(n):\n","    # Display original\n","    ax = plt.subplot(2, n, i + 1)\n","    plt.imshow(x_test[i].reshape(28, 28))\n","    plt.gray()\n","    ax.get_xaxis().set_visible(False)\n","    ax.get_yaxis().set_visible(False)\n","\n","    # Display reconstruction\n","    ax = plt.subplot(2, n, i + 1 + n)\n","    plt.imshow(decoded_imgs[i].reshape(28, 28))\n","    plt.gray()\n","    ax.get_xaxis().set_visible(False)\n","    ax.get_yaxis().set_visible(False)\n","plt.show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":248},"id":"iDhpfRK7p9mQ","executionInfo":{"status":"ok","timestamp":1655992023511,"user_tz":240,"elapsed":524,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"9b181857-243a-46fc-b529-af6aa8d65aff"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 1440x288 with 20 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAABG0AAADnCAYAAACkCqtqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debxN9f7H8c9JhhBFxkxRJGPmBoV0C5GKFM2mfle3QRnqdkvS7T4oqQylUZIU6kooCSXkcs1TIXQyz5lKOb8/evS5n+/X3ts+x977rLP36/nXe/X9nn2Ws/Zae+3V9/P9pmVkZAgAAAAAAACC5bTs3gEAAAAAAACciIc2AAAAAAAAAcRDGwAAAAAAgADioQ0AAAAAAEAA8dAGAAAAAAAggHhoAwAAAAAAEECnZ6ZzWloa64Nnk4yMjLRYvA7HMFvtysjIKBaLF+I4Zh/OxaTAuZgEOBeTAudiEuBcTAqci0mAczEphDwXGWkDJM6m7N4BACLCuQgEBeciEAyci0AwhDwXeWgDAAAAAAAQQDy0AQAAAAAACCAe2gAAAAAAAAQQD20AAAAAAAACiIc2AAAAAAAAAcRDGwAAAAAAgADioQ0AAAAAAEAA8dAGAAAAAAAggE7P7h1AanrkkUc0n3HGGU5bzZo1Nbdr1y7sa4wYMULzvHnznLbRo0ef6i4CAAAAAJCtGGkDAAAAAAAQQDy0AQAAAAAACCAe2gAAAAAAAAQQc9ogYcaNG6c50lw11vHjx8O2de/eXXPz5s2dttmzZ2vevHlztLuIbFa5cmVne82aNZofeOABzS+//HLC9imVFShQQPOgQYM023NPRGTRokWa27dv77Rt2rQpTnsHAACQPc4++2zN5cqVi+pn/Huihx56SPOKFSs0f/fdd06/pUuXZmUXkUQYaQMAAAAAABBAPLQBAAAAAAAIIMqjEDe2HEok+pIoWxLz2Wefaa5YsaLTr3Xr1porVarktHXq1Enzs88+G9XvRfa7+OKLnW1bHpeenp7o3Ul5pUqV0ty1a1fNftli3bp1NV933XVO27Bhw+K0d7Dq1KmjeeLEiU5bhQoV4vZ7//KXvzjbq1ev1vzjjz/G7ffi5OxnpIjIpEmTNN93332aX3nlFaff77//Ht8dS0LFixfX/MEHH2ieO3eu02/kyJGaN27cGPf9+lPhwoWd7SuuuELztGnTNB87dixh+wTkBK1atdLcpk0bp61Jkyaazz///Khezy97Kl++vOa8efOG/blcuXJF9fpIXoy0AQAAAAAACCAe2gAAAAAAAAQQ5VGIqXr16mm+4YYbwvZbuXKlZn+44a5duzQfPHhQc548eZx+8+fP11yrVi2nrWjRolHuMYKkdu3azvahQ4c0f/TRR4nenZRTrFgxZ3vUqFHZtCfIrGuuuUZzpCHWseaX4Nxzzz2ab7nlloTtB/5gP/uGDx8ett/QoUM1v/nmm07bkSNHYr9jScauGiPi3tPYUqTt27c7/bKrJMqu8CfiXutteeu6deviv2M5TKFChZxtW3JfvXp1zf4qppSaBZudVqFHjx6abSm4iMgZZ5yhOS0t7ZR/r79KKhAtRtoAAAAAAAAEEA9tAAAAAAAAAoiHNgAAAAAAAAGUrXPa+EtA2zrCLVu2OG1Hjx7VPGbMGM3btm1z+lGPm73sEsF+7aet+bbzL2zdujWq13744Yed7Ysuuihs308//TSq10T2szXhdhlaEZHRo0cnendSzv3336+5bdu2TluDBg0y/Xp2KVkRkdNO+9//G1i6dKnmr776KtOvDdfpp//vI7xly5bZsg/+XBk9e/bUXKBAAafNzlGF+LDnX5kyZcL2Gzt2rGZ7f4XwzjnnHM3jxo1z2ooUKaLZziX0t7/9Lf47Fsbjjz+u+bzzznPaunfvrpn75hN16tRJ8zPPPOO0lS1bNuTP+HPf7N69O/Y7hpix18cHHnggrr9rzZo1mu13IcSOXXLdXqtF3DlW7TLtIiLHjx/X/Morr2j+5ptvnH5BuE4y0gYAAAAAACCAeGgDAAAAAAAQQNlaHjVw4EBnu0KFClH9nB3W+fPPPzttiRx2lp6ertn/tyxcuDBh+xEkn3zyiWY7VE3EPVZ79uzJ9Gv7y8fmzp0706+B4Lnwwgs1++UU/hB0xN4LL7yg2Q4Tzaobb7wx7PamTZs0d+jQwennl9ng5Jo2bar5kksu0ex/HsWTv/SxLVvNnz+/00Z5VOz5y7v//e9/j+rnbOlpRkZGTPcpWdWpU0ezP8Te6t+/fwL25kTVqlVztm1J+UcffeS08dl6IlsuM2TIEM1FixZ1+oU7X15++WVn25Z7Z+WeF9HxS2FsqZMtcZk2bZrT75dfftG8f/9+zf7nlL0v/fzzz522FStWaP722281L1682Ol35MiRsK+P6NnpFETcc8zea/rviWg1bNhQ82+//ea0rV27VvOcOXOcNvue+/XXX7P0u6PBSBsAAAAAAIAA4qENAAAAAABAAPHQBgAAAAAAIICydU4bu8S3iEjNmjU1r1692mmrWrWq5kh1xY0aNdL8448/ag63RF8oto5t586dmu1y1r7Nmzc726k6p41l56/Iql69emmuXLly2H62ljTUNoKrd+/emv33DOdRfEyZMkWzXZI7q+zSpgcPHnTaypcvr9kuO7tgwQKnX65cuU55P5KdX89tl21ev3695n/+858J26frr78+Yb8LJ6pRo4azXbdu3bB97b3N1KlT47ZPyaJ48eLO9k033RS2b+fOnTXb+8Z4s/PYfPHFF2H7+XPa+PNBQuSRRx7RbJdwj5Y/T9u1116r2V823M5/E885MJJVpHlmatWqpdku9eybP3++Zvu9cuPGjU6/cuXKabZzmYrEZh5AnMg+D+jRo4dm/xwrVKhQyJ//6aefnO2vv/5a8w8//OC02e8gdm7FBg0aOP3sNaFly5ZO29KlSzXbZcNjjZE2AAAAAAAAAcRDGwAAAAAAgADK1vKoGTNmRNy2/KXa/uQvN1q7dm3NdphT/fr1o96vo0ePav7uu+80+yVbdqiUHZqOU3Pddddptktn5smTx+m3Y8cOzY8++qjTdvjw4TjtHU5VhQoVnO169epptuebCEsjxsqVV17pbFepUkWzHd4b7VBff/inHZ5sl84UEWnWrJnmSMsR/9///Z/mESNGRLUfqebxxx93tu0QcTsU3y9RizX72ee/txgunliRSnZ8fhkBInv++eed7dtuu02zvb8UEfnwww8Tsk++xo0bay5RooTT9vbbb2t+9913E7VLOYYt3RURufvuu0P2W7ZsmbO9fft2zc2bNw/7+oULF9ZsS69ERMaMGaN527ZtJ9/ZFOff/7/33nuabTmUiFseHKlk0PJLoix/+gvE3quvvups27K2SMt32+cGy5cv1/zYY485/ez3et+ll16q2d6Hvvnmm04/+3zBXgNERIYNG6Z5woQJmmNdKstIGwAAAAAAgADioQ0AAAAAAEAAZWt5VCzs3bvX2Z45c2bIfpFKryKxQ4/9Uiw7FGvcuHFZen2cyJbL+EMiLfs3nz17dlz3CbHjl1NYiVx1I9nZMrT333/faYs03NSyq3nZIZ9PPfWU0y9SOaJ9jW7dumkuVqyY02/gwIGa8+XL57QNHTpU87Fjx06220mlXbt2mv0VC9atW6c5kSut2TI3vxxq1qxZmvft25eoXUpZV1xxRdg2f1WaSOWJOFFGRoazbd/rW7ZscdriuQLQGWec4Wzbof9//etfNfv7e88998Rtn5KBLXcQETnzzDM129Vm/HsW+/l06623avZLMipVqqS5ZMmSTtu///1vzS1atNC8Z8+eqPY9FRQsWFCzPwWCnUZh165dTttzzz2nmakSgsO/r7OrNnXp0sVpS0tL02y/F/il84MGDdKc1ekUihYtqtmuYtqvXz+nn52mxS+tTBRG2gAAAAAAAAQQD20AAAAAAAACiIc2AAAAAAAAAZTj57SJh+LFi2sePny45tNOc59x2eWoqUPNuo8//tjZ/stf/hKy3zvvvONs+8vfImeoUaNG2DY7rwlOzemn/+/yHu0cNv7cULfccotmv248WnZOm2effVbz4MGDnX758+fX7L8PJk2apHn9+vVZ2o+cqn379prt30jE/XyKNztHUqdOnTT//vvvTr8BAwZoTrX5hxLFLlFqs8+v8V+yZEnc9inVtGrVytm2y6nbuZz8ORiiZedRadKkidPWqFGjkD8zfvz4LP2uVJU3b15n284J9MILL4T9Obt88FtvvaXZXqtFRCpWrBj2NexcK/GcDykna9u2rea+ffs6bXYZbrvsvYjI/v3747tjyBL/OtarVy/Ndg4bEZGffvpJs51bdsGCBVn63XaumrJlyzpt9rvllClTNPvz2Fr+/o4ePVpzPOfyY6QNAAAAAABAAPHQBgAAAAAAIIAojwqhR48emu2ytP7y4mvXrk3YPiWbUqVKafaHd9shq7Ykww67FxE5ePBgnPYOsWaHc999991O2+LFizVPnz49YfuEP9ilov0lYrNaEhWOLXOyJTYiIvXr14/p78qpChcu7GyHK4UQyXrpRVbY5dptud3q1audfjNnzkzYPqWqaM+VRL4/ktGLL77obDdt2lRz6dKlnTa79LodOt+mTZss/W77Gv5S3taGDRs0+0tOIzK7XLfPlr/5Jfzh1KtXL+rfPX/+fM3cy4YWqfTT3jemp6cnYndwimyJksiJpdXWb7/9prlhw4aa27Vr5/S78MILQ/78kSNHnO2qVauGzCLufW6JEiXC7pO1fft2ZztRZeGMtAEAAAAAAAggHtoAAAAAAAAEEOVRInLZZZc52/4s5X+yM5mLiKxYsSJu+5TsJkyYoLlo0aJh+7377ruaU23VmGTSvHlzzUWKFHHapk2bptmuyoDY8Ve+s+zQ03izQ/79fYq0j/369dN8++23x3y/gsRf0eTcc8/VPHbs2ETvjqpUqVLI/87nYOJFKsOIxcpF+MOiRYuc7Zo1a2quXbu203bttddqtqui7Ny50+k3atSoqH63XY1k6dKlYfvNnTtXM/dImeNfT20pmy1B9Esw7AqYN9xwg2Z/tRl7LvptXbt21WyP9apVq6La91Tgl8JY9nx78sknnbZ///vfmlkxLzi+/PJLZ9uWUtvvCCIi5cqV0/zSSy9pjlQqasut/FKsSMKVRB0/ftzZ/uijjzTff//9TtvWrVuj/n2ngpE2AAAAAAAAAcRDGwAAAAAAgADioQ0AAAAAAEAAMaeNiLRs2dLZzp07t+YZM2ZonjdvXsL2KRnZeuE6deqE7Tdr1izNfq0qcqZatWpp9mtSx48fn+jdSQn33nuvZr82N7u0bt1a88UXX+y02X3099fOaZPsfv75Z2fb1uTbOTVE3Pmh9uzZE9P9KF68uLMdbn6BOXPmxPT3IrTLL79cc8eOHcP2279/v2aWwo2tvXv3avaXtrfbffr0OeXfVbFiRc12LjAR95rwyCOPnPLvSlVffPGFs23PHTtvjT/PTLh5NfzX69Gjh+bJkyc7bRdccIFmOz+G/dxOdcWKFdPs3xPYud+eeOIJp+3xxx/X/Morr2i2y6yLuPOmrFu3TvPKlSvD7lO1atWcbfu9kOttZP4y3HY+qLPOOstps3PL2nlnd+/e7fTbvHmzZvuesN85REQaNGiQ6f0dOXKks/3YY49ptvNVJRIjbQAAAAAAAAKIhzYAAAAAAAABlLLlUWeccYZmu3SciMivv/6q2ZbnHDt2LP47lkT8pbzt0DJbguazQ38PHjwY+x1DQpQsWVJz48aNNa9du9bpZ5fRQ+zYUqREskOaRUQuuugizfYaEIm/TG4qXXv9IcR2Gd+bbrrJafv00081Dx48ONO/q3r16s62LcmoUKGC0xauJCAopXfJzn6ennZa+P/fNn369ETsDuLMlnz4554tv/KvlYieX1J68803a7Zl24ULFw77Gi+//LJmvyzu6NGjmidOnOi02fKPa665RnOlSpWcfqm8jPtzzz2nuWfPnlH/nL0+/vWvfw2ZY8Wef3Zqh1tuuSXmvyuZ+eVG9vzIinfeecfZjlQeZUvS7fvs7bffdvrZJcWzCyNtAAAAAAAAAoiHNgAAAAAAAAHEQxsAAAAAAIAAStk5bXr16qXZX3p22rRpmufOnZuwfUo2Dz/8sLNdv379kP0+/vhjZ5tlvpPDXXfdpdkuHzx16tRs2Bskyt///ndn2y57GsnGjRs133nnnU6bXdYx1djrob/0b6tWrTSPHTs206+9a9cuZ9vOnXHOOedE9Rp+3TfiI9yS6/5cAK+++moidgcx1r59e2f7jjvu0GznXBA5cdlbxIZdstuebx07dnT62XPOzj1k57DxPf3008521apVNbdp0ybk64mc+FmYSuy8JuPGjXPa3nvvPc2nn+5+lS1btqzmSPN/xYKdw8++Z+yy4yIiAwYMiOt+QKR3796aMzOn0L333qs5K/dRicRIGwAAAAAAgADioQ0AAAAAAEAApUx5lB1GLiLyj3/8Q/OBAwectv79+ydkn5JdtEv03Xfffc42y3wnh/Lly4f873v37k3wniDepkyZorlKlSpZeo1Vq1ZpnjNnzinvU7JYs2aNZrskrYhI7dq1NZ9//vmZfm27rK1v1KhRznanTp1C9vOXKEdslClTxtn2SzT+lJ6e7mwvXLgwbvuE+GnRokXYtsmTJzvb//3vf+O9OynPlkrZnFX+ddKW+9jyqKZNmzr9ihQpotlfojzZ2SWW/eta5cqVw/7cVVddpTl37tya+/Xr5/QLN2VDVtny5bp168b0tRFaly5dNNuSNL9kzlq5cqWzPXHixNjvWJww0gYAAAAAACCAeGgDAAAAAAAQQEldHlW0aFHNL730ktOWK1cuzXZov4jI/Pnz47tjcNjhnyIix44dy/Rr7N+/P+xr2OGRhQsXDvsaZ511lrMdbXmXHcLZp08fp+3w4cNRvUYyuu6660L+908++STBe5Ka7FDdSCsoRBqWP3LkSM2lS5cO28++/vHjx6PdRUfr1q2z9HOpbMmSJSFzLGzYsCGqftWrV3e2V6xYEdP9SFWXXnqpsx3uHPZXX0TO5F+HDx06pPn5559P9O4gzj744APNtjyqQ4cOTj87fQBTN0RnxowZIf+7LScWccujfvvtN81vvfWW0++1117T/OCDDzpt4cpWER8NGjRwtu21sWDBgmF/zk67YVeLEhH55ZdfYrR38cdIGwAAAAAAgADioQ0AAAAAAEAA8dAGAAAAAAAggJJuThs7V820adM0n3feeU6/9evXa7bLfyPxli1bdsqv8eGHHzrbW7du1VyiRAnNfr1wrG3bts3ZfuaZZ+L6+4Lk8ssvd7ZLliyZTXsCEZERI0ZoHjhwYNh+djnZSPPRRDtXTbT9Xnnllaj6IXvYOZFCbf+JOWziw87J59u1a5fmF198MRG7gziwcyvY+xQRkR07dmhmie/kYz8n7efz9ddf7/R78sknNb///vtO23fffRenvUtOn3/+ubNt78/tEtFdu3Z1+p1//vmamzRpEtXvSk9Pz8Ie4mT8uQ/PPPPMkP3snGAi7rxR33zzTex3LEEYaQMAAAAAABBAPLQBAAAAAAAIoKQrj6pUqZLmunXrhu1nl3O2pVKIHX8pdX/YZyy1b98+Sz9nl/mLVNYxadIkzQsXLgzb7+uvv87SfiSDG264wdm2pYqLFy/W/NVXXyVsn1LZxIkTNffq1ctpK1asWNx+786dO53t1atXa+7WrZtmW8KI4MnIyIi4jfi65pprwrZt3rxZ8/79+xOxO4gDWx7ln1+ffvpp2J+zJQFnn322Zvu+QM6xZMkSzU888YTTNmjQIM3//Oc/nbbbb79d85EjR+K0d8nD3ouIuMuu33zzzWF/rmnTpmHbfv/9d832nO3bt29WdhEh2Otd7969o/qZMWPGONuzZs2K5S5lG0baAAAAAAAABBAPbQAAAAAAAAKIhzYAAAAAAAABlOPntClfvryz7S/p9id/Tge7zC3i48Ybb3S2bS1i7ty5o3qNatWqac7Mct1vvvmm5o0bN4btN2HCBM1r1qyJ+vXxh/z582tu2bJl2H7jx4/XbGuAET+bNm3SfMsttzhtbdu21fzAAw/E9Pf6y9wPGzYspq+PxMiXL1/YNuZPiA/7uWjn5/MdPXpU87Fjx+K6T8ge9nOyU6dOTttDDz2keeXKlZrvvPPO+O8Y4uqdd95xtrt3767Zv6fu37+/5mXLlsV3x5KA/7n14IMPai5YsKDmevXqOf2KFy+u2f8+MXr0aM39+vWLwV5CxD0eq1at0hzpu6M9B+yxTSaMtAEAAAAAAAggHtoAAAAAAAAEUI4vj7JLyIqIlCtXLmS/2bNnO9ssX5p4AwcOPKWf79ixY4z2BLFih+bv3bvXabPLpL/44osJ2yecyF9m3W7bklL/etq6dWvN9niOHDnS6ZeWlqbZDmVFznX33Xc72/v27dP89NNPJ3p3UsLx48c1L1y40GmrXr265nXr1iVsn5A9unTporlz585O2xtvvKGZczG57Ny509lu3ry5Zr80p0+fPpr9Ejqc3Pbt2zXbex27lLqISKNGjTQ/9dRTTtuOHTvitHeprVmzZprLlCmjOdJ3d1s2akuIkwkjbQAAAAAAAAKIhzYAAAAAAAABlJaZMqG0tLRA1BRdfvnlmqdMmeK02RmnrQYNGjjb/tDjoMvIyEg7ea+TC8oxTFGLMjIy6p2828lxHLMP52JS4Fw8iU8++cTZHjx4sOaZM2cmendCSuZzsXTp0s72gAEDNC9atEhzEqzOlrLnor2XtSsBibglrCNGjHDabCnyr7/+Gqe9y5xkPheDwl8d95JLLtHcsGFDzadQopyy52IySYZzcenSpZpr1KgRtt+gQYM023LBJBDyXGSkDQAAAAAAQADx0AYAAAAAACCAeGgDAAAAAAAQQDlyye/GjRtrDjeHjYjI+vXrNR88eDCu+wQAQLKwS6Ai8bZs2eJs33PPPdm0J4iXOXPmaLZL3AKhtGvXztm2836cf/75mk9hThsgEIoUKaI5Le1/U/T4S6wPGTIkYfsUBIy0AQAAAAAACCAe2gAAAAAAAARQjiyPisQOF7zqqqs079mzJzt2BwAAAACy7MCBA872eeedl017AsTX4MGDQ+ann37a6bd169aE7VMQMNIGAAAAAAAggHhoAwAAAAAAEEA8tAEAAAAAAAigtIyMjOg7p6VF3xkxlZGRkXbyXifHMcxWizIyMurF4oU4jtmHczEpcC4mAc7FpMC5mAQ4F5MC52IS4FxMCiHPRUbaAAAAAAAABBAPbQAAAAAAAAIos0t+7xKRTfHYEURUPoavxTHMPhzHnI9jmBw4jjkfxzA5cBxzPo5hcuA45nwcw+QQ8jhmak4bAAAAAAAAJAblUQAAAAAAAAHEQxsAAAAAAIAA4qENAAAAAABAAPHQBgAAAAAAIIB4aAMAAAAAABBAPLQBAAAAAAAIIB7aAAAAAAAABBAPbQAAAAAAAAKIhzYAAAAAAAABxEMbAAAAAACAAOKhDQAAAAAAQADx0AYAAAAAACCAeGgDAAAAAAAQQDy0AQAAAAAACCAe2gAAAAAAAAQQD20AAAAAAAACiIc2AAAAAAAAAcRDGwAAAAAAgADioQ0AAAAAAEAA8dAGAAAAAAAggHhoAwAAAAAAEEA8tAEAAAAAAAig0zPTOS0tLSNeO4LIMjIy0mLxOhzDbLUrIyOjWCxeiOOYfTgXkwLnYhLgXEwKnItJgHMxKXAuJgHOxaQQ8lxkpA2QOJuyewcAiAjnIhAUnItAMHAuAsEQ8lzkoQ0AAAAAAEAA8dAGAAAAAAAggHhoAwAAAAAAEEA8tAEAAAAAAAggHtoAAAAAAAAEEA9tAAAAAAAAAoiHNgAAAAAAAAF0enbvAJLXaae5zwQrVKiguUOHDpqbNWvm9CtYsKDmX3/9VfMvv/zi9Pv88881v/76607bvn37Mr/DCJxcuXJpzpMnj+YjR45kx+6knLS0NM32fM6dO3fYfkePHnXaMjIy4rR3AAAAQPJjpA0AAAAAAEAA8dAGAAAAAAAggHhoAwAAAAAAEEDMaYOYOv30/72lGjVq5LQ98cQTmuvUqaO5cOHCTj87d4adK8NmEZGrr75ac8+ePZ22unXrat66dWtU+47sYY9rt27dnLZHHnlE85IlSzTfcccdTj/muImPM888U/NNN92kuVevXk6//fv3a+7bt6/T9tVXX2lmfpv4iTTn0G+//ab5+PHjmrN6POw5a6/5IiJnnHGG5mPHjjltdr4j3guJd9ZZZ2nOly+f5h07djj97HsEAHBy9nMxb968TtvZZ5+tuXz58k6b/QzduXOn5j179jj9du3apZnPz9TESBsAAAAAAIAA4qENAAAAAABAAFEehVPilyzVqlVL84gRI5y24sWLa7ZD+/yh2Rs2bNBsh3PbJcNFRAoUKBCyn4hI69atNY8cOTLs/iP72fdQlSpVnLZixYqF7OcvK43Y8M/nK6+8UvO//vUvzUWKFHH62TIYv1TxP//5j+bDhw/HZD/xBzusulSpUprPPfdcp196erpmO8T6l19+cfpFO+TalmL5196LLrpI8/fff++0bdu2LdO/C5ljz+EuXbo4bc8884xmWzZ86aWXOv0OHToUp71LHvb+Q8Q9/+zfz5aOirilvIk8B/wyxjx58miOVLbIeYpU498HXXDBBZqbNGnitNlrp/3+kytXLqdfoUKFNBcsWNBps+WomzZt0vzFF184/ex3qs2bN4fdfyQvRtoAAAAAAAAEEA9tAAAAAAAAAihby6PsEOuTsUM0Ga4ZHP4M6Z06ddLsD5u3Q3BHjRql2S+j2rdvn2Y7pLdhw4ZOP/tzJUuWdNpsWcdrr72mmfdO8NhhpH5Zhx3CP3DgQM0cx/ioWLGisz1kyBDNdvUD/9ptVyuyK8OJiNSvX18zK0nFll0BqEWLFiH/u4i7etTevXtP+ffa63LLli2dNlua+txzzzltrOQXf7Z08amnngrbZj+P/VW+8Ae/TMJeA++8806nrXHjxppticPkyZOdfrZk0K4O46+A+Pvvv7prehMAAB04SURBVGd6f/2SDFteXL16daft4MGDmpctW6bZL5nMyn4kG7sinojIjTfeqNmW/E6dOtXpRxl3sPjns72PKVq0qGZ/SgX7GeevzGjZz1n/mmo/M/37J3uO2Wkk/HPWXr8pj8o6/30QThDvURlpAwAAAAAAEEA8tAEAAAAAAAggHtoAAAAAAAAEUELmtLF1trZer3///k4/W7/nLw27ZcsWzbNmzdK8aNEip9/27dtDvob/erb20Gfr3Wz264UjvZ5dwi2Z+XMn2BrAnTt3Om1vvvmm5tdff11ztHW/CxcudLZtHahfc2zrSYNYl4j/sct816tXz2mzy0UvXrw4YfuUSmyd9Jdffum0lS5dWnOk65+9TvpzWT366KOa7fV5zZo1md/ZFOfXwteuXVtzq1atNPtLbU+ZMkWznbMiFtfG888/39m253CNGjWctm+//faUfx9cfn3+gw8+qNneb/l9d+zYofnXX3+N097lbP41z87X1aFDB6fNnlejR4/WvGLFCqffgQMHNNv7lFjcM9p5OURE+vXrp9ley0Xca729t2IOmz+ceeaZmpcsWeK02b+lvX/t2bOn0+/tt9/WzH1o9rBz0Pjnh50n087/Va1aNaefnY/GZ89bOy+VnddKxP2OaOeTEhHZsGGD5vHjx2tetWqV089/TbjX6DJlymju0qWL08/OS+R/LtrPvx9++EHzjBkznH6TJk3S/N133zltiZoXjpE2AAAAAAAAAcRDGwAAAAAAgACKS3mUP1zXbtshh3Y5QhG3TMIfjmaHbF522WWad+/e7fSzQ+Fs6U6hQoWcfvb1/SUOww1z8v9dS5cu1fziiy86bXZp22QebmqH+oq4y3BPnDjRabNDcP2/eTQKFizobNvlof2yAVtWg2Dxz6NIy8RPmzZNM8vSxk6BAgU0z507V3PZsmWdfvZY2WHA/lB+O/Q7T548TptdCnf69Omar7rqKqefP9wUJ8qfP7+z3blzZ812yK8tRRVxh1VHKg2Oln3/NGnSJOw+2iWSRSgRiIe8efM627fddptmv7zH3osMGjQovjuWBPzlfa+++mrNhQsXdtpsWcOCBQs022W9RdzzLxbngz3+vXv3dtrat2+v+dChQ06bLeWJxTUhGdjzZebMmZrPO+88p1+4KRT8a6Etr0hPT3faUmUKhXixf3f/+6It+bb3NH6JoC3ltd8lf/rpJ6effV/4Zd1z5szRbJd8t2U2Iu611z/ffv75Z818Rp7IXoevu+46p+2ZZ57RbL8T2nsUkRO/I1r2b25LrOrXr+/0syWxw4cPd9rGjh2r2R7PWGOkDQAAAAAAQADx0AYAAAAAACCAeGgDAAAAAAAQQHGZ08avybPbtrZ+8ODBTr9du3ZpLlGihNNmaxQ3b96s2a8LveCCCzSXLFlSc6Taen9eFltvaJeS9muYbe3+5MmTnbbZs2dLKvD//uvXrw+ZRbJWq2nnsRk2bJjTZo+NXWpP5MQ5HRAc/hwMjRo10uzPUfXZZ58lZJ+SnT+3xfvvv6/ZXjP9+YbsOWvnFPLnpIo0p42tN7dLbo4cOdLpd8MNN2jeu3dviH9FarLH5NJLL3Xa7HxBy5cv1zxv3jynXyzmrLD7UapUKc3+kt/2M8FfQh6xZ+v4Rdy5jfzPXPs5aZeBR2jlypVztq+99lrN/hx79v51//79mv1zLxZzVtjruf38vOOOO5x+9h7VLj8tcuL9WSryP+8GDBig2S7vHulz0X4W1q5d2+k3ZMgQzXaOHBGRt956S7O/BDRO5M9JYu9batas6bTZc27lypWa/e8JW7Zs0Txu3DjN/ndCe19ql4cWcT/vknn+0njz5yWy55Kdf80u0y7iXgvDLb/ut/nHyZ7f9nz2v9/aZwoPPPBA2DZ73vvz25zq9Z+RNgAAAAAAAAHEQxsAAAAAAIAAikt5lM8ORbL566+/dvotWrRIs12uW8RdKtwOPfWHGtl+tsTKX7rLDmVdu3at02aH9/fo0UNz27ZtnX47duzQvGrVKqctVZfzy8rQL3/Yox1mZpelbN68udPPDml79913nTZ/aByCwx5fEZFq1app9pfF9JdLRdbYYf0iIi1bttQcaSnEo0ePal68eLHmnTt3Ov1sSYZfNmCH6Nsy11q1ajn9+vXrp7lPnz5h9yPV2M+0Ll26OG12WcvPP/9c8759+5x+sS7J6Nixo+azzjrL6ffjjz9q/v7770/59+JE9rPPP7dt2bDPLk/rLwGNP9i/rf9ZZcvs/ZJTe45FW47ol92Eez3/Gm2XoB46dKhme30Vce9t+/fvH/b1U5U/DcNDDz2k2R4b/2+1bds2zfZ7i11CWsS9t7nmmmucNvv+efnllzVTYvM/9hg0a9bMabv//vs1+8fH/j1tCZRf1m1f356zqfr9LdHs9fSxxx5z2uy9jr3POXz4sNNvyZIlmm1544YNG5x+9nOxQoUKTluNGjU023PYP59teb8/5Uq7du0029LjpUuXOv3s+yxL35cz/RMAAAAAAACIOx7aAAAAAAAABFBCyqPC8YcG2eG6/tBdWyYRaUiRHRZuh2nbIYw+f4iqHd5vh+Xb1VNERFavXq152bJlYV8fJ7LDff3VL+68807Ndkipf5zWrVun+b777ov1LiJO7rrrLme7UKFCmm3JociJ5xyiZ8s8/ZXXwpVE+SsjvPbaa5pfeOGFkK/tb/urg1144YWa7777bs1169Z1+nXq1EmzX2766quvhtzfVHDRRRdptsPtRdxVtuyQ3HicN7bMrVWrVpr995IdDuyvnIDYsKUVtkxAJPyQfxH3HKYEIDT7frarpIm4pYD+NfDiiy/WbIfiR1o9yh4D//7GHmP/Hun111/XXLlyZc1+Wbi93tppBVKZPb59+/YN22ZLaaZOner069Wrl2ZbQuf3s6/nv1/sdR2h2fPPriAk4pa4/Oc//3HabFmu/S5JSWD28s8BO/1Ft27dnDZbEmWvXY8++qjTz67CFum+x65OZadHEXG/g1x11VWa/XJTu0/+e8k+N7BlVP7nLKtHAQAAAAAAJCEe2gAAAAAAAAQQD20AAAAAAAACKFvntMmMU60Di/Tzfi1x6dKlNds6Zb9ebvr06ZpZYjpzbM23XT5WxF3az9ZA2jmERES6d++umblPgi137tya27ZtG7bf/PnznW3mXcg6WzPvz81g2XPHnzvG1vz7y2WG419P7fw0tnbYXltFRAoXLqz58ccfd9rGjBmj+eDBg1HtR05lzxUR93yxtdIiIpMmTdJslzaNR+2+fQ/ZpXL9eZA++eQTzZy/8VGmTBnNZcuWDdvPX/p91qxZ8dqlpGHfs/6yrvZ+xM6RIOIuD9u6dWvNdklaEXceKjvfjb02iohUqVJF84MPPui02fnA7LyLQ4YMcfotXLhQ4LJzczVs2NBp27lzp2a7RPobb7zh9LPLctsliDNzvbPzfXGdDK1ly5aa7fkg4n5O+te5rVu3as7KZ6F/D8NcOLFh7/FERFq0aKHZfif0LViwQPO4ceOctnDf/fzrs52PrE6dOk5bhw4dNNv7Zn+f7PvCP2ftHLrr16/XbK8VscBIGwAAAAAAgADioQ0AAAAAAEAA5ZjyqFizw5zscEkRdzm/qlWrarbLyIm4w8BjPQQq2V1yySWa77nnHqfNDmPbvHmz5pdeesnp5y8LjOCqVKlSyCwicuDAAc3Dhw9P2D4lG39Ib5s2bcK22evV7NmzNftLoMai7NOWVdlhy/5S0XaJ22LFijlttWvX1jxnzpxT3qcgO/PMM51tu8y3PxT4s88+0+wvLXyq/PfMtddeq9mWjdiSAhGRL774Iqb7gT/Y86Vr166a8+bN6/Szw7ZfeeUVp80vZcOJbCnEtm3bnDb7WeUPnbdl9XZ5YntdE3HLYnbs2BH2d9n7IHsN8F8zPT1ds3+PRNnNiex1bdmyZU6bXQr4448/1uzf39vXsNdCW6omEnmJYNtmj2esr+M5jb3O2SXr/fPN/j390kL/3iIc/zMu1GsjdvzPKntM/WuVPTbFixfXfM011zj97Dlsv1vUr1/f6de4cWPN1atXd9rsct22BDbSfbNdhlxE5P3339e8ceNGiRdG2gAAAAAAAAQQD20AAAAAAAACKGXKo/xhTnY4op0tWsSd0doOJ3733Xedftu3b4/lLiY9OzTuH//4h2Y7rNhnh6D55VD22DDbe/DYc6xHjx6a/WGun3/+uWa/1ALR84cE+ytjWHYFE7tSk10JI1bsuWlLpfyyVNvPLynIly9fzPcrqPwhxHbVJr9czQ7njzX/b37rrbdqtu81f1U/Phfjw54vdoi4f97bc3jo0KHx37EkNm3aNGd7wIABmv0yX3tM7HB+v7TGltDYVUZ++uknp59dXc8O3/fZlfXsdR2h2fIjf8qDRo0aaf7Xv/6lecKECU4/W8J64403avY/t+yx9+9Jr7jiCs12Nbh4llbkNOeee27YNnu/UL58eafNllOvWLFCc6QyN/u+8PvZY+cfR75rRM+/v7crMPslbrZE3k5TMnLkSKdfuBI3fxVOW/YUqXzOHnv/fuuHH37Q/Oijjzpt9ntMPEscGWkDAAAAAAAQQDy0AQAAAAAACCAe2gAAAAAAAARQysxp4zv99P/90/v16+e0FS5cWPOXX36p+Y033nD6UcuYOXYJNluj6NuwYYNmO9/GypUrnX6R6gZZyi/72RrSK6+8UrO/tN8LL7ygmeOTdfaaJiJSuXJlzf7fdc2aNZqXLl0a0/3wzz1b/2/fB379v+XXlPvzpiSzSDXz/t+sXLlymu2cJ5GWarev4c8vZevAO3Xq5LTZa7bdJ3+Jb//YITbssbJzYPjvly1btmhmjpNT459Hdkltf14Eu22vxf59Srjz2Z/br3///iF/RsSd/2b48OGaWeL75OxciEWKFHHa7PXUznXZsWPHsK8Rbgl3EXfOMbs0uIhI0aJFNffp00fzww8/7PSLxzxzQRZu/phIn4slSpRw2vr27at53rx5mv3vEPYzzZ7r9piKiBw6dEizP4/c5s2bQ+ZUX7o9FDunoYjII488onngwIFOmz3nrr/+es32e4WIe6211+DzzjvP6Wfn6PPvUe11Mz09XfMzzzzj9Hvvvfc0Z9d5yUgbAAAAAACAAOKhDQAAAAAAQAClTHmUP7SuQoUKmi+55BKnzQ6Fe/755zX7Q+YQWYECBZztESNGaLYlaFu3bnX69ezZU/PChQs1Z2bYPWU22c8u2WeH8/tD9pcvX56wfUpm/rBRWy4TrlxQJGvniv96doiqXe5WRKRly5aab7/9ds1+eYHdD7sUrkhqLQW/f/9+Z3v+/Pma27Zt67Tde++9mps1a6bZ//vZc9F+9vlDfO252aJFC6fNLjtsr8XLli078R+BmLNLQNuyDv/8/frrrzUzRD9+/FIkux3t392eR/59kC1VtMuEi7hl43v27Inqd+EP9m8+bNgwp+3mm2/WbMt6/c+7AwcOaH799dc1T506NWy/KlWqOG32PrdWrVqa7XVcRGTy5Mkh/hXJy17P7PLOtuRJxD0//OWdGzRooLlu3bqa/fJie4zt7z148KDTz5b1+J/PtlTxww8/1Dx69GinH2XDJ7LXSVvWKyLy3HPPaR48eLBm/1y0x9SWyY0ZM8bpd/nll4fdD3ue3nbbbZptaZ1IMMpPGWkDAAAAAAAQQDy0AQAAAAAACCAe2gAAAAAAAARQysxp48+fYOeq8Zc9XbBggea5c+fGd8eSjK03tDW7Iu4SxPZ4LFq0yOln//6R6kDt70rkHDaR5vOItLxnqrnhhhs029rh//73v04/v34YWVOoUCFn217X/OufPRdtHfCPP/7o9LPvX/u+96+Zds6iv/3tb06bnYfFzq3is0tuPvjgg07bsWPHwv5csrHLyYqIjB07VrM/X1DZsmU127nZrr32WqefPf52braZM2c6/ezcGX79v32NSLXoiA3/nLVLQNs5HPxlVIcOHao5CDX4iI6/RO2VV16p2Z+3xp87BVlj5yMREalZs6bmcPcvIu73gh9++EGzv0S8vX9dt26d02bndezatavmO++80+k3Y8aMsK+f7IYMGaLZX2rbzj9UsmRJp83Op2nvOfz7Fv8z7k/+HDn2MzNv3rxOmz1v69Spo/mcc85x+tl5WbguZ06kv5dts+eUnSdKxL1/9V/Pvs/sPDZBPE6MtAEAAAAAAAggHtoAAAAAAAAEUMqURzVt2jTstl+eYYcq+kPVEZldirRXr15Om12SeN++fZpfe+01p58dAmqHiOfLl8/pZ4e7+UtixmKpUzt00i6Ta4dAioiUL19e84svvui0pVJZh182duutt2q2x/HLL790+rEUYmz4S1HaoZ1+qYUdRmpLafxlKm3phR0WXLFiRadf9+7dNdvjLuIOLQ9XYiMiMmXKFM2zZ8922lKpzNA/H7799lvNfulZ6dKlNdsh4qVKlXL6HTp0SPOSJUs0+8sMn3322Zr965wtgbOfi36JAWLDLrEuIlKtWrWQ/bZt2+Zsr1ixIm77hNiyZRgjRoxw2uy9lL/0rD2fETuHDx/WbJcM9j8/7b2Ozf61235u+Z93tsTKLvPtl8lVqlRJc6qd2/a7wFtvveW0TZo0SbP9G4mIdOrUSXPjxo01n3vuuU4/e4219/v2fSDilkf512V7X2TLr/zP6lmzZmleuHChIDbsMbXl3n5Joz0X/Skann76ac1BLImyGGkDAAAAAAAQQDy0AQAAAAAACKCkLo+yw6YmTpzotNkVf/ySgM2bN8d3x5KYXc3EL2eyw0PtTPpr1qxx+tmfs2VJHTp0cPqdddZZmv2SGzvjvh3q6Jfw2KGNtWvXdtrsLP4NGzYM+e8QcYdE7t2712mzQzqDPuzuVPmz6letWlWzHZo4YcKEhO1TKvGH9B44cEBz0aJFnTZ7/bPvc3/49Y4dOzQ3b95c8x133OH0s+eO/z6wx96W1UyfPt3p17lz55D9Up0dcm+Ph4jIzp07NS9btkyzf52zxyBcFnHLTFetWuW0XXbZZSH7UaoRH3YlGxH3c9Eet/Hjxzv9UqkkNyey5+ZFF12k2S9/s8d49+7dTluy30sEjf/3tuVS/rU2WnY1pE2bNmn2V0Kyq+CsXbvWaUulc92/77afhf7KUvY+xq6UWa9ePadfly5dNNuyNH/KDHsu+u+FcKVyfkmdLdeJ9PmMyPLnz+9s29JRf8Uuy66MakvmRHLW9ZSRNgAAAAAAAAHEQxsAAAAAAIAA4qENAAAAAABAACXdnDZ2rgY7r4m//Jdd6vTxxx+P/46liD179mj25zooUKCAZjvHRrdu3Zx+tt7z7rvvDvkzIm4daNu2bZ22BQsWaF6+fLlmWx8sInLhhRdqLlGihNMWbg6BOXPmOP2KFy+u2S4NKJKzaiVPlZ3zRMRdGtHOgeHPYYTY8N9r48aN09yrVy+nzZ5jdj6ad955x+ln3/e21t6vK7b12359tp2f5oMPPtDcp08fp5+dgwfRiTQ/TVbY91Ck+W7s52cqXeMSyS5bK+KeY7/88ovmSOcsgscu812jRg3N9t7V58/ZgcTy5yCxy0NHumZafpudg87eE9n5E0VE6tSpo/mzzz5z2lL5fWH/nv5S6/Zewi7v7N+ft2rVSrOdj9POUyninrM2i7jXYvte+Omnn5x+69ev18ycNplj/15PPvmk01a6dOmQP7N//35n++KLL9Zs72VyGkbaAAAAAAAABBAPbQAAAAAAAAIo6cqjunbtqrlixYqa/SVkH330Uc0sWRo7dtj8+++/77TddNNNms8++2zNPXr0cPrZsiQ7FDHSkEK/tKlZs2aaL7/8cs3+sEc7zNVmEZEjR45otsvA33XXXU4/O2x9y5YtkqqeffZZZ9v+XbZt26bZX5oa8TFgwADNV199tdNml5fNkyePZnvN9EW7tKk/VHnmzJmae/bsqdmWUiIYbIlGuXLlnDZ7PttyY1v2KsJy7afCnmPVq1d32uzQe/s588MPP8R/xxAzefPm1XzBBRdo9ksk7PH2y1Hta9j7FMROuKWcRdzlp6MtbfH72bKab775RrO/9Lst6SlcuLDTtnfvXs3+5y7+EKl8155/9jPN/y5g/7b+55tdenz27NmaR40aFbYfJcWZY0ugOnfuHLafvRa2b9/eadu9e3fsdywbMNIGAAAAAAAggHhoAwAAAAAAEEA8tAEAAAAAAAigHD+nTfny5Z1tO6+Grc/fsGGD0+/DDz+M746lqGPHjmm+//77nbZPP/1Us51zpn79+k4/O6+GnS/BryW1NcF+veL27dtD/py/5N93332nuVixYk6bnYtj8uTJIX+vSGov12fnQ/HnFbJ/lwkTJmimnjcx7FxdTZo0cdo+/vhjzXbOJ7+WO9p5bOw5Zuu6Rdw5oJjHJthKlSql+ZJLLnHa7Fxj9n1hr/k4NfbvGmlehe+//16z/3mEYLPXSjuPhv0s9bebN2/utNnlaxcuXKjZPxdT+d4kluJxz2KPjb1fnThxotPPzudRs2ZNp82e+3aJaY57aOnp6c62vY7a75KR5s/05620c9e8+uqrmu2y4yLc92aWnX/0k08+0VyoUCGnn/1cnDFjRsicTBhpAwAAAAAAEEA8tAEAAAAAAAigHFkeZZce9YcS2iXx7LCpp556yunHMonx5w/VtSVGNvvClWRkdcinfb1Ir2HfVz6GNoZml2T3l549ePCg5pdffjlh+4QT2WMhItKyZUvNbdq00dy7d2+nny2XsaWFU6dOdfqNHTtW8/Lly502yjeCy7/WFi9eXLN/zbNlHT/++KNmjm/sRFpm2B4Pe//CZ1POYs+jnTt3avbvP2x5vy2jEhFp0aKFZvu5a19PxF2aGpmTyBIjew31y2/sdAF+Cbr9TA73vhKhXOpPtmRcRKRdu3Yhs1+GZkudbGm5iFti5U/hgKxr2LCh5qpVq2r2y4YPHz6s2X7PT9bPRUbaAAAAAAAABBAPbQAAAAAAAAIoR5ZH1a5dW3OtWrXC9tuxY4fmSZMmxXWfEDuxHsoZ7esl63C6eLLDTRs1ahS2H8Nzg8UO4x0/fnzIjOTnn5fLli3TPGzYMKetQoUKmu0qGZRgxI79DHr44YedtltvvVXzmDFjNNsycASfPeemT5+uuVu3bk6/MmXKaPbLW5csWaLZlm7453Okkm/bl8/n7GXPe3+FU7tdo0YNp23FihUhXwPRsWVp9ppqMxLDloOKiDz55JOa7Up6/rXKrhLll+YnI0baAAAAAAAABBAPbQAAAAAAAAKIhzYAAAAAAAABlGPmtLHLX3bs2FGzv/yXre9+4403NPs1wQBii7p4IGc7evSo5v79+ztt9jOY+RPiw15D586d67TNnz9fM3//5LBw4ULN1atXd9o6deqk+auvvnLa1q1bp5k5pZKLfzxXr16t2V+yOm/evJp3796tmXsx5DT58uVztsuXL6/ZnhN2HiIRkVGjRmk+duxYnPYuOBhpAwAAAAAAEEA8tAEAAAAAAAigHFMeZZcDq1Klima7dK2ISHp6uuaRI0dqZjgxAADR8YfYM+Q+e3EPk9z80hd7/4rUceTIEWd748aNmjdt2hT257g+IyezpdkibjlwrVq1NHfr1s3pN2/ePM2p8BnJSBsAAAAAAIAA4qENAAAAAABAAPHQBgAAAAAAIIByzJw2dimv1q1ba86fP7/Tz9bFpUJ9GwAAAIDkxbw1SFb+UvcdO3bMpj0JNkbaAAAAAAAABBAPbQAAAAAAAAIos+VRu0Qk/Jpz2eDw4cPZvQuJUD6GrxW4Y5hCOI45H8cwOXAccz6OYXLgOOZ8HMPkwHHM+TiGySHkcUyjRhIAAAAAACB4KI8CAAAAAAAIIB7aAAAAAAAABBAPbQAAAAAAAAKIhzYAAAAAAAABxEMbAAAAAACAAOKhDQAAAAAAQADx0AYAAAAAACCAeGgDAAAAAAAQQDy0AQAAAAAACKD/B0abAzckeC5+AAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]}]}

docs/notebooks/ex07a - variational autoencoder.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"ex7 - vae.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true,"authorship_tag":"ABX9TyNS2VkU+59d9c7FppBT4a6T"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["## Variational autoencoder (VAE)\n","\n","Variational autoencoders are a slightly more modern and interesting take on autoencoding.\n","\n","<p align='center'>\n","  <img src='https://miro.medium.com/max/1400/0*ndg6glTk5l_ouLON.png' width=700></img>\n","</p>\n","\n","What is a variational autoencoder, you ask? It's a type of autoencoder with added constraints on the encoded representations being learned. More precisely, it is an autoencoder that learns a [latent variable model](https://en.wikipedia.org/wiki/Latent_variable_model) for its input data. So instead of letting your neural network learn an arbitrary function, you are learning the parameters of a probability distribution modeling your data. If you sample points from this distribution, you can generate new input data samples: a VAE is a \"generative model\".\n","\n","How does a variational autoencoder work?\n","\n","First, an encoder network turns the input samples x into two parameters in a latent space, which we will note z_mean and z_log_sigma. Then, we randomly sample similar points z from the latent normal distribution that is assumed to generate the data, via z = z_mean + exp(z_log_sigma) * epsilon, where epsilon is a random normal tensor. Finally, a decoder network maps these latent space points back to the original input data.\n","\n","The parameters of the model are trained via two loss functions: a reconstruction loss forcing the decoded samples to match the initial inputs (just like in our previous autoencoders), and the KL divergence between the learned latent distribution and the prior distribution, acting as a regularization term. You could actually get rid of this latter term entirely, although it does help in learning well-formed latent spaces and reducing overfitting to the training data.\n","\n","Because a VAE is a more complex example, we have made the code available on [Github](https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py) as a standalone script. Here we will review step by step how the model is created.\n","\n","### Loss\n","\n","The loss function of the VAE is defined by two terms, the reconstruction loss and the regularizer which is essentially a KL divergence between the encoder’s distribution and the latent space.\n","\n","$$\\mathcal{L}(\\theta, \\phi) = -\\mathbb{E}_{X \\sim q_{\\theta}}[P_{\\theta}(x|z)] + \\mathcal{D}_\\text{KL}(q_{\\phi}(z|x) || p_{\\theta}(z)) $$\n","\n","where the KL divergence is defined below\n","\n","$$\\mathcal{D}_\\text{KL}(P||Q) = \\sum_x P(x) \\log(\\frac{P(x)}{Q(x)})$$\n","\n","\n","### KL divergence\n","\n","What does KL divergence mean?\n","\n","<p align='center'>\n","  <img src='https://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/Kullback%E2%80%93Leibler_distributions_example_1.svg/600px-Kullback%E2%80%93Leibler_distributions_example_1.svg.png' width=600></img>\n","</p>\n","\n","Solution for $\\mathcal{D}(P||Q)$ is below\n","\n","<p align='center'>\n","  <img src='https://wikimedia.org/api/rest_v1/media/math/render/svg/40cf6731eed13322031049e156e9e7aebb3ae340' width=600></img>\n","</p>\n","\n","and for $\\mathcal{D}(Q||P)$ is below\n","\n","<p align='center'><img src='https://wikimedia.org/api/rest_v1/media/math/render/svg/7c6f1afcdb28ae9d477e2539c2490784fe89ec36' width=600></img></p>\n","\n"],"metadata":{"id":"0Xn0_vxIFBnp"}},{"cell_type":"code","source":["# import\n","import keras\n","from keras import layers"],"metadata":{"id":"oNKhpEW1FodY","executionInfo":{"status":"ok","timestamp":1655999857963,"user_tz":240,"elapsed":1216,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":15,"outputs":[]},{"cell_type":"code","execution_count":16,"metadata":{"id":"OA6SV_pfE-b-","executionInfo":{"status":"ok","timestamp":1655999858290,"user_tz":240,"elapsed":6,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"outputs":[],"source":["# input\n","original_dim = 28 * 28\n","intermediate_dim = 64\n","latent_dim = 2\n","\n","inputs = keras.Input(shape=(original_dim,))\n","h = layers.Dense(intermediate_dim, activation='relu')(inputs)\n","z_mean = layers.Dense(latent_dim)(h)\n","z_log_sigma = layers.Dense(latent_dim)(h)"]},{"cell_type":"code","source":["from keras import backend as K\n","\n","def sampling(args):\n","    z_mean, z_log_sigma = args\n","    epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim),\n","                              mean=0., stddev=0.00001)\n","    return z_mean + K.exp(z_log_sigma) * epsilon\n","\n","z = layers.Lambda(sampling)([z_mean, z_log_sigma])"],"metadata":{"id":"PTJ43ziqFn-k","executionInfo":{"status":"ok","timestamp":1656000826749,"user_tz":240,"elapsed":3,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":25,"outputs":[]},{"cell_type":"markdown","source":["Finally, we can map these sampled latent points back to reconstructed inputs:"],"metadata":{"id":"TJcaR3eRFsQ7"}},{"cell_type":"code","source":["# Create encoder\n","encoder = keras.Model(inputs, [z_mean, z_log_sigma, z], name='encoder')\n","\n","# Create decoder\n","latent_inputs = keras.Input(shape=(latent_dim,), name='z_sampling')\n","x = layers.Dense(intermediate_dim, activation='relu')(latent_inputs)\n","outputs = layers.Dense(original_dim, activation='sigmoid')(x)\n","decoder = keras.Model(latent_inputs, outputs, name='decoder')\n","\n","# instantiate VAE model\n","outputs = decoder(encoder(inputs)[2])\n","vae = keras.Model(inputs, outputs, name='vae_mlp')"],"metadata":{"id":"AEONH7VhFsvb","executionInfo":{"status":"ok","timestamp":1656000827623,"user_tz":240,"elapsed":368,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":26,"outputs":[]},{"cell_type":"code","source":["# summary\n","vae.summary()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"NvqtVAHNV3hN","executionInfo":{"status":"ok","timestamp":1656001632925,"user_tz":240,"elapsed":361,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"2d909a2e-90ef-4f83-c597-a57e0f5946ad"},"execution_count":33,"outputs":[{"output_type":"stream","name":"stdout","text":["Model: \"vae_mlp\"\n","__________________________________________________________________________________________________\n"," Layer (type)                   Output Shape         Param #     Connected to                     \n","==================================================================================================\n"," input_3 (InputLayer)           [(None, 784)]        0           []                               \n","                                                                                                  \n"," encoder (Functional)           [(None, 2),          50500       ['input_3[0][0]']                \n","                                 (None, 2),                                                       \n","                                 (None, 2)]                                                       \n","                                                                                                  \n"," decoder (Functional)           (None, 784)          51152       ['encoder[0][2]']                \n","                                                                                                  \n"," dense_5 (Dense)                (None, 64)           50240       ['input_3[0][0]']                \n","                                                                                                  \n"," dense_7 (Dense)                (None, 2)            130         ['dense_5[0][0]']                \n","                                                                                                  \n"," dense_6 (Dense)                (None, 2)            130         ['dense_5[0][0]']                \n","                                                                                                  \n"," tf.__operators__.add_4 (TFOpLa  (None, 2)           0           ['dense_7[0][0]']                \n"," mbda)                                                                                            \n","                                                                                                  \n"," tf.math.square_2 (TFOpLambda)  (None, 2)            0           ['dense_6[0][0]']                \n","                                                                                                  \n"," tf.cast_2 (TFOpLambda)         (None, 784)          0           ['input_3[0][0]']                \n","                                                                                                  \n"," tf.convert_to_tensor_6 (TFOpLa  (None, 784)         0           ['decoder[0][0]']                \n"," mbda)                                                                                            \n","                                                                                                  \n"," tf.math.subtract_4 (TFOpLambda  (None, 2)           0           ['tf.__operators__.add_4[0][0]', \n"," )                                                                'tf.math.square_2[0][0]']       \n","                                                                                                  \n"," tf.math.exp_2 (TFOpLambda)     (None, 2)            0           ['dense_7[0][0]']                \n","                                                                                                  \n"," tf.keras.backend.binary_crosse  (None, 784)         0           ['tf.cast_2[0][0]',              \n"," ntropy_2 (TFOpLambda)                                            'tf.convert_to_tensor_6[0][0]'] \n","                                                                                                  \n"," tf.math.subtract_5 (TFOpLambda  (None, 2)           0           ['tf.math.subtract_4[0][0]',     \n"," )                                                                'tf.math.exp_2[0][0]']          \n","                                                                                                  \n"," tf.math.reduce_mean_4 (TFOpLam  (None,)             0           ['tf.keras.backend.binary_crossen\n"," bda)                                                            tropy_2[0][0]']                  \n","                                                                                                  \n"," tf.math.reduce_sum_2 (TFOpLamb  (None,)             0           ['tf.math.subtract_5[0][0]']     \n"," da)                                                                                              \n","                                                                                                  \n"," tf.math.multiply_4 (TFOpLambda  (None,)             0           ['tf.math.reduce_mean_4[0][0]']  \n"," )                                                                                                \n","                                                                                                  \n"," tf.math.multiply_5 (TFOpLambda  (None,)             0           ['tf.math.reduce_sum_2[0][0]']   \n"," )                                                                                                \n","                                                                                                  \n"," tf.__operators__.add_5 (TFOpLa  (None,)             0           ['tf.math.multiply_4[0][0]',     \n"," mbda)                                                            'tf.math.multiply_5[0][0]']     \n","                                                                                                  \n"," tf.math.reduce_mean_5 (TFOpLam  ()                  0           ['tf.__operators__.add_5[0][0]'] \n"," bda)                                                                                             \n","                                                                                                  \n"," add_loss_2 (AddLoss)           ()                   0           ['tf.math.reduce_mean_5[0][0]']  \n","                                                                                                  \n","==================================================================================================\n","Total params: 101,652\n","Trainable params: 101,652\n","Non-trainable params: 0\n","__________________________________________________________________________________________________\n"]}]},{"cell_type":"markdown","source":["What we've done so far allows us to instantiate 3 models:\n","\n","- an end-to-end autoencoder mapping inputs to reconstructions\n","- an encoder mapping inputs to the latent space\n","- a generator that can take points on the latent space and will output the corresponding reconstructed samples.\n"],"metadata":{"id":"F83uMIWgF2ET"}},{"cell_type":"markdown","source":["We train the model using the end-to-end model, with a custom loss function: the sum of a reconstruction term, and the KL divergence regularization term."],"metadata":{"id":"AVJLeODfGJct"}},{"cell_type":"code","source":["reconstruction_loss = keras.losses.binary_crossentropy(inputs, outputs)\n","reconstruction_loss *= original_dim\n","kl_loss = 1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma)\n","kl_loss = K.sum(kl_loss, axis=-1)\n","kl_loss *= -0.5\n","vae_loss = K.mean(reconstruction_loss + kl_loss)\n","vae.add_loss(vae_loss)\n","vae.compile(optimizer='adam')"],"metadata":{"id":"iUt31yd8GJXl","executionInfo":{"status":"ok","timestamp":1656000827624,"user_tz":240,"elapsed":3,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":27,"outputs":[]},{"cell_type":"markdown","source":["We train our VAE on MNIST digits:"],"metadata":{"id":"Nhhkm1D1GMQH"}},{"cell_type":"code","source":["from keras.datasets import mnist\n","import numpy as np"],"metadata":{"id":"xTUZRfgSGTDb","executionInfo":{"status":"ok","timestamp":1656000827624,"user_tz":240,"elapsed":3,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":28,"outputs":[]},{"cell_type":"code","source":["(x_train, y_train), (x_test, y_test) = mnist.load_data()\n","\n","x_train = x_train.astype('float32') / 255.\n","x_test = x_test.astype('float32') / 255.\n","x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))\n","x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))\n","\n","vae.fit(x_train, x_train,\n","        epochs=100,\n","        batch_size=32,\n","        validation_data=(x_test, x_test))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"31VVeOkAF3n0","executionInfo":{"status":"ok","timestamp":1656001630626,"user_tz":240,"elapsed":803004,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"c41bac9b-b4ea-4614-899f-844f5f3cccd4"},"execution_count":29,"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch 1/100\n","1875/1875 [==============================] - 10s 5ms/step - loss: 188.5094 - val_loss: 168.4880\n","Epoch 2/100\n","1875/1875 [==============================] - 10s 5ms/step - loss: 165.7443 - val_loss: 162.4903\n","Epoch 3/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 160.8530 - val_loss: 159.2367\n","Epoch 4/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 157.5824 - val_loss: 156.4051\n","Epoch 5/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 155.4839 - val_loss: 154.7016\n","Epoch 6/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 154.1239 - val_loss: 153.7073\n","Epoch 7/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 153.0794 - val_loss: 153.4391\n","Epoch 8/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 152.2299 - val_loss: 152.1881\n","Epoch 9/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 151.5184 - val_loss: 151.5402\n","Epoch 10/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 150.9030 - val_loss: 150.8686\n","Epoch 11/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 150.3664 - val_loss: 150.4436\n","Epoch 12/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 149.8738 - val_loss: 149.9316\n","Epoch 13/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 149.4327 - val_loss: 149.5811\n","Epoch 14/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 149.0481 - val_loss: 149.1727\n","Epoch 15/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 148.6743 - val_loss: 149.1334\n","Epoch 16/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 148.3363 - val_loss: 148.7839\n","Epoch 17/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 148.0163 - val_loss: 148.8763\n","Epoch 18/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 147.7662 - val_loss: 148.0526\n","Epoch 19/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 147.4717 - val_loss: 148.1878\n","Epoch 20/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 147.2687 - val_loss: 147.8080\n","Epoch 21/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 147.0676 - val_loss: 147.6892\n","Epoch 22/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 146.8312 - val_loss: 147.4462\n","Epoch 23/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 146.6662 - val_loss: 147.3632\n","Epoch 24/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 146.4653 - val_loss: 147.3523\n","Epoch 25/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 146.3300 - val_loss: 147.4919\n","Epoch 26/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 146.1455 - val_loss: 147.1599\n","Epoch 27/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.9840 - val_loss: 147.0888\n","Epoch 28/100\n","1875/1875 [==============================] - 9s 5ms/step - loss: 145.8554 - val_loss: 146.7264\n","Epoch 29/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.7151 - val_loss: 146.7624\n","Epoch 30/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.6190 - val_loss: 146.7112\n","Epoch 31/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.4919 - val_loss: 146.4333\n","Epoch 32/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.3505 - val_loss: 146.5393\n","Epoch 33/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.2668 - val_loss: 147.0874\n","Epoch 34/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.1603 - val_loss: 146.2859\n","Epoch 35/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 145.0606 - val_loss: 146.1930\n","Epoch 36/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.9507 - val_loss: 145.9885\n","Epoch 37/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.8741 - val_loss: 146.0838\n","Epoch 38/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.7839 - val_loss: 146.0788\n","Epoch 39/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.7077 - val_loss: 145.8134\n","Epoch 40/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.6270 - val_loss: 146.0575\n","Epoch 41/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.5752 - val_loss: 145.8880\n","Epoch 42/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.5074 - val_loss: 145.8622\n","Epoch 43/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.4054 - val_loss: 145.7831\n","Epoch 44/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.3751 - val_loss: 145.8580\n","Epoch 45/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.2587 - val_loss: 145.5450\n","Epoch 46/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.2138 - val_loss: 145.6825\n","Epoch 47/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.1333 - val_loss: 145.7980\n","Epoch 48/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.0699 - val_loss: 145.6631\n","Epoch 49/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 144.0382 - val_loss: 145.4380\n","Epoch 50/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.9829 - val_loss: 145.5930\n","Epoch 51/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.8797 - val_loss: 145.4716\n","Epoch 52/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.8812 - val_loss: 145.2853\n","Epoch 53/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.7988 - val_loss: 145.5275\n","Epoch 54/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.7534 - val_loss: 145.3301\n","Epoch 55/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.6887 - val_loss: 145.5430\n","Epoch 56/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.6289 - val_loss: 145.2061\n","Epoch 57/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.5893 - val_loss: 145.1447\n","Epoch 58/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.5442 - val_loss: 145.6349\n","Epoch 59/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.4988 - val_loss: 145.0847\n","Epoch 60/100\n","1875/1875 [==============================] - 9s 5ms/step - loss: 143.4332 - val_loss: 145.0155\n","Epoch 61/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.4090 - val_loss: 145.1808\n","Epoch 62/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.3701 - val_loss: 145.2663\n","Epoch 63/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.3111 - val_loss: 144.9385\n","Epoch 64/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.2792 - val_loss: 145.0052\n","Epoch 65/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.2265 - val_loss: 144.9833\n","Epoch 66/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.1718 - val_loss: 145.1300\n","Epoch 67/100\n","1875/1875 [==============================] - 9s 5ms/step - loss: 143.1454 - val_loss: 144.8909\n","Epoch 68/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.0817 - val_loss: 144.8549\n","Epoch 69/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.0709 - val_loss: 144.6850\n","Epoch 70/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 143.0379 - val_loss: 145.0110\n","Epoch 71/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.9976 - val_loss: 144.7770\n","Epoch 72/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.9654 - val_loss: 144.9857\n","Epoch 73/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.9173 - val_loss: 144.5193\n","Epoch 74/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.8678 - val_loss: 144.9479\n","Epoch 75/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.8758 - val_loss: 144.7753\n","Epoch 76/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.8078 - val_loss: 144.9018\n","Epoch 77/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.7642 - val_loss: 144.6623\n","Epoch 78/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.7427 - val_loss: 144.6104\n","Epoch 79/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.7003 - val_loss: 144.6850\n","Epoch 80/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.7116 - val_loss: 144.7783\n","Epoch 81/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.6468 - val_loss: 144.6641\n","Epoch 82/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.6089 - val_loss: 144.5801\n","Epoch 83/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.5903 - val_loss: 144.5565\n","Epoch 84/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.5556 - val_loss: 144.6215\n","Epoch 85/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.5426 - val_loss: 144.6276\n","Epoch 86/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.4849 - val_loss: 144.6281\n","Epoch 87/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.4780 - val_loss: 144.5118\n","Epoch 88/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.4492 - val_loss: 144.5258\n","Epoch 89/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.4130 - val_loss: 144.5294\n","Epoch 90/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.3701 - val_loss: 144.3222\n","Epoch 91/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.3708 - val_loss: 144.5715\n","Epoch 92/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.3378 - val_loss: 144.4745\n","Epoch 93/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.3087 - val_loss: 144.6283\n","Epoch 94/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.2634 - val_loss: 144.5271\n","Epoch 95/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.2438 - val_loss: 144.4693\n","Epoch 96/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.2114 - val_loss: 144.2991\n","Epoch 97/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.1932 - val_loss: 144.3207\n","Epoch 98/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.1853 - val_loss: 144.5853\n","Epoch 99/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.1597 - val_loss: 144.3332\n","Epoch 100/100\n","1875/1875 [==============================] - 8s 4ms/step - loss: 142.1251 - val_loss: 144.3431\n"]},{"output_type":"execute_result","data":{"text/plain":["<keras.callbacks.History at 0x7f04d5771cd0>"]},"metadata":{},"execution_count":29}]},{"cell_type":"code","source":["# prediction\n","pred_ = vae.predict(x_test)"],"metadata":{"id":"heuHkUlGGOHO","executionInfo":{"status":"ok","timestamp":1656001630959,"user_tz":240,"elapsed":337,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":30,"outputs":[]},{"cell_type":"code","source":["# import \n","import matplotlib.pyplot as plt"],"metadata":{"id":"RG8ooW_QR0K2","executionInfo":{"status":"ok","timestamp":1656001631307,"user_tz":240,"elapsed":349,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}}},"execution_count":31,"outputs":[]},{"cell_type":"code","source":["# Use Matplotlib (don't ask)\n","n = 10  # How many digits we will display\n","plt.figure(figsize=(20, 4))\n","for i in range(n):\n","    # Display original\n","    ax = plt.subplot(2, n, i + 1)\n","    plt.imshow(x_test[i].reshape(28, 28))\n","    plt.gray()\n","    ax.get_xaxis().set_visible(False)\n","    ax.get_yaxis().set_visible(False)\n","\n","    # Display reconstruction\n","    ax = plt.subplot(2, n, i + 1 + n)\n","    plt.imshow(pred_[i].reshape(28, 28))\n","    plt.gray()\n","    ax.get_xaxis().set_visible(False)\n","    ax.get_yaxis().set_visible(False)\n","plt.show()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":248},"id":"Tfk02IrORu6o","executionInfo":{"status":"ok","timestamp":1656001631974,"user_tz":240,"elapsed":668,"user":{"displayName":"Yiqiao Yin","userId":"17143243938280119775"}},"outputId":"debf656f-5293-4d44-a166-999dbd6959af"},"execution_count":32,"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 1440x288 with 20 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAABG0AAADnCAYAAACkCqtqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd7gV1bnH8ffYQRSliUiTJk1EmqCiosbeo4lXrzfXFnNjbkyx5CbexKhJnkcTE2MMxjzXRI2isRAbYuwFRAQp0gWkCVIEUawo5/7h48pvvZwZ9jnsvc/s2d/PX+84c/YeZmatmT2ud701tbW1BgAAAAAAgGzZprF3AAAAAAAAAJvjpQ0AAAAAAEAG8dIGAAAAAAAgg3hpAwAAAAAAkEG8tAEAAAAAAMggXtoAAAAAAABk0Hb12bimpob64I2ktra2phifwzlsVGtqa2tbF+ODOI+Nh7aYC7TFHKAt5gJtMQdoi7lAW8wB2mIu1NkWGWkDlM/ixt4BAGZGWwSygrYIZANtEciGOtsiL20AAAAAAAAyiJc2AAAAAAAAGcRLGwAAAAAAgAzipQ0AAAAAAEAG8dIGAAAAAAAgg3hpAwAAAAAAkEG8tAEAAAAAAMggXtoAAAAAAABk0HaNvQOoTpdeemmImzRpEq3r169fiE8//fTEzxg5cmSIX3755WjdnXfeubW7CAAAAABAo2KkDQAAAAAAQAbx0gYAAAAAACCDeGkDAAAAAACQQcxpg7K59957Q5w2V43atGlT4rqLLrooxEceeWS07vnnnw/xkiVLCt1FNLIePXpEy3PmzAnxJZdcEuKbbrqpbPtUzXbeeecQX3/99SHWtmdmNnny5BCfccYZ0brFixeXaO8AAAAax+677x7ijh07FvQ3/pno+9//fohnzJgR4nnz5kXbTZs2rSG7iBxhpA0AAAAAAEAG8dIGAAAAAAAgg0iPQsloOpRZ4SlRmhLzxBNPhLhLly7RdieeeGKIu3btGq07++yzQ/yrX/2qoO9F49t///2jZU2PW7ZsWbl3p+rtueeeIb7wwgtD7NMWBw4cGOITTjghWnfzzTeXaO+gBgwYEOIHH3wwWte5c+eSfe9RRx0VLc+ePTvES5cuLdn3Ysv0Hmlm9vDDD4f4O9/5TohvueWWaLvPP/+8tDuWQ23atAnx3//+9xCPHz8+2u7WW28N8aJFi0q+X19q3rx5tHzIIYeEeOzYsSHeuHFj2fYJqATHH398iE866aRo3WGHHRbibt26FfR5Pu2pU6dOId5xxx0T/27bbbct6PORX4y0AQAAAAAAyCBe2gAAAAAAAGQQ6VEoqkGDBoX41FNPTdxu5syZIfbDDdesWRPiDRs2hHiHHXaItpswYUKI99tvv2hdy5YtC9xjZEn//v2j5Q8++CDEo0ePLvfuVJ3WrVtHy7fffnsj7Qnq6+ijjw5x2hDrYvMpOOedd16IzzzzzLLtB76g974//vGPidv94Q9/CPFtt90Wrfvoo4+Kv2M5o1VjzOJnGk1FWrlyZbRdY6VEaYU/s7iv1/TW+fPnl37HKsyuu+4aLWvKfd++fUPsq5iSapZtOq3CxRdfHGJNBTcza9KkSYhramq2+nt9lVSgUIy0AQAAAAAAyCBe2gAAAAAAAGQQL20AAAAAAAAyqFHntPEloDWPcPny5dG6jz/+OMR33XVXiN9+++1oO/JxG5eWCPa5n5rzrfMvrFixoqDP/uEPfxgt9+7dO3Hbxx57rKDPROPTnHAtQ2tmduedd5Z7d6rOd7/73RCfcsop0bohQ4bU+/O0lKyZ2Tbb/Ov/DUybNi3EL7zwQr0/G7HttvvXLfy4445rlH3wc2X84Ac/CPHOO+8crdM5qlAa2v7at2+fuN2oUaNCrM9XSNaqVasQ33vvvdG6Fi1ahFjnEvrv//7v0u9YgiuvvDLEe++9d7TuoosuCjHPzZs7++yzQ/yLX/wiWtehQ4c6/8bPffPOO+8Uf8dQNNo/XnLJJSX9rjlz5oRYfwuheLTkuvbVZvEcq1qm3cxs06ZNIb7llltCPG7cuGi7LPSTjLQBAAAAAADIIF7aAAAAAAAAZFCjpkddd9110XLnzp0L+jsd1vn+++9H68o57GzZsmUh9v+WSZMmlW0/suSRRx4JsQ5VM4vP1dq1a+v92b587Pbbb1/vz0D29OzZM8Q+ncIPQUfx/fa3vw2xDhNtqNNOOy1xefHixSH++te/Hm3n02ywZSNGjAjxsGHDQuzvR6XkSx9r2mrTpk2jdaRHFZ8v7/6Tn/ykoL/T1NPa2tqi7lNeDRgwIMR+iL26+uqry7A3m+vTp0+0rCnlo0ePjtZxb92cpsv87ne/C3HLli2j7ZLay0033RQta7p3Q555URifCqOpTpriMnbs2Gi7Tz75JMTr168Psb9P6XPpP//5z2jdjBkzQvzKK6+EeMqUKdF2H330UeLno3A6nYJZ3Mb0WdNfE4U64IADQvzZZ59F6+bOnRvil156KVqn19ynn37aoO8uBCNtAAAAAAAAMoiXNgAAAAAAABnESxsAAAAAAIAMatQ5bbTEt5lZv379Qjx79uxoXa9evUKcllc8dOjQEC9dujTESSX66qJ5bKtXrw6xlrP2lixZEi1X65w2SuevaKjLLrssxD169EjcTnNJ61pGdl1++eUh9tcM7ag0xowZE2Ityd1QWtp0w4YN0bpOnTqFWMvOTpw4Mdpu22233er9yDufz61lmxcsWBDiX/7yl2Xbp5NPPrls34XN7bvvvtHywIEDE7fVZ5vHH3+8ZPuUF23atImWv/rVryZue/7554dYnxtLTeexeeqppxK383Pa+PkgYXbppZeGWEu4F8rP03bMMceE2JcN1/lvSjkHRl6lzTOz3377hVhLPXsTJkwIsf6uXLRoUbRdx44dQ6xzmZoVZx5AbE7fB1x88cUh9m1s1113rfPv33rrrWj5xRdfDPGbb74ZrdPfIDq34pAhQ6LttE847rjjonXTpk0LsZYNLzZG2gAAAAAAAGQQL20AAAAAAAAyqFHTo55++unUZeVLtX3Jlxvt379/iHWY0+DBgwver48//jjE8+bNC7FP2dKhUjo0HVvnhBNOCLGWztxhhx2i7VatWhXi//mf/4nWffjhhyXaO2ytzp07R8uDBg0KsbY3M0ojFsuhhx4aLe+zzz4h1uG9hQ719cM/dXiyls40Mzv88MNDnFaO+L/+679CPHLkyIL2o9pceeWV0bIOEdeh+D5Frdj03uevLYaLl1dayo7n0wiQ7je/+U20/O///u8h1udLM7P77ruvLPvkDR8+PMR77LFHtO6vf/1riP/2t7+Va5cqhqbumpmde+65dW43ffr0aHnlypUhPvLIIxM/v3nz5iHW1Cszs7vuuivEb7/99pZ3tsr55/+77747xJoOZRanB6elDCqfEqX89Bcovj/96U/Rsqa1pZXv1vcGr7/+eoh//OMfR9vp73rvwAMPDLE+h952223Rdvp+QfsAM7Obb745xA888ECIi50qy0gbAAAAAACADOKlDQAAAAAAQAY1anpUMaxbty5afvbZZ+vcLi31Ko0OPfapWDoU6957723Q52Nzmi7jh0QqPebPP/98SfcJxePTKVQ5q27knaah3XPPPdG6tOGmSqt56ZDPn//859F2aemI+hnf/OY3Q9y6detou+uuuy7EO+20U7TuD3/4Q4g3bty4pd3OldNPPz3EvmLB/PnzQ1zOSmua5ubToZ577rkQv/vuu+Xapap1yCGHJK7zVWnS0hOxudra2mhZr/Xly5dH60pZAahJkybRsg79//a3vx1iv7/nnXdeyfYpDzTdwcxsl112CbFWm/HPLHp/+rd/+7cQ+5SMrl27hrht27bRuoceeijExx57bIjXrl1b0L5Xg2bNmoXYT4Gg0yisWbMmWvfrX/86xEyVkB3+uU6rNl1wwQXRupqamhDr7wKfOn/99deHuKHTKbRs2TLEWsX0qquuirbTaVp8amW5MNIGAAAAAAAgg3hpAwAAAAAAkEG8tAEAAAAAAMigip/TphTatGkT4j/+8Y8h3mab+B2XlqMmD7Xh/vGPf0TLRx11VJ3b3XHHHdGyL3+LyrDvvvsmrtN5TbB1ttvuX917oXPY+LmhzjzzzBD7vPFC6Zw2v/rVr0J8ww03RNs1bdo0xP46ePjhh0O8YMGCBu1HpTrjjDNCrMfILL4/lZrOkXT22WeH+PPPP4+2u/baa0NcbfMPlYuWKNXY8zn+U6dOLdk+VZvjjz8+WtZy6jqXk5+DoVA6j8phhx0WrRs6dGidf3P//fc36Luq1Y477hgt65xAv/3tbxP/TssH/+Uvfwmx9tVmZl26dEn8DJ1rpZTzIVWyU045JcQ/+tGPonVahlvL3puZrV+/vrQ7hgbx/dhll10WYp3DxszsrbfeCrHOLTtx4sQGfbfOVdOhQ4donf62HDNmTIj9PLbK7++dd94Z4lLO5cdIGwAAAAAAgAzipQ0AAAAAAEAGkR5Vh4svvjjEWpbWlxefO3du2fYpb/bcc88Q++HdOmRVUzJ02L2Z2YYNG0q0dyg2Hc597rnnRuumTJkS4ieffLJs+4QvaKloXyK2oSlRSTTNSVNszMwGDx5c1O+qVM2bN4+Wk1IhzBqeetEQWq5d0+1mz54dbffss8+WbZ+qVaFtpZzXRx7deOON0fKIESNC3K5du2idll7XofMnnXRSg75bP8OX8lYLFy4MsS85jXRartvT9Defwp9k0KBBBX/3hAkTQsyzbN3SUj/1uXHZsmXl2B1sJU1RMts8tVp99tlnIT7ggANCfPrpp0fb9ezZs86//+ijj6LlXr161Rmbxc+5e+yxR+I+qZUrV0bL5UoLZ6QNAAAAAABABvHSBgAAAAAAIINIjzKzgw46KFr2s5R/SWcyNzObMWNGyfYp7x544IEQt2zZMnG7v/3tbyGutqoxeXLkkUeGuEWLFtG6sWPHhlirMqB4fOU7pUNPS02H/Pt9StvHq666KsTnnHNO0fcrS3xFk7322ivEo0aNKvfuBF27dq3zv3MfLL+0NIxiVC7CFyZPnhwt9+vXL8T9+/eP1h1zzDEh1qooq1evjra7/fbbC/purUYybdq0xO3Gjx8fYp6R6sf3p5rKpimIPgVDK2CeeuqpIfbVZrQt+nUXXnhhiPVcz5o1q6B9rwY+FUZpe/vZz34WrXvooYdCTMW87HjmmWeiZU2l1t8IZmYdO3YM8e9///sQp6WKarqVT8VKk5QStWnTpmh59OjRIf7ud78brVuxYkXB37c1GGkDAAAAAACQQby0AQAAAAAAyCBe2gAAAAAAAGQQc9qY2XHHHRctb7/99iF++umnQ/zyyy+XbZ/ySPOFBwwYkLjdc889F2Kfq4rKtN9++4XY56Tef//95d6dqvCtb30rxD43t7GceOKJId5///2jdbqPfn91Tpu8e//996NlzcnXOTXM4vmh1q5dW9T9aNOmTbScNL/ASy+9VNTvRd0OPvjgEJ911lmJ261fvz7ElMItrnXr1oXYl7bX5SuuuGKrv6tLly4h1rnAzOI+4dJLL93q76pWTz31VLSsbUfnrfHzzCTNq+E/7+KLLw7xo48+Gq3r3r17iHV+DL1vV7vWrVuH2D8T6NxvP/3pT6N1V155ZYhvueWWEGuZdbN43pT58+eHeObMmYn71KdPn2hZfxfS36bzZbh1PqjddtstWqdzy+q8s++880603ZIlS0Ks14T+5jAzGzJkSL3399Zbb42Wf/zjH4dY56sqJ0baAAAAAAAAZBAvbQAAAAAAADKoatOjmjRpEmItHWdm9umnn4ZY03M2btxY+h3LEV/KW4eWaQqap0N/N2zYUPwdQ1m0bds2xMOHDw/x3Llzo+20jB6KR1ORykmHNJuZ9e7dO8TaB6TxZXKrqe/1Q4i1jO9Xv/rVaN1jjz0W4htuuKHe39W3b99oWVMyOnfuHK1LSgnISupd3un9dJttkv9/25NPPlmO3UGJacqHb3uafuX7ShTOp5R+7WtfC7GmbTdv3jzxM2666aYQ+7S4jz/+OMQPPvhgtE7TP44++ugQd+3aNdqumsu4//rXvw7xD37wg4L/TvvHb3/723XGxaLtT6d2OPPMM4v+XXnm0420fTTEHXfcES2npUdpSrpeZ3/961+j7bSkeGNhpA0AAAAAAEAG8dIGAAAAAAAgg3hpAwAAAAAAkEFVO6fNZZddFmJfenbs2LEhHj9+fNn2KW9++MMfRsuDBw+uc7t//OMf0TJlvvPhP//zP0Os5YMff/zxRtgblMtPfvKTaFnLnqZZtGhRiL/xjW9E67SsY7XR/tCX/j3++ONDPGrUqHp/9po1a6JlnTujVatWBX2Gz/tGaSSVXPdzAfzpT38qx+6gyM4444xo+T/+4z9CrHMumG1e9hbFoSW7tb2dddZZ0Xba5nTuIZ3Dxrvmmmui5V69eoX4pJNOqvPzzDa/F1YTndfk3nvvjdbdfffdId5uu/inbIcOHUKcNv9XMegcfnrNaNlxM7Nrr722pPsBs8svvzzE9ZlT6Fvf+laIG/IcVU6MtAEAAAAAAMggXtoAAAAAAABkUNWkR+kwcjOz//3f/w3xe++9F627+uqry7JPeVdoib7vfOc70TJlvvOhU6dOdf73devWlXlPUGpjxowJ8T777NOgz5g1a1aIX3rppa3ep7yYM2dOiLUkrZlZ//79Q9ytW7d6f7aWtfVuv/32aPnss8+ucztfohzF0b59+2jZp2h8admyZdHypEmTSrZPKJ1jjz02cd2jjz4aLb/22mul3p2qp6lSGjeU7yc13UfTo0aMGBFt16JFixD7EuV5pyWWfb/Wo0ePxL874ogjQrz99tuH+Kqrroq2S5qyoaE0fXngwIFF/WzU7YILLgixpqT5lDk1c+bMaPnBBx8s/o6VCCNtAAAAAAAAMoiXNgAAAAAAABmU6/Soli1bhvj3v/99tG7bbbcNsQ7tNzObMGFCaXcMER3+aWa2cePGen/G+vXrEz9Dh0c2b9488TN22223aLnQ9C4dwnnFFVdE6z788MOCPiOPTjjhhDr/+yOPPFLmPalOOlQ3rYJC2rD8W2+9NcTt2rVL3E4/f9OmTYXuYuTEE09s0N9Vs6lTp9YZF8PChQsL2q5v377R8owZM4q6H9XqwAMPjJaT2rCvvojK5PvhDz74IMS/+c1vyr07KLG///3vIdb0qK9//evRdjp9AFM3FObpp5+u879rOrFZnB712Wefhfgvf/lLtN2f//znEH/ve9+L1iWlraI0hgwZEi1r39isWbPEv9NpN7RalJnZJ598UqS9Kz1G2gAAAAAAAGQQL20AAAAAAAAyiJc2AAAAAAAAGZS7OW10rpqxY8eGeO+99462W7BgQYi1/DfKb/r06Vv9Gffdd1+0vGLFihDvscceIfb5wsX29ttvR8u/+MUvSvp9WXLwwQdHy23btm2kPYGZ2ciRI0N83XXXJW6n5WTT5qMpdK6aQre75ZZbCtoOjUPnRKpr+UvMYVMaOieft2bNmhDfeOON5dgdlIDOraDPKWZmq1atCjElvvNH75N6fz755JOj7X72s5+F+J577onWzZs3r0R7l0///Oc/o2V9PtcS0RdeeGG0Xbdu3UJ82GGHFfRdy5Yta8AeYkv83Ie77LJLndvpnGBm8bxR48aNK/6OlQkjbQAAAAAAADKIlzYAAAAAAAAZlLv0qK5du4Z44MCBidtpOWdNlULx+FLqfthnMZ1xxhkN+jst85eW1vHwww+HeNKkSYnbvfjiiw3ajzw49dRTo2VNVZwyZUqIX3jhhbLtUzV78MEHQ3zZZZdF61q3bl2y7129enW0PHv27BB/85vfDLGmMCJ7amtrU5dRWkcffXTiuiVLloR4/fr15dgdlICmR/n29dhjjyX+naYE7L777iHW6wKVY+rUqSH+6U9/Gq27/vrrQ/zLX/4yWnfOOeeE+KOPPirR3uWHPouYxWXXv/a1ryX+3YgRIxLXff755yHWNvujH/2oIbuIOmh/d/nllxf0N3fddVe0/NxzzxVzlxoNI20AAAAAAAAyiJc2AAAAAAAAGcRLGwAAAAAAgAyq+DltOnXqFC37km5f8nM6aJlblMZpp50WLWsu4vbbb1/QZ/Tp0yfE9SnXfdttt4V40aJFids98MADIZ4zZ07Bn48vNG3aNMTHHXdc4nb3339/iDUHGKWzePHiEJ955pnRulNOOSXEl1xySVG/15e5v/nmm4v6+SiPnXbaKXEd8yeUht4XdX4+7+OPPw7xxo0bS7pPaBx6nzz77LOjdd///vdDPHPmzBB/4xvfKP2OoaTuuOOOaPmiiy4KsX+mvvrqq0M8ffr00u5YDvj71ve+970QN2vWLMSDBg2KtmvTpk2I/e+JO++8M8RXXXVVEfYSZvH5mDVrVojTfjtqG9BzmyeMtAEAAAAAAMggXtoAAAAAAABkUMWnR2kJWTOzjh071rnd888/Hy1TvrT8rrvuuq36+7POOqtIe4Ji0aH569ati9ZpmfQbb7yxbPuEzfky67qsKaW+Pz3xxBNDrOfz1ltvjbarqakJsQ5lReU699xzo+V33303xNdcc025d6cqbNq0KcSTJk2K1vXt2zfE8+fPL9s+oXFccMEFIT7//POjdf/3f/8XYtpivqxevTpaPvLII0PsU3OuuOKKEPsUOmzZypUrQ6zPOlpK3cxs6NChIf75z38erVu1alWJ9q66HX744SFu3759iNN+u2vaqKYQ5wkjbQAAAAAAADKIlzYAAAAAAAAZVFOfNKGamppM5BQdfPDBIR4zZky0TmecVkOGDImW/dDjrKutra3Z8lZblpVzWKUm19bWDtryZlvGeWw8tMVcoC1uwSOPPBIt33DDDSF+9tlny707dcpzW2zXrl20fO2114Z48uTJIc5BdbaqbYv6LKuVgMziFNaRI0dG6zQV+dNPPy3R3tVPnttiVvjquMOGDQvxAQccEOKtSFGu2raYJ3loi9OmTQvxvvvum7jd9ddfH2JNF8yBOtsiI20AAAAAAAAyiJc2AAAAAAAAGcRLGwAAAAAAgAyqyJLfw4cPD3HSHDZmZgsWLAjxhg0bSrpPAADkhZZARfktX748Wj7vvPMaaU9QKi+99FKItcQtUJfTTz89WtZ5P7p16xbirZjTBsiEFi1ahLim5l9T9PgS67/73e/Ktk9ZwEgbAAAAAACADOKlDQAAAAAAQAZVZHpUGh0ueMQRR4R47dq1jbE7AAAAANBg7733XrS89957N9KeAKV1ww031Blfc8010XYrVqwo2z5lASNtAAAAAAAAMoiXNgAAAAAAABnESxsAAAAAAIAMqqmtrS1845qawjdGUdXW1tZseast4xw2qsm1tbWDivFBnMfGQ1vMBdpiDtAWc4G2mAO0xVygLeYAbTEX6myLjLQBAAAAAADIIF7aAAAAAAAAZFB9S36vMbPFpdgRpOpUxM/iHDYezmPl4xzmA+ex8nEO84HzWPk4h/nAeax8nMN8qPM81mtOGwAAAAAAAJQH6VEAAAAAAAAZxEsbAAAAAACADOKlDQAAAAAAQAbx0gYAAAAAACCDeGkDAAAAAACQQby0AQAAAAAAyCBe2gAAAAAAAGQQL20AAAAAAAAyiJc2AAAAAAAAGcRLGwAAAAAAgAzipQ0AAAAAAEAG8dIGAAAAAAAgg3hpAwAAAAAAkEG8tAEAAAAAAMggXtoAAAAAAABkEC9tAAAAAAAAMoiXNgAAAAAAABnESxsAAAAAAIAM4qUNAAAAAABABvHSBgAAAAAAIIN4aQMAAAAAAJBBvLQBAAAAAADIoO3qs3FNTU1tqXYE6Wpra2uK8Tmcw0a1pra2tnUxPojz2Hhoi7lAW8wB2mIu0BZzgLaYC7TFHKAt5kKdbZGRNkD5LG7sHQBgZrRFICtoi0A20BaBbKizLfLSBgAAAAAAIIN4aQMAAAAAAJBBvLQBAAAAAADIIF7aAAAAAAAAZFC9qkcB9bHNNvE7wZqaf01ovv3224d42223jbb77LPPQvz555+HuLY2eSLzTZs2Rctp2wIAAAAAUAkYaQMAAAAAAJBBvLQBAAAAAADIINKjsFW22y6+hHbZZZcQ77HHHtG6vfbaK8Rdu3YNca9evaLtWrRoEWJNe1q7dm203ezZs0M8derUaN3ixf8qcf/ee++FWFOvzOI0Kp9SRYpV+e2www7RsqbRffDBB+XenaqnqYua7qipjn6dp204rU1pKqRPdwRQf2ntVNsbAADINkbaAAAAAAAAZBAvbQAAAAAAADKIlzYAAAAAAAAZxJw2W+BzwpNU0/wnOs9Is2bNonV77rlniLt16xat6969e4j79+8f4gMOOCDarm3btiHeuHFjiJcuXRpt1759+8T92HnnnUM8f/78EK9fvz7a7tNPPw2xz/EvdC4O1I+f/0SvmaOOOipat99++4V45MiRIdZzasb8DFtD+zidw8bMbKeddgpxy5YtQ9yhQ4douz59+oRY57UyM3v33XdDrOdt2bJl0XZr1qwJ8UcffRSt036A+W7Kw9/7kuY3SpsLzLdL+tHi8/PK7brrriHWvtXMrFWrViGeNWtWiNetWxdt5+d+Q7YkPZf6e6u2WZ5vvqDHLm3OJ439XHu67uOPPw6xPk/6z/f31qT5pXiWKS49B76v3HHHHbcYm8W/eTxtR5988kmI9ZnFLO5T09alzbPJs095pfUPac89pexPGWkDAAAAAACQQby0AQAAAAAAyKCKSY9KGg5a6PCltOFKfkhp0pBGv50OVfPDIvM2jC3pmGh5bjOzjh071hmbxelR++yzT4j9EO4mTZqEeMOGDSFu3rx5tN3ee++duL86nFFTLXQoq1l83qppiHBj8m22U6dOIb7wwgujdTpstEuXLiF+8803o+0YUtxw2q9paoVZfMw1jfGQQw6JttNUyN122y1a995774VYUzKeeuqpaLtXX301xMuXL4/WaT+QVhq8Uttw2pD9tBRdPXfaL/vh3En3Kn/89Lv8EPGmTZuGWNPm/LB/Hfq9evXqxHWVeq6yQM+TP9d6373ooouidXoOR48eHT1NrlQAACAASURBVOJnnnkm2u79998PMeepMEltOK09F3ps055ztf3ps5NZfL59yqkua5+Qh9Q4PT7aV5nF7cX3f5o+o/2ff/bUz1i7dm2IfZqh8vdWPVeaQqz3y7r2sZoU2o78uqQpHPxvkl69eoVYf0/4ZxhNe/K/IT744IMQ63n0zzB6L/TXiU7boO1Sv9eseq+FQp+BVEPvW2kpjT5NUiWlvxX7twkjbQAAAAAAADKIlzYAAAAAAAAZVPb0qEJnU0+bBV+HMPrhSjoUUocf+u102Q8v1n3UYaM+BUqHu+mwOL9t3oYXpw3H1SGgWsHJr9Pz+/bbb0fb6dDsd955p6D98ClW/fr1C7FWnVq1alW0nQ5tTNOQIc2om0+7OPzww0O8xx57ROt0iGna0GPUj7Y/bcOaDmVmdswxx4T4wAMPDHGPHj2i7XQYvj+/2g9o3+37XW2L2geYJffDlTxcWPuUtOH8ejz9fVGPrQ7h98O79VjrMF5NOzOLh2P7e6YOM09LHVi0aFHi51MlpTj0OvDH/+CDDw7xwIEDo3V67vWe6e/Vvv1Vk0JTMnxbTHou9cdW27du55+HG/LMsfvuu0fL+nf+WUqXtR/NW8W3tOdQ/2/TdZq27e+Leq4WL14c4gULFkTb6b3KVzjVPlT7yUIr1laDtDRQbW/+d4hWoNUKpD6te8CAASHWipj+Hqz94Ycffhit03QmrYD51ltvRdtNnjw5xG+88Ua0bsWKFXV+hv9dqd9ViffP+vSnSeneab/5k55zzOK2nvYZ2i59m9XnI/97RJf1eil2Cj8jbQAAAAAAADKIlzYAAAAAAAAZxEsbAAAAAACADCrLnDaam6b5pT7/VvMSNV/RLM4507/zc2Docps2ber8bP95npZ00xw2n184YcKEEE+bNi1apyXcNLeuUvOD0+YiUppn6XMUNR9T84D9vDK6To+/zpthZrbXXnuF2Occt2/fPsT7779/iOfMmRNtp/mjWqbNq9TzlkW77LJLtKylpP06ne9o/vz5IfbzSyGdzyXW/lX7zBEjRkTbDRkyJMTapnxbWblyZeJ36bLOv6Flws3MBg8eHOKFCxdG65JKfqflLWdBUjlKszj/Wo9Lq1atou00r9rfMzV3X+co8eVltc/WtpNWWj1tbiK9t3p6bfk5xLQ/11zvrJ23rNPryl8v/fv3D7F/PtL+NCkH36z6zkeh8ydom/DPkNo+tC22a9cu2k7vcfpcmjb3gS/Xre1U98PPZaXnVZ+r/Pcllas1q8y5M/TY+bLJep503hqzuD/VEtD+2TNpzkr/bKzn2l8v06dPD7Ee/0qep60Y9HlB7yX++Ok9zpfy1jkthw8fHmLtG83i861t27cBPcd+bh09x9rftmjRIvEz/L9F263Owfnmm29G22lb9M/AWW2n2ibS5pLxx0TbnD73+PlLtX/V4+j7TG1X/tkm6VnMP+fo7/q5c+dG6/R9gJ43P68fc9oAAAAAAADkEC9tAAAAAAAAMqgk6VF+SLgOidKhTDrc3iwetujTJPTvdNiiT4vRocI6XMyn4OiwOz+sTIdp6RAtHRrlP0PTbMySh/PnYdixDl/06Ql6jPzQ+yS+NN7atWtDrMfODzfU68wPc9VhbXrt6HBIs+RULLPkYap5OIflptdMy5Yto3Xahn3f8frrr4f4vffeK9He5Z9PWdK+dtCgQSHWVEKzuK/VIaWrV6+OtktrR/p3nTt3DrFP3dDym3rezcyWLVsWYr1G/PXS2EPL09LQ/LBqHcrbq1evEPv2ocdJy5KaxSlmOlzc98t6/9Ohu74v01QsPzS4T58+Ie7atWuI/TnQlA9NxzGLU4yTUgywZTrk3A/11mcgf83pdaBtyg8lzzt/zepx0nbkn0N12acq9ujRI8TanjWN2yw+X9oe9LnH76MvL6v9nD7T+Gck/Ts/RYCW/NZ7ax5KTmt/4p/vtY/TfszMrHXr1iHW+5j//aDnbebMmSH2x07Tdvy1pGkeM2bMqONfUZ0KTY/S+6S/L2p6lJ5j31fq7xVtK1qC22/ny0Anpef4vlf7C7+/WkZc26IvV6/pjlmeIiCpVLtP29Y24VM7td/U/lRLuJvFv/203fvflXqMffqb/s7XNutTj/U+6Z/TFi1aFGK9fnzfsbUYaQMAAAAAAJBBvLQBAAAAAADIIF7aAAAAAAAAZFBJ5rRJmz9Bc9q0tJZZnBuouaVm8dwK3bt3D7HPOdO8RM3d9/nCmtPmcw91jg3NW/PbaQ6zzyXWeXzSSklXIp0vwefr6dw+PpdYz42WVfdzBel8QHotaU6iWTzngn6e/zs9Fz6vW9ellTLXHHJ/fTMfw5bpsdX2axafRz8Xx7hx40Kct3ZUakm54WZxvrCW9fYlMTXPWOcnmTVrVrTdnDlzQqz5/mZxG0vLCdY8b18OXK+DLM9t5OfKSJunTXPt9d/u58DQ+1Fa29Ecdy3Bbhbnd8+ePTvEOq+JWXK/aRbfa/35UXr+J0+eHK2bP39+iIud611N0uYJ0OvHt3udi2rhwoUhzmq52GLSa9vf6/W5QI+nn2dB5yHxbWDw4MEh1rbty0VrW9R5Zfy8Nfq840uy637ps6efJ1LbrH6XWdwfpT3fVCL9N9RnnjFtH3q/888l2nb0mde3Re1r/Xxx+htHrxF/HVTb82XSnHV+Thv9/ejnx9M2oX2bn2dTz6M+w6xatSraTuc38vdn/R2oc/b5+fW0PfvfK3p96Xb+M/RaaOz5+9LoedPnCH8O9Tz546rtZeDAgSH2/a62bz2fWjrdLL0963frnDa+79Y5kdLWlfJ+ykgbAAAAAACADOKlDQAAAAAAQAaVJT1Kh3Fp6otPd9G/86XZVFLZSjOzKVOmhFiHR/kUHB1K7stA6xBHTdnyQ411iK0fSq7b5mG4qQ7L0zSVtHPohwDq0FP9DF9uVIeu6XH1Q5X1GvHXi54bHdroU2x0f31qg/6bNSY9qv50KPZZZ50VrdP25tNutJwmx7l+9Hr2w7a1bKKmR/m+UFOR5s6dG+LXXnst2m7BggUh9qmoOkRfY00n8Ot8eqxeP76MdJb4PkSHA/vhtDq8W7fTFAyzeDiw7wOTypROnTo12k5Lyup2PtVM+02fvqb7qOkk/r6o15AvG563+2I56fHS8rEHH3xwtJ0O0ff3O22nPl2mmqSlzGjsy/tqedmePXtG6zRFQ9NuNE3fLE4ZTEqVMktOEzcz69OnT4i1zfoSwfrs7VOBtK/S5+Esp10UKu15TY+Xv1fp7wR9bvTppnqu9Lt8Gps+H/v+f5999gnxyy+/nPhdWS7tXAxp94G085iW5uZ/731pyZIl0fL06dNDrKlT/u/1d4h/DtV2r/dTnyKj15r+FjKL7+Maax9gFv82yvLzsPYthaYq+ilG9HlQ+zWffq9pbfr7X59XzeLj79O0lP4e9cdYf9P6NEa9Znz6VTEx0gYAAAAAACCDeGkDAAAAAACQQSVJj/JDinSIrg4p8sMFdQiUn91fh7/pzN5+GPgbb7wRYp3VXYc6mm2ezqSSUrN8uo8OY/TD2HQYWJaHsTWEDp/1x0SlDRHTz/DDCPXc6DXiUyZ0GL4f7qbDUnXYo08H0HNTaHqUP595PtdbQ4+LVsDp169ftJ229bFjx0br/BBEJPPXr/ZjvqrI0KFDQ9y7d+8Q+1SXxYsXh1iHcGvamlncxnxfq32Epv74Yd+acuOrQfhh/1/KWnvzw3/Thn5r/6htwKey6TpNpzCL73fz5s0LsU9f0+HY2jem9b1pQ7M19kPT9Zz4flmvrzykYZSSv3b03GjFEl/xTVMJtXqNmdn48eNDnPe0izS+30iqyuKrTWpamm+n+hlaJe3FF1+MttN2qs+QPtVc25hPHdB2pX2jb4t6jn0auj6X5y09Kq1yot4n/fHSvlaPj0/J0L/T8+7vU5pi6u9pvXr1CrFWx/VtNu/ttNC26FM99TnDpzNpu0pK8TaLp9fQZ03fVrTd+9+t2jb1WvP7q9N6+BRv/U2raTz+HqyfmeVpGnRf9BnDX8t6fv1vcl3W9vfKK69E202YMCHE2rfq73+/Tz5tW8+hpqD7Y6qf6duprtPzVOz+lJE2AAAAAAAAGcRLGwAAAAAAgAzipQ0AAAAAAEAGlWVOG81j05J4vuyZllf0pU01z0/zFTWn3yzOFdQcYZ//p/M9aH64WVzWUfPDFy5cGG2npcb8XAN5yxFOypf0/13/rT5/MakMnM9l1PzRvfbaK8RaItHMrGvXriH2ucSan6q5hj5vPK2koO5vHs5huWl++IABA0LcsWPHaDvtE1544YVonc8LRiwtd1/nXNh3332jdQcddFCIte34kph6PiZOnBhincPGLG5Xfi6rpPmrfLtPKyOt+etZyt32/Bwxev36f1PSvDB+TjXNr/e58IsWLQpx2v0obR4bpedA731mcXvWc+DbqM4hoLn6ZvE9IcvnsbHoPcfPUaUl4nU+DD+njZ5rnVvFzGzatGkhrubjnzYfhPZXfv4KfTbx5cC1neqcX37ug6VLl4ZY5+Lw50PnrdHnILN4HjK9n/p/lz4HaV9hljwHQ96uCz9vjd4Xdc4Zs/iepOfT/x7ROTG0T/N9tz6jHnLIIdE6/Uyd08ZfVzoXSjVImj/SP1focfdzv+gx0+38vUrPl7Y3Pxdbt27dQrz//vtH67p37x5i7Tv8/Jl6L/RtUX/TJpWd95+fZUnPB2m/CdPKcOvxmjRpUrRO+1ft0/xzjvbl/rekPg9rX6vvHczMZsyYEWI/t46e31I+5zDSBgAAAAAAIIN4aQMAAAAAAJBBZUmPShqO7Yf96RA3PyxMUyj8sDOlQ/116JsvIa4pUH7Yog431XK1Wk7MLB5qrPtnFg+tzNtwU+WHcOuyP+Y6/E2HrPqhp3pudNioDlE0i4fv+6GTmuahaXj+XOg+pqUv6Hb+eqbkd910mO/AgQND7Muojhs3LsQ+3ZHjmU6vS59u1LZt2xBriW+zeHio9sN+yKeWq9Vh/b7v1v7aD+FN6hN8+o32Cf4zfPv+UtauD9836HHyZUS1jGVaWVc9fmmpTXqM/DD9pL/zfbReM126dInW6flKKo1rFt8n/TBwvU/Sb25Oj4k/N5qSofdFX75U07h96XdNl6k2SWkXZslpmz7tWlPUfBvTofTvvPNOiPUZ0iw+r2nlurVEtA7fN4tT4jRly6et6v00LT3Kp1XliX+u0+fNdu3aRev0WLZq1SrE/n6n9ydNZ/FtVu+z/rv0mLdu3TrE2gebxdeS78fz2G8m/Zv8M4G2Kz/tQVIp6Q4dOkTbacqM3p99u+/Zs2eIBw0aFK3TdDttf77k9IIFC0Ls05z1HKc9S6X1YVlS6H7qs43v//R8rFixIsS+vLvSc+F/m+q5P/nkk6N1/fr1C7Eecz1nZvHzsJaLN4vfS6Q9p20tRtoAAAAAAABkEC9tAAAAAAAAMqjs6VG6zg8b1fQon27kUyq+pMOhzOIhjTrk3A9bHDJkSIh1aJRZPAxch7v5ocaaguOHvmd56FpD6FBOjf2QNj1PfjZwHZaqw4x9NaHOnTuHWFOl/LBR/XxNNTCLhyrrufD7q0Mn09I69BpOG/pWKbO7l4IfYt2+ffsQa/Uiv92UKVNC7Ichk0KxuaQKM76P1BQKTU8zi9uBzr4/fvz4aDutPqPnxqevJqUv+f3SlBufHqWf4VNgdfhzJV0Hepz8PUKHT2v1CD/UW4f36/B9szg1Ru99hVZd89eMDgPXCkX+u7Xf1KHdZnFKhv67zOJzTHW+LyTdW/3wbq1qoW3b39P0fPiU7rR2Wk38PVyfFTX29yp/rJM+U1OnNM3JLH6O0fPhn2X12WfYsGHROn1G0udoX5FVK8r5Sija5pKuQbPK6m+/pPvs71XK9399+/YNsaY2+X5Mj7Oma/j0G61M69O0tJ/XZ2Dtg83itGR/D9FzX4nnaUsKTWn09w895/q7w1dh0+Ouvyd8W9Rz4tNRtV1p7NNntKKjT/FJqu5Yqec06bylpb379wH6DkCfPX1FP63Qpp/htxs+fHiIv/KVr0TrtIqc9pl+uoC0KVHK9QzDSBsAAAAAAIAM4qUNAAAAAABABvHSBgAAAAAAIINKMqeNl5TT5sucap6fz83VfFDNF/blonXuGv1eP7+K5oRrPptZnFOoOWzTp0+PttM5eHyueKXmIhZCz4XPCdYyzzrHglmcy625v717907crkWLFonfpcfclzLVHFGdGyCthKfPV9dzqNeqP7e6rhpKMibxczDoPDY6b5RvK5pD6ufzqKbjVyidc0Dbor+2dd6DPffcM1qnOcI6B4mfA0P7OM0X9vm7Sfvkv3u//fYLsfYVZnGO8Ny5c6N1el3od2f9+tD+wM+7pf2GluN98803o+20VKyWhvWfoXxJUT1Oeo/0ufs6D5W/L+p9V8+3XiNmcV/s89Szfr4aQ1K5ZV9SWuc4GTBgQIj9PUfPvfatZoXPdZR3vv/S46LzafmyvXpt+2dPvf/pudJnDLP4OSPpudZ/vp8rRftOncNI5yAzi+fV8PNopM0Jkid+njydw82XUtfz0b179xDr3DRmcX+tn+/nXdRrws/9pfc77cd9H6/n2vefet3mcb6qQue08fMW6bLOv+bvaUlzZvq543T+PX+/0/Oo59j3HXqOfT+s7S/t+aYS75+6z/7frX2tnzcq6dz7c6PHK6kPNjM79NBDQ+znGdNzo/PY+Dke9Zz6a445bQAAAAAAAKoYL20AAAAAAAAyqCzpUUn8sF4dKqXltM3iIYh+eJTS4VeaEqVlwczicn4+LUbLfD/33HMhXrFiRbSdDqnyQ6MqcRhbGh12psMNfYqDlsPTY2wWl9vr06dPiDWNwyx9yLDyQ++VDjn3w02VXiP+8/ScaqlFP5xdr1s/ZC5v10EaX2JvyJAhIdbUHT9sdNasWSH2Q5mr6fgVKqk8qz/+OqTeD9HXofKajuOHqOpQYO1bfSqc9qF+CLKmcugwc9+2NTVr5syZ0bpKTY9Sfvh6UolzTck1i1NEtWS6WXzOu3XrFmLf5+nn69/49Cg9x76kZVIajy/P7ttwkqTPqzZJw8B1SL5Z/AyjKYc+7eXVV18NsaZxmOU7DWZr6HHR5wBNHTWL+zZfflmfJbR/9M9I+tyoqYq+PehzrvYBZnE71X187bXXou30mdXvbx5KCxfCp5Bq6u3zzz8frdNzpcfc/0bQ8s16rv12WgLap8fq/U9T//19XJ+3/TQPel/MS3qUtgM9tv6ZI+lvzOL2nJYOrOdY1/njrHx6lLYxvffp95ptnjZejXx6lE6D4p/5NM1Xz42/h2k/pundvXr1irbTsu1+P6ZMmRLihx56KMQLFy6MttM+NG0qjFL2p4y0AQAAAAAAyCBe2gAAAAAAAGQQL20AAAAAAAAyqOxJdknlv83i+UB8eTzNI9R5F3wuoy5r3reW7zOLS377nFct+TV16tQQ+7LSmheXh9JshdLcTD93gs5Z0b9//2idng8tY5uW852WL6w5hT5XVefP0bkB/HxImgPpc4I1H1nnbUgr7+5LVieV5M0LbW9+LpPDDjuszu18nqjOqZKXvOzG4POw9dr2+dWae639aVrJdc2t1xx8s7iN6Rw2ZmZHH310iLVd+rlQxo0bF2I/l4TmEldq3+rzqPUcaF/j53PTOUr8fFB6TnSOBJ9vrfdPPVeaU24W55H369cvWqf9r96r/f7qv8XPo6F/R1v/QtL1rPPDmZkdeOCBIdZzocfbzOzZZ58NsT/+ldp2Sk2Pi16jvr1NnDgxxDqviVk8b5vOS+LbvX6+3jN1Tiq/zs//pfMuPvXUUyHWvsIs7tv9nH26H3me38b/ztC5up5++ulonR4vLZfuy7u//vrrIe7YsWOI/X1W72N+Tpvjjz8+xPp7xM8/p/zzZB76UP8bTo+h9nPavvyyfx7RZyH9veKvbb0v6nWSNoeUv99pH6Hb+bmJ0vajUufpK4T+2/z1qm3R95PaFvX8+rmBtG3qPJqDBw+OttNnJZ1H08zsnnvuCbHOC+Z/82uf2Vjz2DLSBgAAAAAAIIN4aQMAAAAAAJBBjVqDzA8n0iGafiinDp3SIfx+aJ0OgdJUnSOOOCLaTtNktASgmdnDDz8c4qVLl9b5vX5/8zakzdPjrMMXfXpU7969Q+xLfutwXx2i78tkJ5UX9+lRus6XA9QhczoEz5ej1eGmPl1Dh51rGoHfXx32mvfrwNNrYejQodE6TYXRY6ZDzM02Ly2MhvHtQ4eU+mHV2g50yGpaO0orL66lqI855pho3UEHHVTn5+sQczOzZ555JsQ+9UD3sVLLFvu+Qf9NWrZ5wYIF0XbaD82YMSNap+dBUyjSymLq3/h7mg4r79OnTx3/ii9onzd//vxonaYE+DRn/TdXW19ZCG3DgwYNitZpirc+D2kKt1k8fJ9jXBg9TnqN+ucFfR5ctWpVtE7bn94XfaqirtOUqOHDh0fbaZ/q++8JEyaEWFN8Fi9eHG2n6XFpw/mr6TrR86vlms3i/kp/F/i0etWhQ4cQ+zQ2TTf192fdj06dOoVYpxEwi5+P9D5hVrn3wjT6XK/THrRq1SraTp9N/BQL+ne6zqeLJqXg+PuiplH5lGLti/Uc+2tBz5VPBUpqf3lol2lTouhx8KlT+qyo/bD+djSL+0mdkkGnBzCLz/X9998frXv88cdDrOlu/j1EFtLYGGkDAAAAAACQQby0AQAAAAAAyKBGTY/y0oZr6rIO0/fD+XWImw6V8jPz65DV5557Llqn1aM0ZcYPc83D0LVCJQ1x80M+dWinDhs1i2dk18/w51CXdbh+WnqUn7Vft9XP8DPz63BY/Tyz5NSpQq/NaqDHzA/v1raoKR5PPPFEtJ0fKopYodeUn1VfpVU1SxqObBYPWdV2pKlvZmannnpqiH0qqn6mphRo1ROzeMh/NaSi6r9D24AfAq99j1Y0MYv7Oe0D/THSayjpb8zMunbtGmJfDU7PgaYV+JQtTQkodBh4NdP7nQ7l9+mmWi1F09O08oXZ5ik92DK9LvU698982m/6ofO+LX3J9996X9Sh/r5amz6raLUos/geqtUY0/pNbM73R0lV8fzx13umVr/09J7pn4f1utB279OjdLvGqlhTTtqO9Lj4tJikarRm8bQN+nk+BUf7Ua0U5NN6lZ8eQc+JtlmfRqVt0+9HFtJuiqnQdK+0Z1vdVu+RvpLbV77ylRBr5VJ/H3zyySdD/Oijj0brNB1f+/Us9p+MtAEAAAAAAMggXtoAAAAAAABkEC9tAAAAAAAAMihTc9qkSZrvxs/joKXzTjvttBD7knCLFi0K8ejRo6N1Ov9GHudSaAjNudScP5+3mVb+Ws+B5vr6vMGk3E+fB6rnSWO/H/p3WrrPLM5P9fMvaF6rzsXhcyWreU4WLYXpS7zrcZo9e3aItZQmtixtDiXlr23t4zp37hyt0/kxevToEWJ/bWu50TZt2oRYc4fNzA499NAQ+5xjnf9Ey9M++OCD0XZJ/a5ZPkubFiqpHHFdy0k0d1xz/P08Xtovp81po3M8+JLfaWWGsTk9N23btg2xPsuYxffdadOmhfj111+PtqvWUs7lkDT3jV/Wc+qfUZs2bRriww8/PMT+/qmfMWvWrGjd9OnTQ6x9Nu2teJLmHPPLep78HB26nd5LzeL5kbRP1jmPzMx69uwZYj/XSh7Pd9KcUmnzl/py4Pqsor87fH+o50Tvhf48Js0J5/dLt9O56Px3pc2VUk19dtq/VdfpfEbDhg2Ltjv22GNDrO1o8uTJ0XZ33XVXiBcvXhyt09+BWW9TjLQBAAAAAADIIF7aAAAAAAAAZFDFpEcpHYLmS9Sec845Ie7evXuI/XA0LTc7Z86caJ0OW0wr51ytw9h0mPaCBQui7V544YUQ+2GEWsq2Y8eOIfZD/HUYoaY5+aGhmnah25nFaVuaNuLLcuqyL+Gp36fftXLlymi7NWvWhLgaUqX0mO29994h1jKLZvFxnzdvXojTSryjfpLSFs3iIaB+qK62Py2ZOGTIkMTP33HHHUPsU2d0+Kq2BzOzMWPGhFiHqGr6llllDVGtZNqX+/6wRYsWIfYlVvUequmi/nxz7upHz4GmRKX1p1OnTg2xv3+mlVFF+fkURE0tPfroo0Psn2X1OWP8+PHROn0eyWJZ2mqS9jtAf0toSWkzs0mTJoX41FNPDXGvXr2i7fSe7FMh00pTVyq9f+jztL+v6LJvA9qn+n5UaZq40vNmFv8m0fRfvx/6O0TLSJvF0z4UmvJezfR5s0+fPiE+8cQTo+00FU6P+X333Rdtpymlei7MKmsaFH4tAQAAAAAAZBAvbQAAAAAAADKIlzYAAAAAAAAZVDFz2mietpZ623fffaPtdH4GzYnzeffPPPNMiHWuFbPkHOGs57qVix4fX/Jb57R5++23o3VaulDLAvu5OHRZ5+Lw+bv6+Wk5inrt+PxynU/Fz62i3605rr68eNIcSHmlcxX17t07xGkl2f11oqrhmBWTHi/Np/YlRbUU89KlS6N1HTp0CLHmBO+5557RdknlLP3cTTov2N133x2te/zxx0OsbdaXF6+kvOK80BLfZnFpeL9Oz7nOt+HnS2KOjXR+zhk9zj169AixLxWtzyka++1oO41Pz4kvR6zPqO3bt0/8DJ23Ruc/Mdu870Q2+Lanz0T++XXKlCl1bqf3YzOz448/PsSvvPJKtE6ft/PS7vX+ocfF31e0Dfjnf/3tp78Xfd+r63TOIT9vjT5b+d+S+pyrbXbZsmXRdjpnHzan58ws/m2hcz7pPdIsfv546aWXQvzEE09E2+k90/9WqaS2w0gbAAAAAACADOKlDQAAAAAAQAZVZHpUu3btQnzWWWdF22kJYk0d8GW9tXSeH+qvQ6UqadhUuSSV/zaLS64tX748Wvf8k6bcKQAABWBJREFU889v8fMauh9+2KMup6VA+ZK3Sq8fjSt5aF1D+GOblB6n6ThmcQqFpiP6VCl/PFE4PRe+7L2WOGzWrFm0Tvs8LSnqh/Jr6p+mNunQbjOzUaNGhdgP5ddSxbq/lIYun6Q+ypcZ3meffULsrwUd3q3XWtqw77z3jQ2R1p/qEP3FixdH22l/qs8vaWmGKJ+kFP7+/ftH2w0bNizEWnLYp3hoP/rmm29G67TNaT9Ke8su3y4nTpwY4tNPPz3EI0eOjLbTZ1btn83MJkyYEOK8pN9oO0qaKsEsbjt+yoJVq1aFuEWLFiH2z/uasrZw4cIQ+xQovd/579JtNdZ9MIufc2mnX9Bru2XLltG6gw46KMSHHnpoiP1zif7u0FR8TVUzSz/+lXQ+GGkDAAAAAACQQby0AQAAAAAAyKCKSY/SmaW1YpQfeqo0PUfTM8ziFI1KGhpVSco5BC3tuwpNw/DD1rkuvuCPg6bWjBkzJsQvv/xytJ0O19U0G59Sx3Gun6Th8HqMzeJ0Jj+Tvg6r1uHDPn1QU5vShgjrOSXtKdv0HGslOLO4SoZPd9R1er59+9V+lD51c2n9qVa/0HQos/h5Rp9ffPVL2l/j0Ha1yy67hLhLly7RdlpZSs+jrzaj6eQ+9ZVUi8qn7XTcuHEhPv/886PtRowYEWKfCpnHc6/PMdq3+fRBnYph3rx50brXXnstxFqNy1co0pQrvb/5lH293/n90EpTup0/V7TZzZ8HmjdvHuKBAwdG6/bff/86/86nPT377LMhfvXVV0Psn4fzcswZaQMAAAAAAJBBvLQBAAAAAADIIF7aAAAAAAAAZFBm57TRvF+zOPdNc4R9jpzmu2meqJYCM4tzJX0pvrzkvqF+OO/1p/PW6BwqKA+9Zn0eti77PGxfdh3VQ68ZnU/FzGzWrFkh9nNPKS1H7LdjTpV0/j6jufc6j5CfUwjZ4p89dU4bLS2sZYXN4rajbeWVV16JttN5GH25Y31m5bml8ul1MH369Gidlnv359rf8ytR2vWr/z7/b9W5ZHwb0/mhdtpppxD7Npt0r/Jz+2l78/dMXZfWLqu1neox13NhFs+n6Et567FcsmRJiCdOnBhtN2rUqBDrbxD/u15V8rlgpA0AAAAAAEAG8dIGAAAAAAAggzKbHuWHwmk5tqlTp4a4adOm0XZaGlHLvr3xxhvRdjrErZKHSgEAUCgdruyHeut9UlOlzOJUSE1DTkuP4t6KvPLXtl73mn4/fvz4aDt9LtW2s27dumg7TYny7RT5oteS9rNm6emmpKJ+wR8HPYb+eKK80q5t7eM0Bcos/i2v202bNi3abu3atSGuhvbASBsAAAAAAIAM4qUNAAAAAABABvHSBgAAAAAAIINq6pNzXlNT02gJ6pqH36RJkxD7EmKa+6sl4Sq9rHdtbW3NlrfassY8h7DJtbW1g4rxQZzHxkNbzIWqbYt6L91uu3haux122CFxnZam1vtsY5adpS3mQu7aopYM9mWGtV0llQs24xkVjSJ3bbEaZbUt+r5w2223DbEvs679n85Vk1bKO2fqbIuMtAEAAAAAAMggXtoAAAAAAABkUH1Lfq8xs8Wl2JEt0aFSH374YZ1xjnUq4mc12jkE5zEHOIf5ULXnUe+lvpRwhZUWrtpzmDO5O49ppWdzOrw/d+ewSnEeK19mz6FP+WzM1OoKUOd5rNecNgAAAAAAACgP0qMAAAAAAAAyiJc2AAAAAAAAGcRLGwAAAAAAgAzipQ0AAAAAAEAG8dIGAAAAAAAgg3hpAwAAAAAAkEG8tAEAAAAAAMggXtoAAAAAAABkEC9tAAAAAAAAMuj/AUetO+6UrloAAAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["Investigation ends here."],"metadata":{"id":"mAFw6BJMewIs"}}]}

docs/notebooks/ex10 - dcgan on masked mnist.ipynb

@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","metadata":{"id":"rF2x3qooyBTI"},"source":["# Deep Convolutional Generative Adversarial Network"]},{"cell_type":"markdown","metadata":{"id":"ITZuApL56Mny"},"source":["This tutorial demonstrates how to generate images of handwritten digits using a [Deep Convolutional Generative Adversarial Network](https://arxiv.org/pdf/1511.06434.pdf) (DCGAN). The code is written using the [Keras Sequential API](https://www.tensorflow.org/guide/keras) with a `tf.GradientTape` training loop."]},{"cell_type":"markdown","metadata":{"id":"2MbKJY38Puy9"},"source":["## What are GANs?\n","[Generative Adversarial Networks](https://arxiv.org/abs/1406.2661) (GANs) are one of the most interesting ideas in computer science today. Two models are trained simultaneously by an adversarial process. A *generator* (\"the artist\") learns to create images that look real, while a *discriminator* (\"the art critic\") learns to tell real images apart from fakes.\n","\n","<p align='center'><img src='https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/images/gan1.png?raw=1'></img></p>\n","\n","During training, the *generator* progressively becomes better at creating images that look real, while the *discriminator* becomes better at telling them apart. The process reaches equilibrium when the *discriminator* can no longer distinguish real images from fakes.\n","\n","<p align='center'><img src='https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/images/gan2.png?raw=1'></img></p>\n","\n","This notebook demonstrates this process on the MNIST dataset. The following animation shows a series of images produced by the *generator* as it was trained for 50 epochs. The images begin as random noise, and increasingly resemble hand written digits over time.\n","\n","<p align='center'><img src='https://tensorflow.org/images/gan/dcgan.gif'></img></p>\n","\n","To learn more about GANs, see MIT's [Intro to Deep Learning](http://introtodeeplearning.com/) course."]},{"cell_type":"markdown","metadata":{"id":"e1_Y75QXJS6h"},"source":["### Setup"]},{"cell_type":"code","execution_count":1,"metadata":{"executionInfo":{"elapsed":3486,"status":"ok","timestamp":1711680278794,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"WZKbyU2-AiY-"},"outputs":[],"source":["# import\n","import tensorflow as tf"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5,"status":"ok","timestamp":1711680278795,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"wx-zNbLqB4K8","outputId":"97a909fc-d112-4e4d-caed-e842b00c9481"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'2.15.0'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":2}],"source":["tf.__version__"]},{"cell_type":"code","execution_count":3,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":12482,"status":"ok","timestamp":1711680291273,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"YzTlj4YdCip_","outputId":"702c1c89-90bf-4a5c-d973-4ac4f528404d"},"outputs":[{"output_type":"stream","name":"stdout","text":["Collecting git+https://github.com/tensorflow/docs\n","  Cloning https://github.com/tensorflow/docs to /tmp/pip-req-build-0w_nfusd\n","  Running command git clone --filter=blob:none --quiet https://github.com/tensorflow/docs /tmp/pip-req-build-0w_nfusd\n","  Resolved https://github.com/tensorflow/docs to commit ff989f0d94cd81cce45a8db0f540e605ce05512b\n","  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n","Collecting astor (from tensorflow-docs==2024.3.27.3713)\n","  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)\n","Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (1.4.0)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (3.1.3)\n","Requirement already satisfied: nbformat in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (5.10.3)\n","Requirement already satisfied: protobuf>=3.12 in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (3.20.3)\n","Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (6.0.1)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->tensorflow-docs==2024.3.27.3713) (2.1.5)\n","Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (2.19.1)\n","Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (4.19.2)\n","Requirement already satisfied: jupyter-core in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (5.7.2)\n","Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (5.7.1)\n","Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (23.2.0)\n","Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (2023.12.1)\n","Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (0.34.0)\n","Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (0.18.0)\n","Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core->nbformat->tensorflow-docs==2024.3.27.3713) (4.2.0)\n","Building wheels for collected packages: tensorflow-docs\n","  Building wheel for tensorflow-docs (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for tensorflow-docs: filename=tensorflow_docs-2024.3.27.3713-py3-none-any.whl size=182447 sha256=e60860cfdc398a23695337f5880c04c052d80494bdc26934de8df6629d1bdd93\n","  Stored in directory: /tmp/pip-ephem-wheel-cache-enowi8ad/wheels/86/0f/1e/3b62293c8ffd0fd5a49508e6871cdb7554abe9c62afd35ec53\n","Successfully built tensorflow-docs\n","Installing collected packages: astor, tensorflow-docs\n","Successfully installed astor-0.8.1 tensorflow-docs-2024.3.27.3713\n"]}],"source":["# To generate GIFs\n","# !pip install imageio\n","!pip install git+https://github.com/tensorflow/docs"]},{"cell_type":"code","execution_count":4,"metadata":{"executionInfo":{"elapsed":10,"status":"ok","timestamp":1711680291273,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"YfIk2es3hJEd"},"outputs":[],"source":["# import\n","import glob\n","import imageio\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import os\n","import PIL\n","from tensorflow.keras import layers\n","import time\n","\n","from IPython import display"]},{"cell_type":"markdown","metadata":{"id":"iYn4MdZnKCey"},"source":["### Load and prepare the dataset\n","\n","You will use the MNIST dataset to train the generator and the discriminator. The generator will generate handwritten digits resembling the MNIST data."]},{"cell_type":"code","execution_count":5,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2877,"status":"ok","timestamp":1711680294141,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"a4fYMGxGhrna","outputId":"0ec55962-aa3b-4bd7-c9ff-12656d56262e"},"outputs":[{"output_type":"stream","name":"stdout","text":["Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n","11490434/11490434 [==============================] - 2s 0us/step\n"]}],"source":["# get data\n","(train_images, train_labels), (_, _) = tf.keras.datasets.mnist.load_data()"]},{"cell_type":"code","execution_count":6,"metadata":{"executionInfo":{"elapsed":335,"status":"ok","timestamp":1711680294471,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"NFC2ghIdiZYE"},"outputs":[],"source":["#rescale\n","train_images = train_images.reshape(train_images.shape[0], 28, 28, 1).astype('float32')\n","train_images = (train_images - 127.5) / 127.5  # Normalize the images to [-1, 1]"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":11,"status":"ok","timestamp":1711680294471,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"CsIROTs1zWSG","outputId":"2d940556-55bc-4620-9ce1-e23e5eb20600"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["((60000, 28, 28, 1), numpy.ndarray)"]},"metadata":{},"execution_count":7}],"source":["# display shape\n","train_images.shape, type(train_images)"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":10,"status":"ok","timestamp":1711680294471,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"88h4t6pT2ldg","outputId":"3389902b-c0cc-4163-9164-a916957616c6"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["((60000,), numpy.ndarray, array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8))"]},"metadata":{},"execution_count":8}],"source":["# labels\n","train_labels.shape, type(train_labels), np.unique(train_labels)"]},{"cell_type":"markdown","metadata":{"id":"g5PjXIvVzbPc"},"source":["This is the information you need to verify if you use your own dataset."]},{"cell_type":"code","execution_count":9,"metadata":{"executionInfo":{"elapsed":9,"status":"ok","timestamp":1711680294472,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"S4PIDhoDLbsZ"},"outputs":[],"source":["# args\n","BUFFER_SIZE = 60000\n","BATCH_SIZE = 32"]},{"cell_type":"code","execution_count":10,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":9,"status":"ok","timestamp":1711680294472,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"2tFY7qMfiA6z","outputId":"6bdf36b8-9718-41b8-f33d-8102aeef84b2"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["1875.0"]},"metadata":{},"execution_count":10}],"source":["60000 / 32"]},{"cell_type":"code","execution_count":11,"metadata":{"executionInfo":{"elapsed":1026,"status":"ok","timestamp":1711680295490,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"-yKCCQOoJ7cn"},"outputs":[],"source":["# Batch and shuffle the data\n","train_dataset = tf.data.Dataset.from_tensor_slices(train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"]},{"cell_type":"code","execution_count":12,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":9,"status":"ok","timestamp":1711680295490,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"0RtaLSObzi94","outputId":"0e8a5d53-f438-4262-c4e1-c741020c1d0a"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(<_BatchDataset element_spec=TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name=None)>,\n"," tensorflow.python.data.ops.batch_op._BatchDataset)"]},"metadata":{},"execution_count":12}],"source":["# train_dataset\n","train_dataset, type(train_dataset)"]},{"cell_type":"markdown","metadata":{"id":"THY-sZMiQ4UV"},"source":["## Create the models\n","\n","Both the generator and discriminator are defined using the [Keras Sequential API](https://www.tensorflow.org/guide/keras#sequential_model)."]},{"cell_type":"markdown","metadata":{"id":"-tEyxE-GMC48"},"source":["### The Generator\n","\n","The generator uses `tf.keras.layers.Conv2DTranspose` (upsampling) layers to produce an image from a seed (random noise). Start with a `Dense` layer that takes this seed as input, then upsample several times until you reach the desired image size of 28x28x1. Notice the `tf.keras.layers.LeakyReLU` activation for each layer, except the output layer which uses tanh."]},{"cell_type":"markdown","metadata":{"id":"msfaH8ZYz6FC"},"source":["We use *assert* for debugging purpose. If there is an error in the code, the *assert* function will raise this error. The API for *assert* is [here](https://www.w3schools.com/python/ref_keyword_assert.asp#:~:text=The%20assert%20keyword%20is%20used,program%20will%20raise%20an%20AssertionError.)."]},{"cell_type":"markdown","metadata":{"id":"Gp7BXqDFUIPV"},"source":["#### Conventional Generator"]},{"cell_type":"code","execution_count":13,"metadata":{"executionInfo":{"elapsed":8,"status":"ok","timestamp":1711680295490,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"6bpTcDqoLWjY"},"outputs":[],"source":["# define generator\n","def make_generator_model():\n","\n","    # use sequential API\n","    model = tf.keras.Sequential()\n","\n","    # start with dense layers\n","    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(784,)))\n","    model.add(layers.BatchNormalization())\n","    model.add(layers.LeakyReLU())\n","\n","    # reshape\n","    model.add(layers.Reshape((7, 7, 256)))\n","    assert model.output_shape == (None, 7, 7, 256)  # Note: None is the batch size\n","\n","    # add Conv2DTranspose layer\n","    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))\n","    assert model.output_shape == (None, 7, 7, 128)\n","    model.add(layers.BatchNormalization())\n","    model.add(layers.LeakyReLU())\n","\n","    # add Conv2DTranspose layer\n","    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))\n","    assert model.output_shape == (None, 14, 14, 64)\n","    model.add(layers.BatchNormalization())\n","    model.add(layers.LeakyReLU())\n","\n","    # add Conv2DTranspose layer\n","    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))\n","    assert model.output_shape == (None, 28, 28, 1)\n","\n","    return model"]},{"cell_type":"markdown","metadata":{"id":"WVIlAm8YULEp"},"source":["#### Generator with Attention"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QshRMIu8KDtg"},"outputs":[],"source":["import tensorflow as tf\n","from tensorflow.keras import layers, models\n","\n","def make_generator_model():\n","    # Input layer\n","    inputs = layers.Input(shape=(784,))\n","\n","    # Dense layer for initial processing\n","    x = layers.Dense(7*7*256, use_bias=False)(inputs)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.LeakyReLU()(x)\n","\n","    # Reshape to fit for multi-head attention\n","    x = layers.Reshape((49, 256))(x)  # 7*7 = 49\n","\n","    # MultiHead Attention layer\n","    # Since 'query', 'key', and 'value' can be the same tensor for self-attention,\n","    # we pass 'x' three times to the layer.\n","    attention_output = layers.MultiHeadAttention(num_heads=8, key_dim=64)(x, x, x)\n","    attention_output = layers.LeakyReLU()(attention_output)\n","\n","    # Combine attention output with the original sequence\n","    # Optionally, you might want to combine or process the attention output further\n","    # before moving on to the next layers; a simple way is to concatenate.\n","    x = layers.Concatenate(axis=-1)([x, attention_output])\n","\n","    # Proceed with the model as before\n","    x = layers.Flatten()(x)\n","    x = layers.Reshape((7, 7, 512))(x)  # Adjusted due to concatenation\n","\n","    x = layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False)(x)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.LeakyReLU()(x)\n","\n","    x = layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False)(x)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.LeakyReLU()(x)\n","\n","    x = layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh')(x)\n","\n","    # Create model\n","    model = models.Model(inputs=inputs, outputs=x)\n","\n","    return model\n"]},{"cell_type":"markdown","metadata":{"id":"pNbhVaaGUQFo"},"source":["#### Generator U-net"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"sB0aHCR0URjl"},"outputs":[],"source":["import tensorflow as tf\n","from tensorflow.keras import layers, models\n","\n","def make_generator_model():\n","    inputs = layers.Input(shape=(28, 28, 1))\n","\n","    # Encoder part\n","    c1 = layers.Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)\n","    c1 = layers.BatchNormalization()(c1)\n","    p1 = layers.MaxPooling2D((2, 2))(c1)\n","\n","    c2 = layers.Conv2D(128, (3, 3), activation='relu', padding='same')(p1)\n","    c2 = layers.BatchNormalization()(c2)\n","    p2 = layers.MaxPooling2D((2, 2))(c2)\n","\n","    # Bottleneck with MultiHeadAttention\n","    # We need to flatten and reshape to apply MultiHeadAttention\n","    b = layers.Flatten()(p2)\n","    b = layers.Dense(49*128, activation='relu')(b)  # Adjusted to match the flattened size\n","    b = layers.Reshape((49, 128))(b)  # Prepare for MultiHeadAttention\n","\n","    # Applying MultiHeadAttention\n","    b = layers.MultiHeadAttention(num_heads=4, key_dim=32)(b, b, b)\n","\n","    # Decoder part, reshaping back to the spatial dimension needed for Conv2DTranspose\n","    b = layers.Flatten()(b)\n","    b = layers.Dense(7*7*128, activation='relu')(b)\n","    b = layers.Reshape((7, 7, 128))(b)\n","\n","    u1 = layers.Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same', activation='relu')(b)\n","    u1 = layers.BatchNormalization()(u1)\n","\n","    # Skipping connections\n","    u1 = layers.concatenate([u1, c2])\n","    u2 = layers.Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same', activation='relu')(u1)\n","    u2 = layers.BatchNormalization()(u2)\n","\n","    u2 = layers.concatenate([u2, c1])\n","    outputs = layers.Conv2D(1, (3, 3), activation='tanh', padding='same')(u2)\n","\n","    # Create and compile the model\n","    model = models.Model(inputs=inputs, outputs=outputs)\n","\n","    return model"]},{"cell_type":"markdown","metadata":{"id":"GyWgG09LCSJl"},"source":["Use the (as yet untrained) generator to create an image."]},{"cell_type":"code","execution_count":14,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2430,"status":"ok","timestamp":1711680297912,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"gl7jcC7TdPTG","outputId":"775a5968-dfea-40d1-f5f6-0889a6ca1b25"},"outputs":[{"output_type":"stream","name":"stdout","text":["(1, 28, 28, 1)\n"]},{"output_type":"execute_result","data":{"text/plain":["<matplotlib.image.AxesImage at 0x782de7f46b00>"]},"metadata":{},"execution_count":14},{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAo7klEQVR4nO3de1TXdZ7H8Rcq/ECFHyJyU1Dxvl5o1DRXs1pZL7NjVk5Hq5m1tnRqsTPm2syxncmayzI1F9tmnWr3zGizm3bZTW2azd3UwNOklve1ksS8C6go/AABEb77h0cmUov3N+AD+Hyc8ztH4fPy++HLF17+4Pd7/8I8z/MEAEAL6+B6AwCAaxMFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMCJTq438Hl1dXU6ceKEoqOjFRYW5no7AAAjz/NUVlamlJQUdehw9fs5ra6ATpw4odTUVNfbAAB8RUePHlWvXr2u+v5WV0DR0dGSpOzsbEVGRjY6V1NTYz5Wly5dzBlJ6tTJftrOnTtnzsTFxZkzJSUl5kyPHj3MGUk6c+aMORMMBs0ZPx+T38/tqVOnzJlu3bqZMxcuXDBn/PxEoLa21pyR/J0/P9dDRESEOVNXV2fO+PkcSVJhYaE5Y/m+dYmfiWh+f0Lk51jWj6myslILFiyo/35+Nc1WQMuWLdPPf/5zFRYWKiMjQ7/+9a81ZsyYL81dOqmRkZGKiopq9PH8lILl3/+qx/LzSe/cubM5U11d3SLHkS5eZC1xrJb8mPxcE34yrb2A/Jw/P9dDSxVQS14PFNCffdkem+VBCK+88ooWLlyoJUuWaMeOHcrIyNCUKVN08uTJ5jgcAKANapYC+tWvfqW5c+fqvvvu01/8xV/o+eefV+fOnfW73/2uOQ4HAGiDmryAzp8/r+3btyszM/PPB+nQQZmZmdq8efNl66urqxUKhRrcAADtX5MX0OnTp1VbW6vExMQGb09MTLziL/Sys7MVDAbrbzwCDgCuDc6fiLp48WKVlpbW344ePep6SwCAFtDkj4KLj49Xx44dVVRU1ODtRUVFSkpKumx9IBBQIBBo6m0AAFq5Jr8HFBERoVGjRmnDhg31b6urq9OGDRs0bty4pj4cAKCNapbnAS1cuFBz5szR6NGjNWbMGD3zzDOqqKjQfffd1xyHAwC0Qc1SQLNmzdKpU6f0+OOPq7CwUNddd53WrVt32QMTAADXrmabhDB//nzNnz/fdz4UCun8+fONXu/nGd/9+vUzZyTpww8/NGf8PHvbz7OPY2NjzZmKigpzRpL2799vzvh55rufR0bu3r3bnJGkrl27mjPHjh0zZ/r06WPO+Jk0UF5ebs5IUkxMjDkTHx9vzviZCOFnMsahQ4fMGcnf9eBn7Jaf6QR+xkZJ/r6eqqqqmmW980fBAQCuTRQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwotmGkX5VFy5cMA0qTEhIMB/D78DKIUOGmDNbtmwxZ/wMQiwuLjZn/AwIlaQTJ06YM+PHj/d1LKvo6GhfuZKSEnMmOTnZnLEM2r2kc+fO5szJkyfNGUmaPn26ObNjxw5z5siRI+aMn8HDfr4/SNKZM2fMmfDwcHPGz1DWnj17mjOSv48pPT3dtL5Tp8ZVC/eAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4ESrnYbdv39/0/Tfw4cPm4/Rp08fc0aScnJyzBnrNFlJKi8vN2cGDRpkzuzZs8eckfxNTK6oqDBnOnSw/z9p//795owkDR061JzxM6W6R48e5syHH35ozowePdqckaQ1a9aYM7GxseZMUVGROePnGu/SpYs5I0kjRowwZ3bu3GnO9O/f35zZtWuXOSNJSUlJ5kxlZaVpfVVVVaPWcQ8IAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJxotcNIDx8+rMjIyEavv+6668zH8DvMb9SoUeZMbW2tOdOzZ09zJjc315zxM3BRkvbt22fOnDlzxpwpKyszZ/wMf5WkTp3sXxKFhYXmjJ/z4Geg5rFjx8wZSfI8z5y5cOGCOTNw4EBzxs/14Ofjkfx9jygpKTFnDhw4YM74Gf4qSbt37zZnrN+LGEYKAGjVKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOBEqx1G2qlTJ9NgyLy8PPMx+vTpY85IUrdu3cyZP/3pT+ZMx44dzZmMjAxzJj8/35yRpJMnT5ozNTU15sz8+fPNmTfeeMOckaSPP/7YnPEzzDU1NdWc2bp1qzlTXV1tzkhScXGxOfOd73zHnFm7dq0507VrV3PGz4BQSRo5cqQ5s337dnMmLCzMnKmsrDRnJOnb3/62OXPo0CHT+oiIiEat4x4QAMAJCggA4ESTF9ATTzyhsLCwBrfBgwc39WEAAG1cs/wOaOjQoVq/fv2fD+LjRb4AAO1bszRDp06dlJSU1Bz/NACgnWiW3wHt379fKSkpSk9P1z333KMjR45cdW11dbVCoVCDGwCg/WvyAho7dqxWrFihdevW6bnnntPBgwd14403XvV13LOzsxUMButvfh6eCgBoe5q8gKZNm6Y777xTI0aM0JQpU/Tf//3fKikp0auvvnrF9YsXL1ZpaWn97ejRo029JQBAK9Tsjw6IjY3VwIEDr/pkx0AgoEAg0NzbAAC0Ms3+PKDy8nIdOHBAycnJzX0oAEAb0uQFtGjRIuXm5urQoUN67733dPvtt6tjx4666667mvpQAIA2rMl/BHfs2DHdddddKi4uVo8ePTRhwgRt2bJFPXr0aOpDAQDasCYvoJdffrlJ/p1LUxQaa8CAAeZjnDlzxpyRpD179pgzXbp0MWf8DLn0MwixX79+5owkFRQUmDN/93d/Z84cP37cnImJiTFnJH3hUwau5i//8i/NmV27dpkznTt3NmesQyQvmTp1qjkTFRVlzvh5krqfgba33nqrOSNJH330kTkzfvx4c6awsNCcGTp0qDkjSaWlpeaM9ff0dXV1jVrHLDgAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcKLZX5DOr7i4ONNww7feest8DL/D/OLj482ZUChkzvzLv/yLOZOVlWXO+B0gO2HCBHOmQwf7/3n8DJ+Mi4szZyTpuuuuM2eu9mq/XyQtLc2cOXfunDkzf/58c0aS/vVf/9Wc8TP41M+QXj8vYLl582ZzRrr4gppWfgYC33jjjebM2bNnzRlJSk9PN2cOHz5sWt/YIbPcAwIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATrXYadnFxsSIjIxu93s9k61OnTpkzktSvXz9z5tNPPzVnHn74YXNm586d5kxpaak5I9kn5EpScnKyOeNnknFGRoY5I/mbWj59+nRz5oknnjBnZsyYYc4cO3bMnJGkkSNHmjMfffSROfP1r3/dnPEzdbuwsNCckfxNYh84cKA542dKvN9r/PXXXzdn+vTpY1p//vz5Rq3jHhAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAONFqh5F6nifP8xq9/uOPPzYfY8KECeaMJL3zzjvmTEJCgjnz05/+1JxZuHChOXP69GlzRvJ3Hk6cOGHOfOtb3zJnTp48ac5I0t69e82Zrl27mjPz5s0zZwYNGmTO/OQnPzFnJOns2bPmzM9+9jNzZsOGDeZMWVmZOVNXV2fOSFJmZqY5s3r1anPmyJEj5oyfcyf5G2r7xhtvmNYzjBQA0KpRQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwIlWO4y0trZWtbW1jV4/bNgw8zGKiorMGcnfUEg/wwbvu+8+c8bPEMm7777bnJGktLQ0c2bgwIHmzL59+8wZyyDbz/LzMfkZWLlu3TpzJjc315zxc74lKTY21pwpLCw0Z44dO2bOBINBc2b48OHmjCTt37/fnKmsrDRnLly4YM7069fPnJH8Deq1DlOurq5u1DruAQEAnKCAAABOmAto06ZNmj59ulJSUhQWFqY1a9Y0eL/neXr88ceVnJysqKgoZWZm+robCwBo38wFVFFRoYyMDC1btuyK73/66af17LPP6vnnn9fWrVvVpUsXTZkyRVVVVV95swCA9sP8IIRp06Zp2rRpV3yf53l65pln9IMf/KD+Vfd+//vfKzExUWvWrNHs2bO/2m4BAO1Gk/4O6ODBgyosLGzwqKBgMKixY8dq8+bNV8xUV1crFAo1uAEA2r8mLaBLD8NMTExs8PbExMSrPkQzOztbwWCw/paamtqUWwIAtFLOHwW3ePFilZaW1t+OHj3qeksAgBbQpAWUlJQk6fIneBYVFdW/7/MCgYBiYmIa3AAA7V+TFlDfvn2VlJSkDRs21L8tFApp69atGjduXFMeCgDQxpkfBVdeXq78/Pz6vx88eFC7du1SXFyc0tLStGDBAv3kJz/RgAED1LdvX/3whz9USkqKbrvttqbcNwCgjTMX0LZt23TLLbfU/33hwoWSpDlz5mjFihX63ve+p4qKCs2bN08lJSWaMGGC1q1bp8jIyKbbNQCgzQvz/E5tbCahUEjBYFAvvPCCoqKiGp0LCwszH8vP0EBJ6t27tznjZ9jg+vXrzZmRI0eaM++//745I0ljxowxZ/Ly8syZxYsXmzOLFi0yZyR/H5Ofz9Ott95qzvgZaJuRkWHOSNKHH35ozqSnp5szOTk55sw3v/lNc+bzE1say88Q002bNpkzU6dONWf8fv+KiIgwZz799FPT+qqqKj355JMqLS39wt/rO38UHADg2kQBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIAT5pdjaCnV1dXq0KHx/Thq1CjzMV588UVzRpI++OADc8bPy1F069bNnNm3b58542eSuCR17drVnOnVq5c589hjj5kzfl/+w8818Y1vfMOc2bFjhzkzadIkc+bUqVPmjCRFR0ebM2+99ZY507dvX3Nm48aN5szQoUPNGcnf13p8fLw542da/rlz58wZSfroo4/MmZKSEtP68+fPN2od94AAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwIlWO4y0vLzcNKBvxYoV5mN885vfNGck6b333jNnhgwZYs7853/+pznz7LPPmjP/8R//Yc5I0ssvv2zOzJgxw5z52te+Zs5UVFSYM5J0/PhxcyYQCJgz999/vzkTERFhzuTn55szkvTLX/7SnFm6dKk5EwwGzZmXXnrJnPE7ePjf//3fzZk33njDnHn33XfNmUGDBpkzkhQbG2vODBgwwLS+srJSv/3tb790HfeAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMCJVjuMtFOnTgoPD2/0+gkTJpiPkZuba85IUvfu3c2Zs2fPmjN/+7d/a874GSxaVFRkzkhSXFycOXPgwAFzxs/Qxddff92ckaSysjJzxs95WL58uTlTV1dnzuzbt8+ckaQNGzaYM3l5eebM1q1bzRk/n6OnnnrKnJGkU6dOmTN+htN26GC/LxATE2POSP6+3mtra03rq6qqGrWOe0AAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4ESrHUYaFRWlqKioRq8vLS01H2PIkCHmjGQfzCf5Gxz4wQcfmDOjR482Z3bv3m3OSNKcOXPMGT9DLpOTk82Zm266yZyRZLrmLlm5cqU5c8stt5gzx48fN2ceeOABc0aSFi1aZM4sXbrUnPHzMd16660tchxJGjhwoDlTUVFhzvTp08ec8Tto1s8w17S0NNP6ysrKRq3jHhAAwAkKCADghLmANm3apOnTpyslJUVhYWFas2ZNg/ffe++9CgsLa3CbOnVqU+0XANBOmAuooqJCGRkZWrZs2VXXTJ06VQUFBfW3VatWfaVNAgDaH/ODEKZNm6Zp06Z94ZpAIKCkpCTfmwIAtH/N8jugnJwcJSQkaNCgQXrooYdUXFx81bXV1dUKhUINbgCA9q/JC2jq1Kn6/e9/rw0bNuipp55Sbm6upk2bdtWHLmdnZysYDNbfUlNTm3pLAIBWqMmfBzR79uz6Pw8fPlwjRoxQv379lJOTo0mTJl22fvHixVq4cGH930OhECUEANeAZn8Ydnp6uuLj45Wfn3/F9wcCAcXExDS4AQDav2YvoGPHjqm4uNjXs9kBAO2X+Udw5eXlDe7NHDx4ULt27VJcXJzi4uL05JNPaubMmUpKStKBAwf0ve99T/3799eUKVOadOMAgLbNXEDbtm1rMMfq0u9v5syZo+eee0579uzRiy++qJKSEqWkpGjy5Mn68Y9/rEAg0HS7BgC0eWGe53muN/FZoVBIwWBQS5cuNQ2GTEhIMB/Lz7BPSRoxYoQ542fgZ1xcnDkTERFhzkRHR5szknT06FFzZsCAAebM//3f/5kzd999tzkjSX/84x/NGT//ufrkk0/MGT8/xg4Gg+aM5O/aO336tDmzbt06c2b48OHmzB133GHOSP4Gzd5www3mzIkTJ8wZP4NzJX+fp8jISNP6qqoq/ehHP1JpaekX/l6fWXAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwoslfkruplJWVqaamptHrCwoKzMfwM0Fbkrp06WLO9OzZ05xZtWqVObNgwQJzpqKiwpyR/E1n9jOBfMKECeZMx44dzRlJ+q//+i9zJj093Zx55ZVXzJnf/e535oyfKeyS9NZbb5kz3/rWt1okU1paas48/PDD5owkLVmyxJx59913zRk/E7T/8R//0Zzxm7NO2T937lyj1nEPCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcaLXDSAOBgAKBQKPX9+/f33yM8vJyc0aS9u3bZ85ERUWZMy+88II5c+jQIXNmx44d5owkFRcXmzNxcXHmzOHDh82Z//3f/zVnJCkrK8uc8XPOn3/+eXOmV69e5szZs2fNGUlatmyZObNnzx5z5siRI+ZM165dzZmnnnrKnJEkz/PMmQsXLpgz69evN2f8fI4k6c033zRnrAN3KysrG7WOe0AAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4ESrHUYqSWFhYY1e62eo4blz58wZSRo9enSLHOsXv/iFOTN79mxzxjpo8JLhw4ebMwUFBebMsGHDzJnw8HBzRpJCoZA5c/78eXNm0KBB5oyfoaf33HOPOSNJjz76qDlz5513mjOnT582Z1JSUswZPwOEJemGG24wZ/wM3B04cKA5U1RUZM5I0je+8Q1zZteuXab1NTU1jVrHPSAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcKLVDiOtq6tTbW1to9d37NjRfAw/GUl67733zJni4mJzZtGiRebMiy++aM7cdNNN5owk7dy505zp3r27OfP++++bM9/5znfMGUlasWKFOTNixAhz5o9//KM5U1paas4kJSWZM5I0Y8YMc8bP56m8vNycOX78uDkzd+5cc0aS1q1bZ840dhDnZ/kZNGsdEHpJz549zZmKigrT+sYO6OUeEADACQoIAOCEqYCys7N1/fXXKzo6WgkJCbrtttuUl5fXYE1VVZWysrLUvXt3de3aVTNnzvT9uhUAgPbLVEC5ubnKysrSli1b9Pbbb6umpkaTJ09u8PPBRx55RH/4wx/02muvKTc3VydOnNAdd9zR5BsHALRtpgchfP4XcitWrFBCQoK2b9+uiRMnqrS0VL/97W+1cuVK/dVf/ZUkafny5RoyZIi2bNni69UFAQDt01f6HdClR+Vcegna7du3q6amRpmZmfVrBg8erLS0NG3evPmK/0Z1dbVCoVCDGwCg/fNdQHV1dVqwYIHGjx+vYcOGSZIKCwsVERGh2NjYBmsTExNVWFh4xX8nOztbwWCw/paamup3SwCANsR3AWVlZWnv3r16+eWXv9IGFi9erNLS0vrb0aNHv9K/BwBoG3w9EXX+/Pl68803tWnTJvXq1av+7UlJSTp//rxKSkoa3AsqKiq66hPiAoGAAoGAn20AANow0z0gz/M0f/58rV69Whs3blTfvn0bvH/UqFEKDw/Xhg0b6t+Wl5enI0eOaNy4cU2zYwBAu2C6B5SVlaWVK1dq7dq1io6Orv+9TjAYVFRUlILBoO6//34tXLhQcXFxiomJ0cMPP6xx48bxCDgAQAOmAnruueckSTfffHODty9fvlz33nuvJGnp0qXq0KGDZs6cqerqak2ZMkW/+c1vmmSzAID2I8zzPM/1Jj4rFAopGAzq3/7t39S5c+dG5z799FPzsfw+4i48PNyc+eSTT8yZ3bt3mzMLFy40Zy79x8IqGAyaM34eZDJ16lRzJjIy0pyRpLCwMHPmgQceMGeWLVtmzqxfv96cWbBggTkjydd/Gh955BFzplu3bubMT3/6U3PGz9efJH372982Z0aPHm3OvPTSS+ZMly5dzBlJuuWWW8yZjRs3mtZXV1dr6dKlKi0tVUxMzFXXMQsOAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATvh6RdSWcObMGVVWVjZ6vZ+JyVu2bDFnJCkxMdGciY+PN2fuvPNOc2bHjh3mTP/+/c0Zyd8k47/5m78xZ0pKSsyZ06dPmzOS9D//8z8tkqmurjZnHnvsMXPmn/7pn8wZSfrud79rzrzwwgvmzJgxY8yZPn36mDODBw82ZyTp0KFD5syZM2fMmbNnz5oz06dPN2ck6dy5c+ZMz549Tesb+72be0AAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4ESrHUaakJCgzp07N3r9e++9Zz5GcXGxOSNJJ06cMGemTJlizuzevduc6d69uzlTVlZmzkhSeHi4OdOpk/2S8zM8MTIy0pyRpKysLHPm1VdfNWdmzZplzuzZs8ecWbJkiTkjSb/85S/NmQceeMCc8TOEs0uXLuZMYWGhOSNJ999/vznz4YcfmjN+Bqxu27bNnJGkHj16mDOW78WSFBYW1qh13AMCADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACfCPM/zXG/is0KhkILBoH784x+bBkqmpaWZj+VnaKAkRUdHmzOBQMCcaanBnTt37jRnJGnw4MHmjJ9hqX6Gxo4cOdKckaRdu3aZMx07djRn/AzUjIqKMmf8DpqdPXu2OfOb3/zGnImIiDBnkpKSzJn8/HxzRpLq6urMGT9f6ykpKeaMXyUlJeZMRUWFaX11dbV+8YtfqLS0VDExMVddxz0gAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHDCPu2yhdTV1ZkGAX7wwQfmY8yaNcuckaQtW7aYM5WVleaMn+GTt99+uzkzatQoc0aSNm/ebM58+umn5syjjz5qzjzzzDPmjCQFg0FzJi8vz5z567/+a3Nm06ZN5swNN9xgzkj+zt/XvvY1c+att94yZ/zMT962bZs5I0mvv/66ObNq1Spzxs/Q2OPHj5szktSnTx9zxvp10djvd9wDAgA4QQEBAJwwFVB2drauv/56RUdHKyEhQbfddttlP364+eabFRYW1uD24IMPNummAQBtn6mAcnNzlZWVpS1btujtt99WTU2NJk+efNmLFc2dO1cFBQX1t6effrpJNw0AaPtMD0JYt25dg7+vWLFCCQkJ2r59uyZOnFj/9s6dO/t61UIAwLXjK/0OqLS0VJIUFxfX4O0vvfSS4uPjNWzYMC1evPgLXya6urpaoVCowQ0A0P75fhh2XV2dFixYoPHjx2vYsGH1b7/77rvVu3dvpaSkaM+ePfr+97+vvLy8qz6cMTs7W08++aTfbQAA2ijfBZSVlaW9e/fq3XffbfD2efPm1f95+PDhSk5O1qRJk3TgwAH169fvsn9n8eLFWrhwYf3fQ6GQUlNT/W4LANBG+Cqg+fPn680339SmTZvUq1evL1w7duxYSVJ+fv4VCygQCCgQCPjZBgCgDTMVkOd5evjhh7V69Wrl5OSob9++X5rZtWuXJCk5OdnXBgEA7ZOpgLKysrRy5UqtXbtW0dHRKiwslHRxTENUVJQOHDiglStX6utf/7q6d++uPXv26JFHHtHEiRM1YsSIZvkAAABtk6mAnnvuOUkXn2z6WcuXL9e9996riIgIrV+/Xs8884wqKiqUmpqqmTNn6gc/+EGTbRgA0D6YfwT3RVJTU5Wbm/uVNgQAuDaEeX5GyzajUCikYDCof/7nfzZNg66pqTEf6+DBg+aMJHXp0sWcGTJkiK9jWXXoYH9q16Xnc1mdOXPGnPHzgJNTp061SEbyNz26a9eu5oyfL7uCggJzxs/nSJLS0tLMmdraWnPGz3koKSkxZyyT9T8rMTHRnPmi5z1ezaFDh8wZv99T/EzZr66uNq2vrKxUVlaWSktLFRMTc9V1DCMFADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACd8vyR3c6uoqDANNxw2bJj5GJGRkeaMJEVHR5szfl6Q75NPPjFn/Ax3vHDhgjkjSd27dzdniouLzZkBAwaYM415scQrGTx4sDljHdQoSSdPnjRnevToYc6kpKSYM5L0/vvvmzO33HKLOVNeXm7OdOzY0Zy57rrrzBlJWrt2rTnjZ4DpXXfdZc74GWAq+bterd8rGzv8lXtAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADAiVY3C+7SLLOqqipTrqKiwnysyspKc0byN4uqpfbnZxZcTU2NOSP5myFn/bxK/s6D3/l2fj5PfmZrnTt3zpzx8zGFh4ebM1LLfUx+Mn6uBz8z5yR/56Gl9ufn3En+PqawsDDT+kvn4Mu+H4V5fr5jNaNjx44pNTXV9TYAAF/R0aNH1atXr6u+v9UVUF1dnU6cOKHo6OjLWjcUCik1NVVHjx5VTEyMox26x3m4iPNwEefhIs7DRa3hPHiep7KyMqWkpKhDh6v/pqfV/QiuQ4cOX9iYkhQTE3NNX2CXcB4u4jxcxHm4iPNwkevzEAwGv3QND0IAADhBAQEAnGhTBRQIBLRkyRIFAgHXW3GK83AR5+EizsNFnIeL2tJ5aHUPQgAAXBva1D0gAED7QQEBAJyggAAATlBAAAAn2kwBLVu2TH369FFkZKTGjh2r999/3/WWWtwTTzyhsLCwBrfBgwe73laz27Rpk6ZPn66UlBSFhYVpzZo1Dd7veZ4ef/xxJScnKyoqSpmZmdq/f7+bzTajLzsP995772XXx9SpU91stplkZ2fr+uuvV3R0tBISEnTbbbcpLy+vwZqqqiplZWWpe/fu6tq1q2bOnKmioiJHO24ejTkPN99882XXw4MPPuhox1fWJgrolVde0cKFC7VkyRLt2LFDGRkZmjJlik6ePOl6ay1u6NChKigoqL+9++67rrfU7CoqKpSRkaFly5Zd8f1PP/20nn32WT3//PPaunWrunTpoilTpvgafNqafdl5kKSpU6c2uD5WrVrVgjtsfrm5ucrKytKWLVv09ttvq6amRpMnT24wRPaRRx7RH/7wB7322mvKzc3ViRMndMcddzjcddNrzHmQpLlz5za4Hp5++mlHO74Krw0YM2aMl5WVVf/32tpaLyUlxcvOzna4q5a3ZMkSLyMjw/U2nJLkrV69uv7vdXV1XlJSkvfzn/+8/m0lJSVeIBDwVq1a5WCHLePz58HzPG/OnDnejBkznOzHlZMnT3qSvNzcXM/zLn7uw8PDvddee61+zccff+xJ8jZv3uxqm83u8+fB8zzvpptu8r773e+621QjtPp7QOfPn9f27duVmZlZ/7YOHTooMzNTmzdvdrgzN/bv36+UlBSlp6frnnvu0ZEjR1xvyamDBw+qsLCwwfURDAY1duzYa/L6yMnJUUJCggYNGqSHHnpIxcXFrrfUrEpLSyVJcXFxkqTt27erpqamwfUwePBgpaWltevr4fPn4ZKXXnpJ8fHxGjZsmBYvXuz7JRyaS6sbRvp5p0+fVm1trRITExu8PTExUfv27XO0KzfGjh2rFStWaNCgQSooKNCTTz6pG2+8UXv37lV0dLTr7TlRWFgoSVe8Pi6971oxdepU3XHHHerbt68OHDigxx57TNOmTdPmzZt9vYZVa1dXV6cFCxZo/PjxGjZsmKSL10NERIRiY2MbrG3P18OVzoMk3X333erdu7dSUlK0Z88eff/731deXp5ef/11h7ttqNUXEP5s2rRp9X8eMWKExo4dq969e+vVV1/V/fff73BnaA1mz55d/+fhw4drxIgR6tevn3JycjRp0iSHO2seWVlZ2rt37zXxe9AvcrXzMG/evPo/Dx8+XMnJyZo0aZIOHDigfv36tfQ2r6jV/wguPj5eHTt2vOxRLEVFRUpKSnK0q9YhNjZWAwcOVH5+vuutOHPpGuD6uFx6erri4+Pb5fUxf/58vfnmm3rnnXcavHxLUlKSzp8/r5KSkgbr2+v1cLXzcCVjx46VpFZ1PbT6AoqIiNCoUaO0YcOG+rfV1dVpw4YNGjdunMOduVdeXq4DBw4oOTnZ9Vac6du3r5KSkhpcH6FQSFu3br3mr49jx46puLi4XV0fnudp/vz5Wr16tTZu3Ki+ffs2eP+oUaMUHh7e4HrIy8vTkSNH2tX18GXn4Up27dolSa3renD9KIjGePnll71AIOCtWLHC++ijj7x58+Z5sbGxXmFhoeuttah/+Id/8HJycryDBw96f/rTn7zMzEwvPj7eO3nypOutNauysjJv586d3s6dOz1J3q9+9Stv586d3uHDhz3P87yf/exnXmxsrLd27Vpvz5493owZM7y+fft6lZWVjnfetL7oPJSVlXmLFi3yNm/e7B08eNBbv369N3LkSG/AgAFeVVWV6603mYceesgLBoNeTk6OV1BQUH87d+5c/ZoHH3zQS0tL8zZu3Oht27bNGzdunDdu3DiHu256X3Ye8vPzvR/96Efetm3bvIMHD3pr16710tPTvYkTJzreeUNtooA8z/N+/etfe2lpaV5ERIQ3ZswYb8uWLa631OJmzZrlJScnexEREV7Pnj29WbNmefn5+a631ezeeecdT9Jltzlz5nied/Gh2D/84Q+9xMRELxAIeJMmTfLy8vLcbroZfNF5OHfunDd58mSvR48eXnh4uNe7d29v7ty57e4/aVf6+CV5y5cvr19TWVnp/f3f/73XrVs3r3Pnzt7tt9/uFRQUuNt0M/iy83DkyBFv4sSJXlxcnBcIBLz+/ft7jz76qFdaWup245/DyzEAAJxo9b8DAgC0TxQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABw4v8BmnJVEauVOdUAAAAASUVORK5CYII=\n"},"metadata":{}}],"source":["# create generator model\n","generator = make_generator_model()\n","\n","# create noise and throw noise into the generator\n","noise = tf.random.normal([1, 784])\n","# noise = tf.random.normal([1, 28, 28, 1])\n","generated_image = generator(noise, training=False)\n","print(generated_image.shape)\n","\n","# plot an output from generator that is untrained\n","plt.imshow(generated_image[0, :, :, 0], cmap='gray')"]},{"cell_type":"markdown","metadata":{"id":"D0IKnaCtg6WE"},"source":["### The Discriminator\n","\n","The discriminator is a CNN-based image classifier. Notice that in the beginning of the `make_discriminator_model()` function below the input_shape must be the same as the output shape of the generator."]},{"cell_type":"code","execution_count":15,"metadata":{"executionInfo":{"elapsed":4,"status":"ok","timestamp":1711680297912,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"dw2tPLmk2pEP"},"outputs":[],"source":["# define discriminator model\n","def make_discriminator_model():\n","\n","    # use sequential API\n","    model = tf.keras.Sequential()\n","\n","    # add Conv2D layer\n","    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same', input_shape=[28, 28, 1]))\n","    model.add(layers.LeakyReLU())\n","    model.add(layers.Dropout(0.3))\n","\n","    # add Conv2D layer\n","    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))\n","    model.add(layers.LeakyReLU())\n","    model.add(layers.Dropout(0.3))\n","\n","    # flatten and output 1 neuron\n","    model.add(layers.Flatten())\n","    model.add(layers.Dense(1))\n","\n","    return model"]},{"cell_type":"markdown","metadata":{"id":"QhPneagzCaQv"},"source":["Use the (as yet untrained) discriminator to classify the generated images as real or fake. The model will be trained to output positive values for real images, and negative values for fake images."]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":343,"status":"ok","timestamp":1711680298251,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"gDkA05NE6QMs","outputId":"659e9d02-acc4-46af-e6f2-e2e53c940233"},"outputs":[{"output_type":"stream","name":"stdout","text":["tf.Tensor([[-0.0109086]], shape=(1, 1), dtype=float32)\n"]}],"source":["# define discriminator\n","discriminator = make_discriminator_model()\n","decision = discriminator(generated_image)\n","print(decision)"]},{"cell_type":"markdown","metadata":{"id":"OuPl5e3JCOEf"},"source":["## How does GAN learn?\n","\n","According to the [original paper](https://arxiv.org/pdf/1406.2661.pdf), we can set up the situation as the following. Suppose the discriminator is $D(x; \\theta_d)$ using noisy data $x \\sim p_z(z)$ where $p_z$ is some prior distribution and the generator is $G(z; \\theta_g)$. Then the goal is to learn to train $D$ to maximize the probability of assigning the correct label to both training examples and samples from $G$. We simultaneously train $G$ to minimize $\\log(1 - D(G(z)))$.\n","\n","In other words, $D$ and $G$ play the following two-layer minimax game with the objective function $\\mathcal{V}(G, D)$:\n","$$ \\min_G \\max_D \\mathcal{V}(D, G) = \\mathbb{E}_{x \\sim p_{\\text{data}(x)}} [\\log D(x)] + \\mathbb{E}_{z \\sim p_z(z)} [\\log (1 - D(G(z)))]$$\n","\n","In each learning step, the gradient for the discriminator is updated:\n","$$\\nabla_{\\theta_d} \\frac{1}{m} \\sum_{i=1}^m [\\log D(x^{(i)}) + \\log(1 - D(G(z^{(i)})))]$$\n","where $m$ refers to samples and the gradient is taken w.r.t $\\theta_d$ (parameters for the discriminator $D$).\n","\n","The gradient for the generator is updated:\n","$$\\nabla_{\\theta_g} \\frac{1}{m} \\sum_{i=1}^m \\log(1 - D(G(z^{(i)})))$$\n","where $m$ refers to the samples and the gradient is taken w.r.t. $\\theta_g$ (parameters for the generator $G$)."]},{"cell_type":"markdown","metadata":{"id":"0FMYgY_mPfTi"},"source":["## Define the loss and optimizers\n","\n","Define loss functions and optimizers for both models.\n"]},{"cell_type":"code","execution_count":17,"metadata":{"executionInfo":{"elapsed":303,"status":"ok","timestamp":1711680310566,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"psQfmXxYKU3X"},"outputs":[],"source":["# This method returns a helper function to compute cross entropy loss\n","cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)"]},{"cell_type":"markdown","metadata":{"id":"Jd-3GCUEiKtv"},"source":["### Generator loss\n","The generator's loss quantifies how well it was able to trick the discriminator. Intuitively, if the generator is performing well, the discriminator will classify the fake images as real (or 1). Here, compare the discriminators decisions on the generated images to an array of 1s."]},{"cell_type":"code","execution_count":18,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711680310951,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"90BIcCKcDMxz"},"outputs":[],"source":["# def loss for generator\n","def generator_loss(fake_output):\n","    return cross_entropy(tf.ones_like(fake_output), fake_output)"]},{"cell_type":"markdown","metadata":{"id":"PKY_iPSPNWoj"},"source":["### Discriminator loss\n","\n","This method quantifies how well the discriminator is able to distinguish real images from fakes. It compares the discriminator's predictions on real images to an array of 1s, and the discriminator's predictions on fake (generated) images to an array of 0s."]},{"cell_type":"code","execution_count":19,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711680312222,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"dO3db2Ut5Hzu","outputId":"ecec9235-cd9c-4469-c1d6-eb117d4fa5a9"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(<tf.Tensor: shape=(3,), dtype=int32, numpy=array([1, 1, 1], dtype=int32)>,\n"," <tf.Tensor: shape=(3,), dtype=int32, numpy=array([0, 0, 0], dtype=int32)>)"]},"metadata":{},"execution_count":19}],"source":["# check\n","tf.ones_like([1,2,3]), tf.zeros_like([1,2,3])"]},{"cell_type":"code","execution_count":20,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711680313519,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"wkMNfBWlT-PV"},"outputs":[],"source":["# def loss for discriminator\n","def discriminator_loss(real_output, fake_output):\n","    real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n","    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)\n","    total_loss = real_loss + fake_loss\n","    return total_loss"]},{"cell_type":"markdown","metadata":{"id":"MgIc7i0th_Iu"},"source":["## Optimizers\n","\n","The discriminator and the generator optimizers are different since you will train two networks separately."]},{"cell_type":"code","execution_count":21,"metadata":{"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711680314637,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"iWCn_PVdEJZ7"},"outputs":[],"source":["# define two separate optimizer for the generator and the discriminator\n","generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n","discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"]},{"cell_type":"markdown","metadata":{"id":"mWtinsGDPJlV"},"source":["### Save checkpoints\n","This notebook also demonstrates how to save and restore models, which can be helpful in case a long running training task is interrupted."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"CA1w-7s2POEy"},"outputs":[],"source":["# save and checkpoint\n","checkpoint_dir = './training_checkpoints'\n","checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n","checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n","                                 discriminator_optimizer=discriminator_optimizer,\n","                                 generator=generator,\n","                                 discriminator=discriminator)"]},{"cell_type":"markdown","metadata":{"id":"Rw1fkAczTQYh"},"source":["## Define the training loop\n"]},{"cell_type":"code","execution_count":22,"metadata":{"executionInfo":{"elapsed":309,"status":"ok","timestamp":1711680320885,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"NS2GWywBbAWo"},"outputs":[],"source":["# args\n","EPOCHS = 3\n","noise_dim = 100\n","num_examples_to_generate = 25\n","\n","# You will reuse this seed overtime (so it's easier)\n","# to visualize progress in the animated GIF)\n","seed = tf.random.normal([num_examples_to_generate, noise_dim])"]},{"cell_type":"code","execution_count":23,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711680321296,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"s3DjF6pn1Iok","outputId":"2d29b52a-0c85-4347-c814-f57d272f9e49"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(TensorShape([25, 100]), tensorflow.python.framework.ops.EagerTensor)"]},"metadata":{},"execution_count":23}],"source":["# check\n","seed.shape, type(seed)"]},{"cell_type":"markdown","metadata":{"id":"jylSonrqSWfi"},"source":["The training loop begins with generator receiving a random seed as input. That seed is used to produce an image. The discriminator is then used to classify real images (drawn from the training set) and fakes images (produced by the generator). The loss is calculated for each of these models, and the gradients are used to update the generator and discriminator."]},{"cell_type":"code","execution_count":41,"metadata":{"executionInfo":{"elapsed":317,"status":"ok","timestamp":1711682309022,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"3t5ibNo05jCB"},"outputs":[],"source":["# Notice the use of `tf.function`\n","# This annotation causes the function to be \"compiled\".\n","@tf.function\n","def train_step(images, twoDim=False, print_loss=False):\n","    # Flatten and mask the images instead of using random noise\n","    # Example mask: setting a random portion of the image to zero\n","    # This is a simple mask that zeroes out half of the image\n","    delta = 0\n","    # mask = tf.concat([tf.ones(shape=(784//2-delta,)), tf.zeros(shape=(784//2+delta,))], 0)\n","    # mask = tf.random.shuffle(mask)\n","    mask = np.ones((28, 28), dtype=np.float32)\n","    # Set the bottom right portion of the mask to 1\n","    # mask[7:21, 7:21] = 0\n","    # Convert the numpy array to a TensorFlow tensor\n","    mask = mask.reshape(784, )\n","    mask = tf.convert_to_tensor(mask)\n","    flat_images = tf.reshape(images, [BATCH_SIZE, 784])\n","    masked_images = flat_images * mask\n","    if twoDim:\n","        masked_images = tf.reshape(masked_images, [BATCH_SIZE, 28, 28, 1])\n","\n","    # args\n","    # noise = tf.random.normal([BATCH_SIZE, noise_dim])\n","\n","    # use gradient tape\n","    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n","      generated_images = generator(masked_images, training=True)\n","\n","      real_output = discriminator(images, training=True)\n","      fake_output = discriminator(generated_images, training=True)\n","\n","      gen_loss = generator_loss(fake_output)\n","      disc_loss = discriminator_loss(real_output, fake_output)\n","\n","    # compute gradients\n","    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n","    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n","\n","    # apply gradients\n","    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n","    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n","\n","    # Calculate and print the total loss if `print_loss` is True\n","    total_loss = gen_loss + disc_loss\n","    if print_loss:\n","        tf.print(\"Total Loss:\", total_loss)\n"]},{"cell_type":"markdown","metadata":{"id":"xw0A4Sw36pxl"},"source":["Notice that the `train()` function saves images in the current directory."]},{"cell_type":"code","execution_count":42,"metadata":{"executionInfo":{"elapsed":1,"status":"ok","timestamp":1711682309394,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"GnDRZsPDiKR9"},"outputs":[],"source":["from tqdm import tqdm"]},{"cell_type":"code","execution_count":43,"metadata":{"executionInfo":{"elapsed":1,"status":"ok","timestamp":1711682309693,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"2M7LmLtGEMQJ"},"outputs":[],"source":["# training\n","def train(dataset, epochs):\n","\n","    # for loop: this loop iterates epochs\n","    i=0\n","    for epoch in range(epochs):\n","        start = time.time()\n","\n","        # for loop: iterate images in the dataset\n","        for image_batch in tqdm(dataset):\n","            if i % 100 == 0:\n","                print_loss = True  # Set print_loss flag to True on even epochs\n","            else:\n","                print_loss = False  # Set print_loss flag to False on odd epochs\n","            train_step(image_batch, print_loss=print_loss)  # Perform a training step with the current batch\n","            i += 1\n","\n","        # produce images for the GIF as you go\n","        # display.clear_output(wait=True)\n","        # generate_and_save_images(\n","        #     generator,\n","        #     epoch + 1,\n","        #     seed)\n","\n","        # save the model every 15 epochs\n","        # if (epoch + 1) % 15 == 0:\n","        #     checkpoint.save(file_prefix = checkpoint_prefix)\n","\n","        # print statement\n","        print ('Time for epoch {} is {} sec'.format(epoch + 1, time.time()-start))\n","        print('...')\n","        print('...')\n","        print('...')\n","        print('...')\n","\n","        # generate after the final epoch\n","        # display.clear_output(wait=True)\n","        # generate_and_save_images(generator, epochs, seed)"]},{"cell_type":"markdown","metadata":{"id":"2aFF7Hk3XdeW"},"source":["**Generate and save images**\n"]},{"cell_type":"code","execution_count":44,"metadata":{"id":"RmdVsmvhPxyy","executionInfo":{"status":"ok","timestamp":1711682313722,"user_tz":240,"elapsed":315,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# # def function\n","# def generate_and_save_images(model, epoch, test_input):\n","#     # Notice `training` is set to False.\n","#     # This is so all layers run in inference mode (batchnorm).\n","#     predictions = model(test_input, training=False)\n","\n","#     # figure\n","#     fig = plt.figure(figsize=(5, 5))\n","#     for i in range(predictions.shape[0]):\n","#         plt.subplot(5, 5, i+1)\n","#         if i == 0:\n","#             plt.title('ep='+str(epoch))\n","#         plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n","#         plt.axis('off')\n","\n","#     # save and display\n","#     plt.savefig('image_at_epoch_{:04d}.png'.format(epoch))\n","#     plt.show()"]},{"cell_type":"markdown","metadata":{"id":"dZrd4CdjR-Fp"},"source":["## Train the model\n","\n","Call the `train()` method defined above to train the generator and discriminator simultaneously. Note, training GANs can be tricky. It's important that the generator and discriminator do not overpower each other (e.g., that they train at a similar rate).\n","\n","At the beginning of the training, the generated images look like random noise. As training progresses, the generated digits will look increasingly real. After about 50 epochs, they resemble MNIST digits. This may take about one minute / epoch with the default settings on Colab."]},{"cell_type":"code","execution_count":45,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":35},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1711682314937,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"cyOf3tTrgiRK","outputId":"d16cda8f-5ce6-4819-9095-5c32d4afe66e"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'/device:GPU:0'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":45}],"source":["tf.test.gpu_device_name()"]},{"cell_type":"code","execution_count":46,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":10566,"status":"ok","timestamp":1711682330185,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"Ly3UN0SLLY2l","outputId":"bda8a578-1031-42fc-fa00-248c0f3eab1f"},"outputs":[{"output_type":"stream","name":"stderr","text":["\r  0%|          | 0/1875 [00:00<?, ?it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.06047249\n"]},{"output_type":"stream","name":"stderr","text":["  5%|▍         | 91/1875 [00:02<00:18, 96.01it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.05178857\n"]},{"output_type":"stream","name":"stderr","text":[" 10%|▉         | 179/1875 [00:02<00:09, 172.72it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.15181947\n"]},{"output_type":"stream","name":"stderr","text":[" 16%|█▌        | 292/1875 [00:02<00:07, 212.48it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.06444216\n"]},{"output_type":"stream","name":"stderr","text":[" 20%|██        | 382/1875 [00:03<00:06, 214.13it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.11599946\n"]},{"output_type":"stream","name":"stderr","text":[" 26%|██▋       | 495/1875 [00:03<00:06, 219.79it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.20078373\n"]},{"output_type":"stream","name":"stderr","text":[" 31%|███       | 584/1875 [00:04<00:05, 216.79it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 1.97248387\n"]},{"output_type":"stream","name":"stderr","text":[" 37%|███▋      | 697/1875 [00:04<00:05, 214.76it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.12639523\n"]},{"output_type":"stream","name":"stderr","text":[" 42%|████▏     | 787/1875 [00:05<00:05, 214.25it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.13975\n"]},{"output_type":"stream","name":"stderr","text":[" 48%|████▊     | 899/1875 [00:05<00:04, 219.43it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.12427568\n"]},{"output_type":"stream","name":"stderr","text":[" 53%|█████▎    | 988/1875 [00:06<00:04, 217.05it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.17949843\n"]},{"output_type":"stream","name":"stderr","text":[" 57%|█████▋    | 1078/1875 [00:06<00:03, 218.84it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 1.9999876\n"]},{"output_type":"stream","name":"stderr","text":[" 64%|██████▎   | 1191/1875 [00:07<00:03, 217.77it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.16\n"]},{"output_type":"stream","name":"stderr","text":[" 68%|██████▊   | 1280/1875 [00:07<00:02, 218.56it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.01880741\n"]},{"output_type":"stream","name":"stderr","text":[" 74%|███████▍  | 1391/1875 [00:08<00:02, 214.91it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.12462473\n"]},{"output_type":"stream","name":"stderr","text":[" 79%|███████▉  | 1479/1875 [00:08<00:01, 209.59it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.13059521\n"]},{"output_type":"stream","name":"stderr","text":[" 85%|████████▍ | 1593/1875 [00:09<00:01, 216.86it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.23656344\n"]},{"output_type":"stream","name":"stderr","text":[" 90%|████████▉ | 1682/1875 [00:09<00:00, 217.16it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.18164897\n"]},{"output_type":"stream","name":"stderr","text":[" 96%|█████████▌| 1792/1875 [00:09<00:00, 214.82it/s]"]},{"output_type":"stream","name":"stdout","text":["Total Loss: 2.05040503\n"]},{"output_type":"stream","name":"stderr","text":["100%|██████████| 1875/1875 [00:10<00:00, 181.40it/s]"]},{"output_type":"stream","name":"stdout","text":["Time for epoch 1 is 10.341681241989136 sec\n","...\n","...\n","...\n","...\n","CPU times: user 11.2 s, sys: 409 ms, total: 11.6 s\n","Wall time: 10.3 s\n"]},{"output_type":"stream","name":"stderr","text":["\n"]}],"source":["%%time\n","\n","# train\n","with tf.device('/device:GPU:0'):\n","    train(train_dataset, 1)"]},{"cell_type":"markdown","metadata":{"id":"rfM4YcPVPkNO"},"source":["Restore the latest checkpoint."]},{"cell_type":"markdown","metadata":{"id":"GbrG8yoItE6M"},"source":["## Inference"]},{"cell_type":"code","execution_count":47,"metadata":{"executionInfo":{"elapsed":310,"status":"ok","timestamp":1711682338439,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"X_ScZdCwtFwT"},"outputs":[],"source":["def make_inference(input_image, generator, twoDim=False):\n","    # Ensure input_image is a tensor\n","    if not isinstance(input_image, tf.Tensor):\n","        input_image = tf.convert_to_tensor(input_image, dtype=tf.float32)\n","\n","    # Flatten the input image and apply the mask\n","    delta = 0\n","    # mask = tf.concat([tf.ones(shape=(784//2-delta,)), tf.zeros(shape=(784//2+delta,))], 0)\n","    # Create a binary mask of size 28x28\n","    mask = np.ones((28, 28), dtype=np.float32)\n","    # Set the bottom right portion of the mask to 1\n","    # mask[7:21, 7:21] = 0\n","    # Convert the numpy array to a TensorFlow tensor\n","    mask = mask.reshape(784, )\n","    mask = tf.convert_to_tensor(mask)\n","    # print(mask.shape)\n","\n","    # mask = tf.random.shuffle(mask)\n","    flat_image = tf.reshape(input_image, [784])\n","    masked_image = flat_image * mask\n","    if twoDim:\n","        masked_image = tf.reshape(masked_image, [28, 28, 1])\n","\n","    # Since the generator expects a batch dimension, add a batch dimension to the masked image\n","    masked_image = tf.expand_dims(masked_image, 0)  # Adds a batch dimension\n","\n","    # Generate the image using the generator\n","    # masked_image_for_gen = masked_image[:, 0:392]\n","    generated_image = generator(masked_image, training=False)  # Set training to False for inference\n","\n","    return masked_image, generated_image\n"]},{"cell_type":"code","execution_count":48,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711682338739,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"NmcI5Xi4y6t2","outputId":"b9e0bfed-f76f-4321-f0e4-67fc6319f06b"},"outputs":[{"output_type":"stream","name":"stdout","text":["(1, 784) (1, 28, 28, 1)\n"]}],"source":["masked_output_, output_ = make_inference(train_images[0], generator)\n","\n","print(masked_output_.shape, output_.shape)"]},{"cell_type":"markdown","metadata":{"id":"RR7taPh2G2ZV"},"source":["## Loop: Train + Inference"]},{"cell_type":"code","execution_count":49,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000,"output_embedded_package_id":"1WpUzTc2evMNvXP5QFQuQxjj5iuCJEEts"},"executionInfo":{"elapsed":1528281,"status":"ok","timestamp":1711683867949,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"RCmzlLQoLQ3x","outputId":"2ae925b0-94f4-470d-d210-484f8508d9a6"},"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}],"source":["%%time\n","\n","for k in range(50):\n","\n","    # train\n","    with tf.device('/device:GPU:0'):\n","        train(train_dataset, EPOCHS)\n","\n","    # Assuming 'train_images' and 'train_labels' are defined and 'generator' is your model\n","    plt.figure(figsize=(20, 4))  # Adjusted size for better visibility\n","    T = 20\n","\n","    # Display real images\n","    for i in range(T):\n","        plt.subplot(3, T, i + 1)\n","        plt.imshow(train_images[i], cmap='gray')\n","        plt.title(f\"Real: {train_labels[i]}\")\n","        plt.axis('off')\n","\n","    # Display fake images generated by the model\n","    for i in range(T):\n","        masked_output_, output_ = make_inference(train_images[i], generator)\n","        masked_output_ = tf.reshape(masked_output_, [28, 28])\n","        plt.subplot(3, T, i + T + 1)  # +7 to move to the second row, right below the corresponding real image\n","        plt.imshow(masked_output_, cmap='gray')  # Adjust indexing if needed\n","        plt.title(f\"Masked: {train_labels[i]}\")\n","        plt.axis('off')\n","\n","    # Display fake images generated by the model\n","    for i in range(T):\n","        masked_output_, output_ = make_inference(train_images[i], generator)\n","        plt.subplot(3, T, i + T + T + 1)  # +7 to move to the second row, right below the corresponding real image\n","        plt.imshow(output_[0, :, :, 0], cmap='gray')  # Adjust indexing if needed\n","        plt.title(f\"Gen.: {train_labels[i]}\")\n","        plt.axis('off')\n","\n","    plt.tight_layout()\n","    plt.show()\n","\n","    # Chk point\n","    print(\"===============================!\")\n","    print(f\"Checkpoint {k+1}\")\n"]},{"cell_type":"markdown","metadata":{"id":"k6qC-SbjK0yW"},"source":["## Next steps\n"]},{"cell_type":"markdown","metadata":{"id":"xjjkT9KAK6H7"},"source":["This tutorial has shown the complete code necessary to write and train a GAN. It is an adaption from this [Tensorflow notebook](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/generative/dcgan.ipynb). As a next step, you might like to experiment with a different dataset, for example the Large-scale Celeb Faces Attributes (CelebA) dataset [available on Kaggle](https://www.kaggle.com/jessicali9530/celeba-dataset). To learn more about GANs see the [NIPS 2016 Tutorial: Generative Adversarial Networks](https://arxiv.org/abs/1701.00160).\n"]}],"metadata":{"accelerator":"GPU","colab":{"gpuType":"A100","machine_shape":"hm","provenance":[{"file_id":"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/dcgan.ipynb","timestamp":1657820300439}],"toc_visible":true},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}

docs/notebooks/ex10b - dcgan on masked mnist.ipynb

@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","metadata":{"id":"rF2x3qooyBTI"},"source":["# Deep Convolutional Generative Adversarial Network"]},{"cell_type":"markdown","metadata":{"id":"ITZuApL56Mny"},"source":["This tutorial demonstrates how to generate images of handwritten digits using a [Deep Convolutional Generative Adversarial Network](https://arxiv.org/pdf/1511.06434.pdf) (DCGAN). The code is written using the [Keras Sequential API](https://www.tensorflow.org/guide/keras) with a `tf.GradientTape` training loop."]},{"cell_type":"markdown","metadata":{"id":"2MbKJY38Puy9"},"source":["## What are GANs?\n","[Generative Adversarial Networks](https://arxiv.org/abs/1406.2661) (GANs) are one of the most interesting ideas in computer science today. Two models are trained simultaneously by an adversarial process. A *generator* (\"the artist\") learns to create images that look real, while a *discriminator* (\"the art critic\") learns to tell real images apart from fakes.\n","\n","<p align='center'><img src='https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/images/gan1.png?raw=1'></img></p>\n","\n","During training, the *generator* progressively becomes better at creating images that look real, while the *discriminator* becomes better at telling them apart. The process reaches equilibrium when the *discriminator* can no longer distinguish real images from fakes.\n","\n","<p align='center'><img src='https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/images/gan2.png?raw=1'></img></p>\n","\n","This notebook demonstrates this process on the MNIST dataset. The following animation shows a series of images produced by the *generator* as it was trained for 50 epochs. The images begin as random noise, and increasingly resemble hand written digits over time.\n","\n","<p align='center'><img src='https://tensorflow.org/images/gan/dcgan.gif'></img></p>\n","\n","To learn more about GANs, see MIT's [Intro to Deep Learning](http://introtodeeplearning.com/) course."]},{"cell_type":"markdown","metadata":{"id":"e1_Y75QXJS6h"},"source":["### Setup"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"WZKbyU2-AiY-","executionInfo":{"status":"ok","timestamp":1711638236825,"user_tz":240,"elapsed":3299,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# import\n","import tensorflow as tf"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"executionInfo":{"elapsed":6,"status":"ok","timestamp":1711638236825,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"wx-zNbLqB4K8","outputId":"a0b7549a-ee98-41ea-ddea-928745f8190b"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["'2.15.0'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":2}],"source":["tf.__version__"]},{"cell_type":"code","execution_count":3,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":11931,"status":"ok","timestamp":1711638248752,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"YzTlj4YdCip_","outputId":"ade030d6-1e76-4628-f0e8-639e626e5f48"},"outputs":[{"output_type":"stream","name":"stdout","text":["Collecting git+https://github.com/tensorflow/docs\n","  Cloning https://github.com/tensorflow/docs to /tmp/pip-req-build-_oq5_msu\n","  Running command git clone --filter=blob:none --quiet https://github.com/tensorflow/docs /tmp/pip-req-build-_oq5_msu\n","  Resolved https://github.com/tensorflow/docs to commit ff989f0d94cd81cce45a8db0f540e605ce05512b\n","  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n","Collecting astor (from tensorflow-docs==2024.3.27.3713)\n","  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)\n","Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (1.4.0)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (3.1.3)\n","Requirement already satisfied: nbformat in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (5.10.3)\n","Requirement already satisfied: protobuf>=3.12 in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (3.20.3)\n","Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from tensorflow-docs==2024.3.27.3713) (6.0.1)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->tensorflow-docs==2024.3.27.3713) (2.1.5)\n","Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (2.19.1)\n","Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (4.19.2)\n","Requirement already satisfied: jupyter-core in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (5.7.2)\n","Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.10/dist-packages (from nbformat->tensorflow-docs==2024.3.27.3713) (5.7.1)\n","Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (23.2.0)\n","Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (2023.12.1)\n","Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (0.34.0)\n","Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==2024.3.27.3713) (0.18.0)\n","Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core->nbformat->tensorflow-docs==2024.3.27.3713) (4.2.0)\n","Building wheels for collected packages: tensorflow-docs\n","  Building wheel for tensorflow-docs (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for tensorflow-docs: filename=tensorflow_docs-2024.3.27.3713-py3-none-any.whl size=182447 sha256=ec80b9baf7e738bf3183b24581f7e69c486f686644a1b3a7fec7cc6c638fd3a6\n","  Stored in directory: /tmp/pip-ephem-wheel-cache-v4nlsdyw/wheels/86/0f/1e/3b62293c8ffd0fd5a49508e6871cdb7554abe9c62afd35ec53\n","Successfully built tensorflow-docs\n","Installing collected packages: astor, tensorflow-docs\n","Successfully installed astor-0.8.1 tensorflow-docs-2024.3.27.3713\n"]}],"source":["# To generate GIFs\n","# !pip install imageio\n","!pip install git+https://github.com/tensorflow/docs"]},{"cell_type":"code","execution_count":4,"metadata":{"id":"YfIk2es3hJEd","executionInfo":{"status":"ok","timestamp":1711638249543,"user_tz":240,"elapsed":793,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# import\n","import glob\n","import imageio\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import os\n","import PIL\n","from tensorflow.keras import layers\n","import time\n","\n","from IPython import display"]},{"cell_type":"markdown","metadata":{"id":"iYn4MdZnKCey"},"source":["### Load and prepare the dataset\n","\n","You will use the MNIST dataset to train the generator and the discriminator. The generator will generate handwritten digits resembling the MNIST data."]},{"cell_type":"code","execution_count":5,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1070,"status":"ok","timestamp":1711638250611,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"a4fYMGxGhrna","outputId":"3727b5c2-f1c0-4b6c-ac59-700f80a67b51"},"outputs":[{"output_type":"stream","name":"stdout","text":["Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n","11490434/11490434 [==============================] - 1s 0us/step\n"]}],"source":["# get data\n","(train_images, train_labels), (_, _) = tf.keras.datasets.mnist.load_data()"]},{"cell_type":"code","execution_count":6,"metadata":{"id":"NFC2ghIdiZYE","executionInfo":{"status":"ok","timestamp":1711638250611,"user_tz":240,"elapsed":5,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["#rescale\n","train_images = train_images.reshape(train_images.shape[0], 28, 28, 1).astype('float32')\n","train_images = (train_images - 127.5) / 127.5  # Normalize the images to [-1, 1]"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5,"status":"ok","timestamp":1711638250611,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"CsIROTs1zWSG","outputId":"a73c589c-80b4-4b94-9e82-0e8ceca17b1d"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["((60000, 28, 28, 1), numpy.ndarray)"]},"metadata":{},"execution_count":7}],"source":["# display shape\n","train_images.shape, type(train_images)"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":4,"status":"ok","timestamp":1711638250611,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"88h4t6pT2ldg","outputId":"2c6e2264-c648-444c-d635-a89f68e7ca77"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["((60000,), numpy.ndarray, array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8))"]},"metadata":{},"execution_count":8}],"source":["# labels\n","train_labels.shape, type(train_labels), np.unique(train_labels)"]},{"cell_type":"markdown","metadata":{"id":"g5PjXIvVzbPc"},"source":["This is the information you need to verify if you use your own dataset."]},{"cell_type":"code","execution_count":9,"metadata":{"id":"S4PIDhoDLbsZ","executionInfo":{"status":"ok","timestamp":1711638250611,"user_tz":240,"elapsed":3,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# args\n","BUFFER_SIZE = 60000\n","BATCH_SIZE = 32"]},{"cell_type":"code","source":["60000 / 32"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2tFY7qMfiA6z","executionInfo":{"status":"ok","timestamp":1711638250611,"user_tz":240,"elapsed":3,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"a701e0d1-e27c-49cf-90f4-58dda31ba440"},"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["1875.0"]},"metadata":{},"execution_count":10}]},{"cell_type":"code","execution_count":11,"metadata":{"id":"-yKCCQOoJ7cn","executionInfo":{"status":"ok","timestamp":1711638251426,"user_tz":240,"elapsed":817,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# Batch and shuffle the data\n","train_dataset = tf.data.Dataset.from_tensor_slices(train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"]},{"cell_type":"code","execution_count":12,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711638251426,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"0RtaLSObzi94","outputId":"4fef0146-e55a-48aa-c174-bc4df53a016d"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(<_BatchDataset element_spec=TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32, name=None)>,\n"," tensorflow.python.data.ops.batch_op._BatchDataset)"]},"metadata":{},"execution_count":12}],"source":["# train_dataset\n","train_dataset, type(train_dataset)"]},{"cell_type":"markdown","metadata":{"id":"THY-sZMiQ4UV"},"source":["## Create the models\n","\n","Both the generator and discriminator are defined using the [Keras Sequential API](https://www.tensorflow.org/guide/keras#sequential_model)."]},{"cell_type":"markdown","metadata":{"id":"-tEyxE-GMC48"},"source":["### The Generator\n","\n","The generator uses `tf.keras.layers.Conv2DTranspose` (upsampling) layers to produce an image from a seed (random noise). Start with a `Dense` layer that takes this seed as input, then upsample several times until you reach the desired image size of 28x28x1. Notice the `tf.keras.layers.LeakyReLU` activation for each layer, except the output layer which uses tanh."]},{"cell_type":"markdown","metadata":{"id":"msfaH8ZYz6FC"},"source":["We use *assert* for debugging purpose. If there is an error in the code, the *assert* function will raise this error. The API for *assert* is [here](https://www.w3schools.com/python/ref_keyword_assert.asp#:~:text=The%20assert%20keyword%20is%20used,program%20will%20raise%20an%20AssertionError.)."]},{"cell_type":"markdown","source":["#### Conventional Generator"],"metadata":{"id":"Gp7BXqDFUIPV"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"6bpTcDqoLWjY"},"outputs":[],"source":["# define generator\n","def make_generator_model():\n","\n","    # use sequential API\n","    model = tf.keras.Sequential()\n","\n","    # start with dense layers\n","    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(784,)))\n","    model.add(layers.BatchNormalization())\n","    model.add(layers.LeakyReLU())\n","\n","    # reshape\n","    model.add(layers.Reshape((7, 7, 256)))\n","    assert model.output_shape == (None, 7, 7, 256)  # Note: None is the batch size\n","\n","    # add Conv2DTranspose layer\n","    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))\n","    assert model.output_shape == (None, 7, 7, 128)\n","    model.add(layers.BatchNormalization())\n","    model.add(layers.LeakyReLU())\n","\n","    # add Conv2DTranspose layer\n","    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))\n","    assert model.output_shape == (None, 14, 14, 64)\n","    model.add(layers.BatchNormalization())\n","    model.add(layers.LeakyReLU())\n","\n","    # add Conv2DTranspose layer\n","    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))\n","    assert model.output_shape == (None, 28, 28, 1)\n","\n","    return model"]},{"cell_type":"markdown","source":["#### Generator with Attention"],"metadata":{"id":"WVIlAm8YULEp"}},{"cell_type":"code","source":["import tensorflow as tf\n","from tensorflow.keras import layers, models\n","\n","def make_generator_model():\n","    # Input layer\n","    inputs = layers.Input(shape=(784,))\n","\n","    # Dense layer for initial processing\n","    x = layers.Dense(7*7*256, use_bias=False)(inputs)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.LeakyReLU()(x)\n","\n","    # Reshape to fit for multi-head attention\n","    x = layers.Reshape((49, 256))(x)  # 7*7 = 49\n","\n","    # MultiHead Attention layer\n","    # Since 'query', 'key', and 'value' can be the same tensor for self-attention,\n","    # we pass 'x' three times to the layer.\n","    attention_output = layers.MultiHeadAttention(num_heads=8, key_dim=64)(x, x, x)\n","    attention_output = layers.LeakyReLU()(attention_output)\n","\n","    # Combine attention output with the original sequence\n","    # Optionally, you might want to combine or process the attention output further\n","    # before moving on to the next layers; a simple way is to concatenate.\n","    x = layers.Concatenate(axis=-1)([x, attention_output])\n","\n","    # Proceed with the model as before\n","    x = layers.Flatten()(x)\n","    x = layers.Reshape((7, 7, 512))(x)  # Adjusted due to concatenation\n","\n","    x = layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False)(x)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.LeakyReLU()(x)\n","\n","    x = layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False)(x)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.LeakyReLU()(x)\n","\n","    x = layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh')(x)\n","\n","    # Create model\n","    model = models.Model(inputs=inputs, outputs=x)\n","\n","    return model\n"],"metadata":{"id":"QshRMIu8KDtg","executionInfo":{"status":"ok","timestamp":1711638284691,"user_tz":240,"elapsed":217,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":13,"outputs":[]},{"cell_type":"markdown","source":["#### Generator U-net"],"metadata":{"id":"pNbhVaaGUQFo"}},{"cell_type":"code","source":["import tensorflow as tf\n","from tensorflow.keras import layers, models\n","\n","def make_generator_model():\n","    inputs = layers.Input(shape=(28, 28, 1))\n","\n","    # Encoder part\n","    c1 = layers.Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)\n","    c1 = layers.BatchNormalization()(c1)\n","    p1 = layers.MaxPooling2D((2, 2))(c1)\n","\n","    c2 = layers.Conv2D(128, (3, 3), activation='relu', padding='same')(p1)\n","    c2 = layers.BatchNormalization()(c2)\n","    p2 = layers.MaxPooling2D((2, 2))(c2)\n","\n","    # Bottleneck with MultiHeadAttention\n","    # We need to flatten and reshape to apply MultiHeadAttention\n","    b = layers.Flatten()(p2)\n","    b = layers.Dense(49*128, activation='relu')(b)  # Adjusted to match the flattened size\n","    b = layers.Reshape((49, 128))(b)  # Prepare for MultiHeadAttention\n","\n","    # Applying MultiHeadAttention\n","    b = layers.MultiHeadAttention(num_heads=4, key_dim=32)(b, b, b)\n","\n","    # Decoder part, reshaping back to the spatial dimension needed for Conv2DTranspose\n","    b = layers.Flatten()(b)\n","    b = layers.Dense(7*7*128, activation='relu')(b)\n","    b = layers.Reshape((7, 7, 128))(b)\n","\n","    u1 = layers.Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same', activation='relu')(b)\n","    u1 = layers.BatchNormalization()(u1)\n","\n","    # Skipping connections\n","    u1 = layers.concatenate([u1, c2])\n","    u2 = layers.Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same', activation='relu')(u1)\n","    u2 = layers.BatchNormalization()(u2)\n","\n","    u2 = layers.concatenate([u2, c1])\n","    outputs = layers.Conv2D(1, (3, 3), activation='tanh', padding='same')(u2)\n","\n","    # Create and compile the model\n","    model = models.Model(inputs=inputs, outputs=outputs)\n","\n","    return model"],"metadata":{"id":"sB0aHCR0URjl"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"GyWgG09LCSJl"},"source":["Use the (as yet untrained) generator to create an image."]},{"cell_type":"code","execution_count":50,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":465},"executionInfo":{"elapsed":739,"status":"ok","timestamp":1711639001601,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"gl7jcC7TdPTG","outputId":"1ad6a2f8-67e8-406b-9806-8df1ba459442"},"outputs":[{"output_type":"stream","name":"stdout","text":["(1, 28, 28, 1)\n"]},{"output_type":"execute_result","data":{"text/plain":["<matplotlib.image.AxesImage at 0x798445cb8bb0>"]},"metadata":{},"execution_count":50},{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAApMElEQVR4nO3de1RV953+8QcVjqBwDCo3RQUTNVEk8RpqNCZSL51xxcZ0kjYzox2XRgcziUx+tqZt0kzaxYzNNK50ObrmEp12cqvNxTRtzERUrBO1lZg6RCViSMQoqBjOURCksH9/uGRCIobPDvAF8n6tddYS+D7uL5sNj8dz+JwIz/M8AQDQwXq43gAA4MuJAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgRC/XG/i0xsZGnThxQrGxsYqIiHC9HQCAked5OnfunFJSUtSjR8v3czpdAZ04cUKpqamutwEA+ILKyso0ePDgFj/e6QooNjZWkpSbm6tAINDq3JkzZ8zH6tevnzkj6aqN3pLGxkZz5vK5sDh58qQ507dvX3NGkk6dOmXOxMfHmzNRUVHmzMWLF80ZSb7udSclJZkzhw4dMmcyMzPNmeLiYnNGkpKTk82Zqqoqc8bP17a2ttac6d27tzkjSadPnzZnEhMTzZny8nJzJiMjw5yR/H3fDhw40LS+trZWDz/88Of+DGu3Alq7dq1+8pOfqLy8XJmZmfrZz36mSZMmfW7u8g+AQCBgumj8XMiWgvukjiogP980fj4nv+eho865n+P4/e9bP7no6Ghzxs/n5Oc4fr+2HXXt+TkPfsZXduQ17ufcddT1IPnbn99jfd73U7s8CeGFF15Qbm6uHn30Ub399tvKzMzUrFmzfDUvAKB7apcC+ulPf6rFixfr29/+tm644QatX79eMTExevrpp9vjcACALqjNC+jixYsqLCxUdnb2/x2kRw9lZ2dr9+7dn1lfV1encDjc7AYA6P7avIDOnDmjhoaGzzwQl5iYeMUH2vLy8hQMBptuPAMOAL4cnP8i6qpVqxQKhZpuZWVlrrcEAOgAbf4suAEDBqhnz56qqKho9v6KioorPl01EAj4foYKAKDravN7QFFRURo/frzy8/Ob3tfY2Kj8/HxlZWW19eEAAF1Uu/weUG5urhYsWKAJEyZo0qRJWrNmjaqrq/Xtb3+7PQ4HAOiC2qWA7r77bp0+fVqPPPKIysvLdeONN2rLli2+fkMYANA9tdskhOXLl2v58uW+8zExMabf2LWOipCk/v37mzOSv7E/aWlp5oyfcTLp6enmTExMjDkj+Rvp4ec3viMjIzskI0ljxowxZ55//nlz5rbbbjNnLly4YM6MGDHCnJGkc+fOmTN+RgX5Gatz9uxZc8bPSB1JGj58uDnj5/vJT6awsNCckaRgMGjOxMXFmdbX1dW1ap3zZ8EBAL6cKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOBEuw0j/aJqamrU2NjY6vX19fXmY/gZ9ilJlZWV5kyPHvau93McP0M4x40bZ85IUkJCgjlj+ZpeZhlKe9nBgwfNGb/HmjBhgjnj53rwMyDU7zXe2mGSn/Txxx+bM7162X8E+ZmqP2nSJHNGkl5//XVzxs+wz8GDB5szERER5ozkbwiz9edra9dzDwgA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOdNpp2KFQSLW1ta1eP2DAAPMxampqzBlJGjJkiDlz5swZcyYjI8OcKSoqMmd27txpzkjSoEGDzJkLFy6YMzfddJM5U1hYaM5I0vvvv2/OjBo1ypwpLy83ZyZPnmzOnDx50pyR/E22rqioMGf8fG1Pnz5tzjQ0NJgzkr9r3PJz6zI/U+ynTJlizkjSiRMnzJnrr7/etL61P1u5BwQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATnTaYaQpKSnq3bt3q9dXVVWZj+FncKck3XjjjeaMn6GGJSUl5ozlnF1mHTR4WTgcNmfOnj1rzjzzzDPmTHR0tDkj+RsK6ec8+Bnc+fzzz5szCQkJ5owkZWZmmjMHDhwwZz788ENz5ujRo+bM0KFDzRlJio2NNWfS09PNmcOHD5szfr6XJGnYsGHmTH5+vml9XV1dq9ZxDwgA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnOi0w0gTEhJMAyUrKyvNx5gxY4Y541e/fv3MmR497P8+2LNnjzmzbNkyc0aS3nvvvQ7JjB071pzxO4z09OnT5kzPnj07JJOdnW3OjB8/3pyRpOPHj5szfr5OaWlp5sytt95qzvzxj380ZyTpo48+MmdGjx5tzvgZGutnKKskffDBB+aM9WdlTU2N1q1b97nruAcEAHCCAgIAONHmBfTDH/5QERERzW6jRo1q68MAALq4dnkMaPTo0dq6dev/HaRXp32oCQDgSLs0Q69evZSUlNQefzUAoJtol8eAjhw5opSUFKWnp+vee+/VsWPHWlxbV1encDjc7AYA6P7avIAmT56sjRs3asuWLVq3bp1KS0s1depUnTt37orr8/LyFAwGm26pqaltvSUAQCfU5gU0Z84cfeMb39DYsWM1a9Ys/fa3v1VVVZV++ctfXnH9qlWrFAqFmm5lZWVtvSUAQCfU7s8O6Nevn0aMGKGSkpIrfjwQCCgQCLT3NgAAnUy7/x7Q+fPndfToUSUnJ7f3oQAAXUibF9BDDz2kgoICffDBB3rrrbf09a9/XT179tQ3v/nNtj4UAKALa/P/gjt+/Li++c1vqrKyUgMHDtQtt9yiPXv2aODAgW19KABAFxbheZ7nehOfFA6HFQwGdf/995seG4qJiTEfq76+3pyR5OuJEn6e3Xf27FlzJiUlxZz5xS9+Yc5I/gYoDh482JwZNGiQOZOenm7OSNLevXvNmaKiInNm5syZ5oyf6/Wtt94yZyT5elx2/vz55sxvf/tbc8bP919GRoY5I0k33XSTOfP++++bMzfffLM5c7Vfb7maiIgIc6alx/BbcvHiRb3wwgsKhUKKi4trcR2z4AAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADAiXZ/QTq/EhIS1Lt371avr6ysNB/Dz5BLSerZs6c5k5iYaM4MGTLEnGlsbDRnnnjiCXNGkvLz882Z6dOnmzMVFRXmTHFxsTkjSSNHjjRn/AwWjY2NNWf8fG2TkpLMGcnfoNlTp06ZM0uXLjVndu/ebc74fT2yo0ePmjN+Bp9+/PHH5kxhYaE5I/n7WTRq1CjT+tra2lat4x4QAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnOi007BPnz6tQCDQ6vV+JrxWVVWZM5JUXV1tzhw8eNCcWbRokTnz6quvmjN/+MMfzBlJqqmpMWfOnj1rzrz77rvmjJ+vkST17dvXnNm+fbs542cadmlpqTkTFRVlzkhSTk6OOfPiiy+aM7/4xS/Mmccff9yc8fP9J0nDhg0zZ3bt2mXOWCb/X+ZnKr8kNTQ0mDPWifR1dXWtWsc9IACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwotMOI01OTjYN6Nu5c6f5GH/9139tzkjSxYsXzRk/QyH/9V//1Zzp0cP+b4qpU6eaM5L/QZdW48aNM2dGjBjh61h+BtROmTLFnPnGN75hzqxevdqcKSsrM2ck6e233zZnkpOTzZl//ud/Nmf+67/+y5zxcw1J0n//93+bM0lJSebMwoULzZktW7aYM5JUXl5uzlgGQ0tSREREq9ZxDwgA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnOi0w0gvXLggz/NavT42NtZ8jPPnz5szklRUVGTONDQ0mDN+BosOHDjQnPGzN0n6zW9+Y84MHTrUnDl27Jg5U1BQYM5IUkxMjDnz9NNPmzN/+Zd/ac6sWLHCnBkyZIg5I0n33XefOVNTU2POLF261JwZPXq0ORMKhcwZSaaByJcdOXLEnPnVr35lzgSDQXNG8vdzr7q62rS+tQObuQcEAHCCAgIAOGEuoJ07d2ru3LlKSUlRRESEXnnllWYf9zxPjzzyiJKTkxUdHa3s7Gxfd0kBAN2buYCqq6uVmZmptWvXXvHjq1ev1lNPPaX169dr79696tOnj2bNmqXa2tovvFkAQPdhfhLCnDlzNGfOnCt+zPM8rVmzRt///vd1xx13SJJ+/vOfKzExUa+88oruueeeL7ZbAEC30aaPAZWWlqq8vFzZ2dlN7wsGg5o8ebJ27959xUxdXZ3C4XCzGwCg+2vTArr8WuOJiYnN3p+YmNji65Dn5eUpGAw23VJTU9tySwCATsr5s+BWrVqlUCjUdCsrK3O9JQBAB2jTAkpKSpIkVVRUNHt/RUVF08c+LRAIKC4urtkNAND9tWkBpaWlKSkpSfn5+U3vC4fD2rt3r7KystryUACALs78LLjz58+rpKSk6e3S0lK98847io+P15AhQ/Tggw/qRz/6ka677jqlpaXpBz/4gVJSUjRv3ry23DcAoIszF9C+fft02223Nb2dm5srSVqwYIE2btyolStXqrq6WkuWLFFVVZVuueUWbdmyxddMJQBA9xXhWSZ+doBwOKxgMKiHH37YVFpvvPGG+VhTp041ZyR/Qxf79OljzvgZEjp+/Hhzxs9wVUlKSEgwZ9566y1z5qtf/ao543f4ZGVlpTkTHR1tzrz44ovmTGRkpDlz1113mTOSfP3ieEpKijlz6tQpc+b22283Zz788ENzRvI38NPPtefnH+h+Pyc/Q47PnDljWl9bW6u8vDyFQqGrPq7v/FlwAIAvJwoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJwwvxxDR6moqFBUVFSr12dkZJiPceLECXNGkq8X1/MzudbPtOlXX33VnFm6dKk5I0l79+41Z2bPnm3OrF692px56KGHzBlJev31182Z/v37mzNDhw41Z0aPHm3O/O53vzNnJOmGG24wZ/x8P/kZxr9p0yZzJjMz05yRpPXr15szfn4+JCYmmjOnT582ZyR/U9UtP4slqbGxsVXruAcEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE502mGk06ZNU0xMTKvXHzlyxHyMm266yZyRpGPHjpkzwWDQnJk+fbo586tf/cqcOXv2rDkj+Ruwet9995kzhw4dMmdKSkrMGUl64403zJnz58+bM4WFhebM8OHDzRm/ioqKzJnvfve75swLL7xgzgwcONCcae1wzE/r0cP+b/T6+npzpnfv3uaMn+8/SRo2bJg5Yx2wWl1d3ap13AMCADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACciPM/zXG/ik8LhsILBoJYsWaKoqKhW5xoaGszH+spXvmLOSNJLL71kzkyaNMmcsXz+l7V2COAnffzxx+aMJPXp08ec8TOEc+7cueaM389p0KBB5sy7775rzvgZYOpn4K6f4a+Sv2s8OzvbnHnllVfMmdzcXHPmzJkz5owkbd682ZyJiIgwZ8aOHWvOREZGmjOSVFxcbM7Ex8eb1tfV1emJJ55QKBRSXFxci+u4BwQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATvRyvYGWDB48WL179271+hMnTpiPsXPnTnNGki5cuGDOWIf5SdKOHTvMmT//8z83Z7Zu3WrOSNI111xjzgwbNsycqampMWdOnz5tzkjSzJkzzZlz586ZM6NGjTJnGhsbzZk1a9aYM5K0cuVKc8bP99Nf/MVfmDP79+83ZzIyMswZSZo4caI5c+2115ozfgaE+r3GJ0yYYM5cbaDolbT2e5Z7QAAAJyggAIAT5gLauXOn5s6dq5SUFEVERHzm9TwWLlyoiIiIZrfZs2e31X4BAN2EuYCqq6uVmZmptWvXtrhm9uzZOnnyZNPtueee+0KbBAB0P+YnIcyZM0dz5sy56ppAIKCkpCTfmwIAdH/t8hjQjh07lJCQoJEjR2rZsmWqrKxscW1dXZ3C4XCzGwCg+2vzApo9e7Z+/vOfKz8/X//0T/+kgoICzZkzRw0NDVdcn5eXp2Aw2HRLTU1t6y0BADqhNv89oHvuuafpzxkZGRo7dqyGDx+uHTt2aMaMGZ9Zv2rVKuXm5ja9HQ6HKSEA+BJo96dhp6ena8CAASopKbnixwOBgOLi4prdAADdX7sX0PHjx1VZWank5OT2PhQAoAsx/xfc+fPnm92bKS0t1TvvvKP4+HjFx8frscce0/z585WUlKSjR49q5cqVuvbaazVr1qw23TgAoGszF9C+fft02223Nb19+fGbBQsWaN26dTpw4ID+8z//U1VVVUpJSdHMmTP1+OOPKxAItN2uAQBdnrmApk+fLs/zWvz4G2+88YU2dNmpU6dMpVVXV2c+hp8hl5I0evRoc6aoqMic6dHD/j+kfgZC1tbWmjOSvwGwDzzwgDmzePFic+aWW24xZyTpiSeeMGf8DNR88cUXzZndu3ebM5MmTTJnJLX4mO3VfPTRR+ZMVVWVORMTE2POHDlyxJyRpN///vfmjGWI8mV+BouOGDHCnPF7rPfff9+0vrU/j5kFBwBwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACfa/CW520pWVpZp6m1hYaH5GH5fIsLPlOq/+Zu/MWc2bNhgzkybNs2cOXnypDkjSfHx8ebM+fPnzRk/L2b40EMPmTOSdO7cOXNmz5495oyfScaLFi0yZ/r27WvOSNKuXbvMmT59+pgzd911lznzpz/9yZy5+eabzRlJioqKMmciIiLMmau9wkBL8vPzzRlJSkxMNGe++tWvmtZXV1e3ah33gAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADAiQjPzxS8dhQOhxUMBrVs2TLTsNCOGowpSaFQyJzxM1DzxhtvNGfee+89c2b79u3mjCRlZmaaMw0NDebMnDlzzJl/+7d/M2ck6fbbbzdn+vfvb86sWbPGnPEz7PO2224zZyTppZdeMmfuvfdec2bbtm3mTHp6ujkzdOhQc0aSKioqzJkf//jH5syyZcvMGb8/v6677jpzpqamxrS+rq5OTz75pEKhkOLi4lpcxz0gAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCi0w4jffLJJxUdHd3qXFlZmflYx48fN2ckKSUlxZw5fPiwOXPHHXeYMwUFBebMgQMHzBnJ34DVW2+91Zx55plnzJmMjAxzRrIPXZSkSZMmmTN+riE/gzH9fm1XrFhhzvTq1cvXsazuu+8+c2bgwIG+jhUZGWnO/NVf/ZU5c/bs2Q7JSNKxY8fMmdmzZ5vWV1dX66677mIYKQCgc6KAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAEx0zPdCHqqoq1dbWtnq9n0GN4XDYnJGkIUOGmDPx8fHmjJ/9+TnOypUrzRlJeuutt8yZ//3f/zVn/AwW9bM3Sfrud79rzvTp08ec2bFjhzkzdepUc+bUqVPmjCT96Ec/MmcmTJhgznz00UfmzKBBg8yZefPmmTOS9Lvf/c6c2bdvnzlTUlJizowbN86ckfz9jNi1a5dpfV1dXavWcQ8IAOAEBQQAcMJUQHl5eZo4caJiY2OVkJCgefPmqbi4uNma2tpa5eTkqH///urbt6/mz5/v67/HAADdm6mACgoKlJOToz179ujNN99UfX29Zs6cqerq6qY1K1as0K9//Wtt2rRJBQUFOnHihO6888423zgAoGszPQlhy5Ytzd7euHGjEhISVFhYqGnTpikUCuk//uM/9Oyzz+r222+XJG3YsEHXX3+99uzZo5tvvrntdg4A6NK+0GNAoVBI0v89q6KwsFD19fXKzs5uWjNq1CgNGTJEu3fvvuLfUVdXp3A43OwGAOj+fBdQY2OjHnzwQU2ZMkVjxoyRJJWXlysqKkr9+vVrtjYxMVHl5eVX/Hvy8vIUDAabbqmpqX63BADoQnwXUE5OjoqKivT8889/oQ2sWrVKoVCo6VZWVvaF/j4AQNfg6xdRly9frtdee007d+7U4MGDm96flJSkixcvqqqqqtm9oIqKCiUlJV3x7woEAgoEAn62AQDowkz3gDzP0/Lly/Xyyy9r27ZtSktLa/bx8ePHKzIyUvn5+U3vKy4u1rFjx5SVldU2OwYAdAume0A5OTl69tlntXnzZsXGxjY9rhMMBhUdHa1gMKhFixYpNzdX8fHxiouL0/3336+srCyeAQcAaMZUQOvWrZMkTZ8+vdn7N2zYoIULF0qSnnzySfXo0UPz589XXV2dZs2apX/5l39pk80CALoPUwF5nve5a3r37q21a9dq7dq1vjclSZGRkYqMjGz1+q985SvmY9x4443mjORvkGR0dLQ5M2XKFHOmd+/e5swnnzZvMXHiRHPmwoUL5sxTTz1lzvjZm9/cnj17fB3L6tPPLm2N9evX+zrW3r17zZmqqipzpm/fvubM9773PXPGz7BPSbrvvvvMmWHDhpkzr7/+ujnTv39/c0ZSi78SczWffKy/NVr7fc4sOACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADjh6xVRO8KJEydMr5R6+PBh8zH++Mc/mjOSWnx116u59dZbzZlt27aZM3369DFnHnnkEXNG8jfJuLa21pzxM134+PHj5owkHTx40Jypq6szZyZMmGDO/PjHPzZn/ExUl6Trr7/enPEznfnVV181ZyorK80Zv+fhN7/5jTmzdetWc2bq1KnmTHp6ujkjSTExMebMzp07Tevr6+tbtY57QAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgRITneZ7rTXxSOBxWMBjU008/bRqa9+KLL5qPlZGRYc5I0oULFzoks2/fPnNm0KBB5swtt9xizkjSe++9Z84MHTrUnGloaDBn/Aw9laQDBw6YM6WlpebM3LlzzZmPP/7YnGlsbDRnJOm6664zZw4dOmTO3H777ebMAw88YM7k5uaaM5K/a69HD/u/6999911zxjKs+ZMiIyPNGevg09raWn3ve99TKBRSXFxci+u4BwQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATvRyvYGWbNmyRVFRUa1ef8MNN5iPsX//fnNGkpKSknzlrK699lpzxnLOLjt48KA5I0mZmZnmzL//+7+bM1OnTjVnysrKzBlJGjVqlDmzcOFCc8bPeYiIiDBnsrKyzBlJOn36tDlTUVFhzrz//vvmzPTp080ZP98XklRUVGTO1NXVmTN+Bs1OmDDBnJGkAQMGmDPWIb0XL15s1TruAQEAnKCAAABOUEAAACcoIACAExQQAMAJCggA4AQFBABwggICADhBAQEAnKCAAABOUEAAACcoIACAE512GOnUqVMVHR3d6vXXXHON+RgjR440ZySpqqrKnPEzHHPmzJnmzKFDh8yZQYMGmTOSv6GG27dvN2dWrlxpzixatMickfxdR3/4wx/MmWAwaM74GUaakpJizkjSrFmzzJnHH3/cnLnzzjvNmXnz5pkzzz33nDkjSb162X9E5uXlmTMffPCBOVNSUmLOSNLhw4fNmfHjx5vWX7hwoVXruAcEAHCCAgIAOGEqoLy8PE2cOFGxsbFKSEjQvHnzVFxc3GzN9OnTFRER0ey2dOnSNt00AKDrMxVQQUGBcnJytGfPHr355puqr6/XzJkzVV1d3Wzd4sWLdfLkyabb6tWr23TTAICuz/QI25YtW5q9vXHjRiUkJKiwsFDTpk1ren9MTEyHvWooAKBr+kKPAYVCIUlSfHx8s/c/88wzGjBggMaMGaNVq1appqamxb+jrq5O4XC42Q0A0P35fhp2Y2OjHnzwQU2ZMkVjxoxpev+3vvUtDR06VCkpKTpw4IC+853vqLi4WC+99NIV/568vDw99thjfrcBAOiifBdQTk6OioqKtGvXrmbvX7JkSdOfMzIylJycrBkzZujo0aMaPnz4Z/6eVatWKTc3t+ntcDis1NRUv9sCAHQRvgpo+fLleu2117Rz504NHjz4qmsnT54s6dIvTV2pgAKBgAKBgJ9tAAC6MFMBeZ6n+++/Xy+//LJ27NihtLS0z8288847kqTk5GRfGwQAdE+mAsrJydGzzz6rzZs3KzY2VuXl5ZIujRWJjo7W0aNH9eyzz+prX/ua+vfvrwMHDmjFihWaNm2axo4d2y6fAACgazIV0Lp16yRd+mXTT9qwYYMWLlyoqKgobd26VWvWrFF1dbVSU1M1f/58ff/732+zDQMAugfzf8FdTWpqqgoKCr7QhgAAXw4R3ue1SgcLh8MKBoP6u7/7O9OTExISEszHau3E1k/zMz3az5Tq48ePmzN+pjlHRkaaM5KUmJhozpw5c8acGTZsmDnj94kt7777rjnTv39/c8bPL2q39KsMV+N34ruficmf/p+R1vAzWd6PP/3pT75yfh67/vQzg1sjKirKnLnSk7paw8/XdtKkSab1tbW1evzxxxUKhRQXF9fiOoaRAgCcoIAAAE5QQAAAJyggAIATFBAAwAkKCADgBAUEAHCCAgIAOEEBAQCcoIAAAE5QQAAAJyggAIATvl+Su70lJiaqd+/erV5fUVHh6xh+fPjhh+ZMa16879MyMjLMmQ8++MCcGTdunDkjSZs2bTJnsrKyzJm6ujpzpra21pyRpJ49e3bazMCBA82Zvn37mjOStHDhQnPmyJEj5oyf4bR/9md/Zs4UFhaaM5LUq5f9R+TXvvY1cyYUCpkzfn4+SP4Gn1qH57Z20DP3gAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATlBAAAAnKCAAgBOdbhac53mS7LO8OnJemJ9jtXY20if5mRfm53OqqakxZyTp4sWL5oyf8+DnOH511HXk5zzU19ebM34+H8nfNeHnPPj52vrZm9/z4OdzioiI6JDj+P2+7YifX5fXX/553pII7/NWdLDjx48rNTXV9TYAAF9QWVmZBg8e3OLHO10BNTY26sSJE4qNjf3MvyTC4bBSU1NVVlamuLg4Rzt0j/NwCefhEs7DJZyHSzrDefA8T+fOnVNKSop69Gj5kZ5O919wPXr0uGpjSlJcXNyX+gK7jPNwCefhEs7DJZyHS1yfh2Aw+LlreBICAMAJCggA4ESXKqBAIKBHH31UgUDA9Vac4jxcwnm4hPNwCefhkq50HjrdkxAAAF8OXeoeEACg+6CAAABOUEAAACcoIACAE12mgNauXathw4apd+/emjx5sn7/+9+73lKH++EPf6iIiIhmt1GjRrneVrvbuXOn5s6dq5SUFEVEROiVV15p9nHP8/TII48oOTlZ0dHRys7O1pEjR9xsth193nlYuHDhZ66P2bNnu9lsO8nLy9PEiRMVGxurhIQEzZs3T8XFxc3W1NbWKicnR/3791ffvn01f/58VVRUONpx+2jNeZg+ffpnroelS5c62vGVdYkCeuGFF5Sbm6tHH31Ub7/9tjIzMzVr1iydOnXK9dY63OjRo3Xy5Mmm265du1xvqd1VV1crMzNTa9euveLHV69eraeeekrr16/X3r171adPH82aNcv3sNnO6vPOgyTNnj272fXx3HPPdeAO219BQYFycnK0Z88evfnmm6qvr9fMmTNVXV3dtGbFihX69a9/rU2bNqmgoEAnTpzQnXfe6XDXba8150GSFi9e3Ox6WL16taMdt8DrAiZNmuTl5OQ0vd3Q0OClpKR4eXl5DnfV8R599FEvMzPT9TackuS9/PLLTW83NjZ6SUlJ3k9+8pOm91VVVXmBQMB77rnnHOywY3z6PHie5y1YsMC74447nOzHlVOnTnmSvIKCAs/zLn3tIyMjvU2bNjWtOXTokCfJ2717t6tttrtPnwfP87xbb73Ve+CBB9xtqhU6/T2gixcvqrCwUNnZ2U3v69Gjh7Kzs7V7926HO3PjyJEjSklJUXp6uu69914dO3bM9ZacKi0tVXl5ebPrIxgMavLkyV/K62PHjh1KSEjQyJEjtWzZMlVWVrreUrsKhUKSpPj4eElSYWGh6uvrm10Po0aN0pAhQ7r19fDp83DZM888owEDBmjMmDFatWqV75dwaC+dbhjpp505c0YNDQ1KTExs9v7ExEQdPnzY0a7cmDx5sjZu3KiRI0fq5MmTeuyxxzR16lQVFRUpNjbW9facKC8vl6QrXh+XP/ZlMXv2bN15551KS0vT0aNH9fDDD2vOnDnavXu3r9eW6uwaGxv14IMPasqUKRozZoykS9dDVFSU+vXr12xtd74ernQeJOlb3/qWhg4dqpSUFB04cEDf+c53VFxcrJdeesnhbpvr9AWE/zNnzpymP48dO1aTJ0/W0KFD9ctf/lKLFi1yuDN0Bvfcc0/TnzMyMjR27FgNHz5cO3bs0IwZMxzurH3k5OSoqKjoS/E46NW0dB6WLFnS9OeMjAwlJydrxowZOnr0qIYPH97R27yiTv9fcAMGDFDPnj0/8yyWiooKJSUlOdpV59CvXz+NGDFCJSUlrrfizOVrgOvjs9LT0zVgwIBueX0sX75cr732mrZv397s5VuSkpJ08eJFVVVVNVvfXa+Hls7DlUyePFmSOtX10OkLKCoqSuPHj1d+fn7T+xobG5Wfn6+srCyHO3Pv/PnzOnr0qJKTk11vxZm0tDQlJSU1uz7C4bD27t37pb8+jh8/rsrKym51fXiep+XLl+vll1/Wtm3blJaW1uzj48ePV2RkZLProbi4WMeOHetW18PnnYcreeeddySpc10Prp8F0RrPP/+8FwgEvI0bN3oHDx70lixZ4vXr188rLy93vbUO9fd///fejh07vNLSUu9//ud/vOzsbG/AgAHeqVOnXG+tXZ07d87bv3+/t3//fk+S99Of/tTbv3+/9+GHH3qe53n/+I//6PXr18/bvHmzd+DAAe+OO+7w0tLSvAsXLjjeedu62nk4d+6c99BDD3m7d+/2SktLva1bt3rjxo3zrrvuOq+2ttb11tvMsmXLvGAw6O3YscM7efJk062mpqZpzdKlS70hQ4Z427Zt8/bt2+dlZWV5WVlZDnfd9j7vPJSUlHj/8A//4O3bt88rLS31Nm/e7KWnp3vTpk1zvPPmukQBeZ7n/exnP/OGDBniRUVFeZMmTfL27Nnjeksd7u677/aSk5O9qKgob9CgQd7dd9/tlZSUuN5Wu9u+fbsn6TO3BQsWeJ536anYP/jBD7zExEQvEAh4M2bM8IqLi91uuh1c7TzU1NR4M2fO9AYOHOhFRkZ6Q4cO9RYvXtzt/pF2pc9fkrdhw4amNRcuXPD+9m//1rvmmmu8mJgY7+tf/7p38uRJd5tuB593Ho4dO+ZNmzbNi4+P9wKBgHfttdd6/+///T8vFAq53fin8HIMAAAnOv1jQACA7okCAgA4QQEBAJyggAAATlBAAAAnKCAAgBMUEADACQoIAOAEBQQAcIICAgA4QQEBAJyggAAATvx/dOMHMvdQK0EAAAAASUVORK5CYII=\n"},"metadata":{}}],"source":["# create generator model\n","generator = make_generator_model()\n","\n","# create noise and throw noise into the generator\n","noise = tf.random.normal([1, 784])\n","# noise = tf.random.normal([1, 28, 28, 1])\n","generated_image = generator(noise, training=False)\n","print(generated_image.shape)\n","\n","# plot an output from generator that is untrained\n","plt.imshow(generated_image[0, :, :, 0], cmap='gray')"]},{"cell_type":"markdown","metadata":{"id":"D0IKnaCtg6WE"},"source":["### The Discriminator\n","\n","The discriminator is a CNN-based image classifier. Notice that in the beginning of the `make_discriminator_model()` function below the input_shape must be the same as the output shape of the generator."]},{"cell_type":"code","execution_count":15,"metadata":{"id":"dw2tPLmk2pEP","executionInfo":{"status":"ok","timestamp":1711638300202,"user_tz":240,"elapsed":339,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# define discriminator model\n","def make_discriminator_model():\n","\n","    # use sequential API\n","    model = tf.keras.Sequential()\n","\n","    # add Conv2D layer\n","    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same', input_shape=[28, 28, 1]))\n","    model.add(layers.LeakyReLU())\n","    model.add(layers.Dropout(0.3))\n","\n","    # add Conv2D layer\n","    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))\n","    model.add(layers.LeakyReLU())\n","    model.add(layers.Dropout(0.3))\n","\n","    # flatten and output 1 neuron\n","    model.add(layers.Flatten())\n","    model.add(layers.Dense(1))\n","\n","    return model"]},{"cell_type":"markdown","metadata":{"id":"QhPneagzCaQv"},"source":["Use the (as yet untrained) discriminator to classify the generated images as real or fake. The model will be trained to output positive values for real images, and negative values for fake images."]},{"cell_type":"code","execution_count":51,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":337,"status":"ok","timestamp":1711639005426,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"gDkA05NE6QMs","outputId":"b9cd15f5-6616-4c35-fafb-7165956f3789"},"outputs":[{"output_type":"stream","name":"stdout","text":["tf.Tensor([[0.00580299]], shape=(1, 1), dtype=float32)\n"]}],"source":["# define discriminator\n","discriminator = make_discriminator_model()\n","decision = discriminator(generated_image)\n","print(decision)"]},{"cell_type":"markdown","metadata":{"id":"OuPl5e3JCOEf"},"source":["## How does GAN learn?\n","\n","According to the [original paper](https://arxiv.org/pdf/1406.2661.pdf), we can set up the situation as the following. Suppose the discriminator is $D(x; \\theta_d)$ using noisy data $x \\sim p_z(z)$ where $p_z$ is some prior distribution and the generator is $G(z; \\theta_g)$. Then the goal is to learn to train $D$ to maximize the probability of assigning the correct label to both training examples and samples from $G$. We simultaneously train $G$ to minimize $\\log(1 - D(G(z)))$.\n","\n","In other words, $D$ and $G$ play the following two-layer minimax game with the objective function $\\mathcal{V}(G, D)$:\n","$$ \\min_G \\max_D \\mathcal{V}(D, G) = \\mathbb{E}_{x \\sim p_{\\text{data}(x)}} [\\log D(x)] + \\mathbb{E}_{z \\sim p_z(z)} [\\log (1 - D(G(z)))]$$\n","\n","In each learning step, the gradient for the discriminator is updated:\n","$$\\nabla_{\\theta_d} \\frac{1}{m} \\sum_{i=1}^m [\\log D(x^{(i)}) + \\log(1 - D(G(z^{(i)})))]$$\n","where $m$ refers to samples and the gradient is taken w.r.t $\\theta_d$ (parameters for the discriminator $D$).\n","\n","The gradient for the generator is updated:\n","$$\\nabla_{\\theta_g} \\frac{1}{m} \\sum_{i=1}^m \\log(1 - D(G(z^{(i)})))$$\n","where $m$ refers to the samples and the gradient is taken w.r.t. $\\theta_g$ (parameters for the generator $G$)."]},{"cell_type":"markdown","metadata":{"id":"0FMYgY_mPfTi"},"source":["## Define the loss and optimizers\n","\n","Define loss functions and optimizers for both models.\n"]},{"cell_type":"code","execution_count":52,"metadata":{"id":"psQfmXxYKU3X","executionInfo":{"status":"ok","timestamp":1711639008994,"user_tz":240,"elapsed":1,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# This method returns a helper function to compute cross entropy loss\n","cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)"]},{"cell_type":"markdown","metadata":{"id":"Jd-3GCUEiKtv"},"source":["### Generator loss\n","The generator's loss quantifies how well it was able to trick the discriminator. Intuitively, if the generator is performing well, the discriminator will classify the fake images as real (or 1). Here, compare the discriminators decisions on the generated images to an array of 1s."]},{"cell_type":"code","execution_count":53,"metadata":{"id":"90BIcCKcDMxz","executionInfo":{"status":"ok","timestamp":1711639009252,"user_tz":240,"elapsed":1,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# def loss for generator\n","def generator_loss(fake_output):\n","    return cross_entropy(tf.ones_like(fake_output), fake_output)"]},{"cell_type":"markdown","metadata":{"id":"PKY_iPSPNWoj"},"source":["### Discriminator loss\n","\n","This method quantifies how well the discriminator is able to distinguish real images from fakes. It compares the discriminator's predictions on real images to an array of 1s, and the discriminator's predictions on fake (generated) images to an array of 0s."]},{"cell_type":"code","execution_count":54,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2,"status":"ok","timestamp":1711639009983,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"dO3db2Ut5Hzu","outputId":"cd616c46-a886-408c-a75a-f8689e5acdd2"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(<tf.Tensor: shape=(3,), dtype=int32, numpy=array([1, 1, 1], dtype=int32)>,\n"," <tf.Tensor: shape=(3,), dtype=int32, numpy=array([0, 0, 0], dtype=int32)>)"]},"metadata":{},"execution_count":54}],"source":["# check\n","tf.ones_like([1,2,3]), tf.zeros_like([1,2,3])"]},{"cell_type":"code","execution_count":55,"metadata":{"id":"wkMNfBWlT-PV","executionInfo":{"status":"ok","timestamp":1711639009983,"user_tz":240,"elapsed":1,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# def loss for discriminator\n","def discriminator_loss(real_output, fake_output):\n","    real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n","    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)\n","    total_loss = real_loss + fake_loss\n","    return total_loss"]},{"cell_type":"markdown","metadata":{"id":"MgIc7i0th_Iu"},"source":["## Optimizers\n","\n","The discriminator and the generator optimizers are different since you will train two networks separately."]},{"cell_type":"code","execution_count":56,"metadata":{"id":"iWCn_PVdEJZ7","executionInfo":{"status":"ok","timestamp":1711639012333,"user_tz":240,"elapsed":1,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# define two separate optimizer for the generator and the discriminator\n","generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n","discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"]},{"cell_type":"markdown","metadata":{"id":"mWtinsGDPJlV"},"source":["### Save checkpoints\n","This notebook also demonstrates how to save and restore models, which can be helpful in case a long running training task is interrupted."]},{"cell_type":"code","execution_count":22,"metadata":{"id":"CA1w-7s2POEy","executionInfo":{"status":"ok","timestamp":1711638319353,"user_tz":240,"elapsed":445,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# save and checkpoint\n","checkpoint_dir = './training_checkpoints'\n","checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n","checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n","                                 discriminator_optimizer=discriminator_optimizer,\n","                                 generator=generator,\n","                                 discriminator=discriminator)"]},{"cell_type":"markdown","metadata":{"id":"Rw1fkAczTQYh"},"source":["## Define the training loop\n"]},{"cell_type":"code","execution_count":57,"metadata":{"id":"NS2GWywBbAWo","executionInfo":{"status":"ok","timestamp":1711639017264,"user_tz":240,"elapsed":586,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# args\n","EPOCHS = 3\n","noise_dim = 100\n","num_examples_to_generate = 25\n","\n","# You will reuse this seed overtime (so it's easier)\n","# to visualize progress in the animated GIF)\n","seed = tf.random.normal([num_examples_to_generate, noise_dim])"]},{"cell_type":"code","execution_count":58,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1,"status":"ok","timestamp":1711639017719,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"s3DjF6pn1Iok","outputId":"e6d812eb-b84f-4f51-9aac-a7915576b397"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["(TensorShape([25, 100]), tensorflow.python.framework.ops.EagerTensor)"]},"metadata":{},"execution_count":58}],"source":["# check\n","seed.shape, type(seed)"]},{"cell_type":"markdown","metadata":{"id":"jylSonrqSWfi"},"source":["The training loop begins with generator receiving a random seed as input. That seed is used to produce an image. The discriminator is then used to classify real images (drawn from the training set) and fakes images (produced by the generator). The loss is calculated for each of these models, and the gradients are used to update the generator and discriminator."]},{"cell_type":"code","execution_count":66,"metadata":{"id":"3t5ibNo05jCB","executionInfo":{"status":"ok","timestamp":1711639375300,"user_tz":240,"elapsed":617,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# Notice the use of `tf.function`\n","# This annotation causes the function to be \"compiled\".\n","@tf.function\n","def train_step(images, twoDim=False):\n","    # Flatten and mask the images instead of using random noise\n","    # Example mask: setting a random portion of the image to zero\n","    # This is a simple mask that zeroes out half of the image\n","    delta = 0\n","    # mask = tf.concat([tf.ones(shape=(784//2-delta,)), tf.zeros(shape=(784//2+delta,))], 0)\n","    # mask = tf.random.shuffle(mask)\n","    mask = np.ones((28, 28), dtype=np.float32)\n","    # Set the bottom right portion of the mask to 1\n","    mask[14:, 14:] = 0\n","    # Convert the numpy array to a TensorFlow tensor\n","    mask = mask.reshape(784, )\n","    mask = tf.convert_to_tensor(mask)\n","    flat_images = tf.reshape(images, [BATCH_SIZE, 784])\n","    masked_images = flat_images * mask\n","    if twoDim:\n","        masked_images = tf.reshape(masked_images, [BATCH_SIZE, 28, 28, 1])\n","\n","    # args\n","    # noise = tf.random.normal([BATCH_SIZE, noise_dim])\n","\n","    # use gradient tape\n","    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n","      generated_images = generator(masked_images, training=True)\n","\n","      real_output = discriminator(images, training=True)\n","      fake_output = discriminator(generated_images, training=True)\n","\n","      gen_loss = generator_loss(fake_output)\n","      disc_loss = discriminator_loss(real_output, fake_output)\n","\n","    # compute gradients\n","    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n","    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n","\n","    # apply gradients\n","    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n","    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n"]},{"cell_type":"markdown","metadata":{"id":"xw0A4Sw36pxl"},"source":["Notice that the `train()` function saves images in the current directory."]},{"cell_type":"code","source":["from tqdm import tqdm"],"metadata":{"id":"GnDRZsPDiKR9","executionInfo":{"status":"ok","timestamp":1711639375300,"user_tz":240,"elapsed":6,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":67,"outputs":[]},{"cell_type":"code","execution_count":68,"metadata":{"id":"2M7LmLtGEMQJ","executionInfo":{"status":"ok","timestamp":1711639375300,"user_tz":240,"elapsed":6,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# training\n","def train(dataset, epochs):\n","\n","    # for loop: this loop iterates epochs\n","    for epoch in range(epochs):\n","        start = time.time()\n","\n","        # for loop: iterate images in the dataset\n","        for image_batch in tqdm(dataset):\n","            train_step(image_batch)\n","\n","        # produce images for the GIF as you go\n","        # display.clear_output(wait=True)\n","        # generate_and_save_images(\n","        #     generator,\n","        #     epoch + 1,\n","        #     seed)\n","\n","        # save the model every 15 epochs\n","        # if (epoch + 1) % 15 == 0:\n","        #     checkpoint.save(file_prefix = checkpoint_prefix)\n","\n","        # print statement\n","        print ('Time for epoch {} is {} sec'.format(epoch + 1, time.time()-start))\n","        print('...')\n","        print('...')\n","        print('...')\n","        print('...')\n","\n","        # generate after the final epoch\n","        # display.clear_output(wait=True)\n","        # generate_and_save_images(generator, epochs, seed)"]},{"cell_type":"markdown","metadata":{"id":"2aFF7Hk3XdeW"},"source":["**Generate and save images**\n"]},{"cell_type":"code","execution_count":69,"metadata":{"id":"RmdVsmvhPxyy","executionInfo":{"status":"ok","timestamp":1711639375300,"user_tz":240,"elapsed":6,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[],"source":["# # def function\n","# def generate_and_save_images(model, epoch, test_input):\n","#     # Notice `training` is set to False.\n","#     # This is so all layers run in inference mode (batchnorm).\n","#     predictions = model(test_input, training=False)\n","\n","#     # figure\n","#     fig = plt.figure(figsize=(5, 5))\n","#     for i in range(predictions.shape[0]):\n","#         plt.subplot(5, 5, i+1)\n","#         if i == 0:\n","#             plt.title('ep='+str(epoch))\n","#         plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n","#         plt.axis('off')\n","\n","#     # save and display\n","#     plt.savefig('image_at_epoch_{:04d}.png'.format(epoch))\n","#     plt.show()"]},{"cell_type":"markdown","metadata":{"id":"dZrd4CdjR-Fp"},"source":["## Train the model\n","\n","Call the `train()` method defined above to train the generator and discriminator simultaneously. Note, training GANs can be tricky. It's important that the generator and discriminator do not overpower each other (e.g., that they train at a similar rate).\n","\n","At the beginning of the training, the generated images look like random noise. As training progresses, the generated digits will look increasingly real. After about 50 epochs, they resemble MNIST digits. This may take about one minute / epoch with the default settings on Colab."]},{"cell_type":"code","source":["tf.test.gpu_device_name()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":36},"id":"cyOf3tTrgiRK","executionInfo":{"status":"ok","timestamp":1711639375300,"user_tz":240,"elapsed":6,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"10d9affb-ed98-4cc7-80ac-a7dea5c4d6e0"},"execution_count":70,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'/device:GPU:0'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":70}]},{"cell_type":"code","execution_count":71,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Ly3UN0SLLY2l","outputId":"a156961a-c7dc-4a6f-9e58-08754e5bfdfe","executionInfo":{"status":"ok","timestamp":1711639405160,"user_tz":240,"elapsed":29864,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"outputs":[{"output_type":"stream","name":"stderr","text":["100%|██████████| 1875/1875 [00:29<00:00, 62.62it/s]"]},{"output_type":"stream","name":"stdout","text":["Time for epoch 1 is 29.94382071495056 sec\n","...\n","...\n","...\n","...\n","CPU times: user 13.4 s, sys: 2.47 s, total: 15.8 s\n","Wall time: 29.9 s\n"]},{"output_type":"stream","name":"stderr","text":["\n"]}],"source":["%%time\n","\n","# train\n","with tf.device('/device:GPU:0'):\n","    train(train_dataset, 1)"]},{"cell_type":"markdown","metadata":{"id":"rfM4YcPVPkNO"},"source":["Restore the latest checkpoint."]},{"cell_type":"markdown","source":["## Inference"],"metadata":{"id":"GbrG8yoItE6M"}},{"cell_type":"code","source":["def make_inference(input_image, generator, twoDim=False):\n","    # Ensure input_image is a tensor\n","    if not isinstance(input_image, tf.Tensor):\n","        input_image = tf.convert_to_tensor(input_image, dtype=tf.float32)\n","\n","    # Flatten the input image and apply the mask\n","    delta = 0\n","    # mask = tf.concat([tf.ones(shape=(784//2-delta,)), tf.zeros(shape=(784//2+delta,))], 0)\n","    # Create a binary mask of size 28x28\n","    mask = np.ones((28, 28), dtype=np.float32)\n","    # Set the bottom right portion of the mask to 1\n","    mask[14:, 14:] = 0\n","    # Convert the numpy array to a TensorFlow tensor\n","    mask = mask.reshape(784, )\n","    mask = tf.convert_to_tensor(mask)\n","    # print(mask.shape)\n","\n","    # mask = tf.random.shuffle(mask)\n","    flat_image = tf.reshape(input_image, [784])\n","    masked_image = flat_image * mask\n","    if twoDim:\n","        masked_image = tf.reshape(masked_image, [28, 28, 1])\n","\n","    # Since the generator expects a batch dimension, add a batch dimension to the masked image\n","    masked_image = tf.expand_dims(masked_image, 0)  # Adds a batch dimension\n","\n","    # Generate the image using the generator\n","    # masked_image_for_gen = masked_image[:, 0:392]\n","    generated_image = generator(masked_image, training=False)  # Set training to False for inference\n","\n","    return masked_image, generated_image\n"],"metadata":{"id":"X_ScZdCwtFwT","executionInfo":{"status":"ok","timestamp":1711639405160,"user_tz":240,"elapsed":13,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}}},"execution_count":72,"outputs":[]},{"cell_type":"code","source":["masked_output_, output_ = make_inference(train_images[0], generator)\n","\n","print(masked_output_.shape, output_.shape)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"NmcI5Xi4y6t2","executionInfo":{"status":"ok","timestamp":1711639405160,"user_tz":240,"elapsed":12,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"4a01a842-bec1-48ed-d042-d1efc66a23e6"},"execution_count":73,"outputs":[{"output_type":"stream","name":"stdout","text":["(1, 784) (1, 28, 28, 1)\n"]}]},{"cell_type":"markdown","source":["## Loop: Train + Inference"],"metadata":{"id":"RR7taPh2G2ZV"}},{"cell_type":"code","source":["%%time\n","\n","for k in range(50):\n","\n","    # train\n","    with tf.device('/device:GPU:0'):\n","        train(train_dataset, EPOCHS)\n","\n","    # Assuming 'train_images' and 'train_labels' are defined and 'generator' is your model\n","    plt.figure(figsize=(20, 4))  # Adjusted size for better visibility\n","    T = 20\n","\n","    # Display real images\n","    for i in range(T):\n","        plt.subplot(3, T, i + 1)\n","        plt.imshow(train_images[i], cmap='gray')\n","        plt.title(f\"Real: {train_labels[i]}\")\n","        plt.axis('off')\n","\n","    # Display fake images generated by the model\n","    for i in range(T):\n","        masked_output_, output_ = make_inference(train_images[i], generator)\n","        masked_output_ = tf.reshape(masked_output_, [28, 28])\n","        plt.subplot(3, T, i + T + 1)  # +7 to move to the second row, right below the corresponding real image\n","        plt.imshow(masked_output_, cmap='gray')  # Adjust indexing if needed\n","        plt.title(f\"Masked: {train_labels[i]}\")\n","        plt.axis('off')\n","\n","    # Display fake images generated by the model\n","    for i in range(T):\n","        masked_output_, output_ = make_inference(train_images[i], generator)\n","        plt.subplot(3, T, i + T + T + 1)  # +7 to move to the second row, right below the corresponding real image\n","        plt.imshow(output_[0, :, :, 0], cmap='gray')  # Adjust indexing if needed\n","        plt.title(f\"Gen.: {train_labels[i]}\")\n","        plt.axis('off')\n","\n","    plt.tight_layout()\n","    plt.show()\n","\n","    # Chk point\n","    print(\"===============================!\")\n","    print(f\"Checkpoint {k+1}\")\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000,"output_embedded_package_id":"1tb9tiMdAQdksQo_qMGxggpfbm3ZpjpOB"},"id":"RCmzlLQoLQ3x","executionInfo":{"status":"ok","timestamp":1711643887695,"user_tz":240,"elapsed":4482546,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"57b58069-afc6-4047-d8f9-8417bec1029d"},"execution_count":74,"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}]},{"cell_type":"markdown","metadata":{"id":"k6qC-SbjK0yW"},"source":["## Next steps\n"]},{"cell_type":"markdown","metadata":{"id":"xjjkT9KAK6H7"},"source":["This tutorial has shown the complete code necessary to write and train a GAN. It is an adaption from this [Tensorflow notebook](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/generative/dcgan.ipynb). As a next step, you might like to experiment with a different dataset, for example the Large-scale Celeb Faces Attributes (CelebA) dataset [available on Kaggle](https://www.kaggle.com/jessicali9530/celeba-dataset). To learn more about GANs see the [NIPS 2016 Tutorial: Generative Adversarial Networks](https://arxiv.org/abs/1701.00160).\n"]}],"metadata":{"colab":{"machine_shape":"hm","provenance":[{"file_id":"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/dcgan.ipynb","timestamp":1657820300439}],"toc_visible":true,"gpuType":"T4"},"kernelspec":{"display_name":"Python 3","name":"python3"},"accelerator":"GPU"},"nbformat":4,"nbformat_minor":0}

docs/notebooks/ex11a - huggingface on names.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"collapsed_sections":[],"toc_visible":true,"authorship_tag":"ABX9TyMadllWCw44GdaKaDnCqZhQ"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"widgets":{"application/vnd.jupyter.widget-state+json":{"4a9d5e02f6e44367a218cc31ae7e2165":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_c01700362a7d4925964eb39ef4bdec42","IPY_MODEL_eea9da0bb4b14de1968de294af0ae697","IPY_MODEL_844ab1dfb0564ac9b064efb0da269132"],"layout":"IPY_MODEL_c4f8d862871049de84348db8476117ca"}},"c01700362a7d4925964eb39ef4bdec42":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_4f79214778ca4c0887299ecc32a6feea","placeholder":"​","style":"IPY_MODEL_2c90677882174d9d9134e96d0a421777","value":"Downloading: 100%"}},"eea9da0bb4b14de1968de294af0ae697":{"model_module":"@jupyter-widgets/controls","model_name":"FloatProgressModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_40dd6cebc71847749dad72af14e3a7ad","max":231508,"min":0,"orientation":"horizontal","style":"IPY_MODEL_bdee7fa9f5c44bafacc1dc514e95d755","value":231508}},"844ab1dfb0564ac9b064efb0da269132":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_580622c11c2740b399dd6bb3a7671e1b","placeholder":"​","style":"IPY_MODEL_f4ed18979f0c46e29e518f8c8daa7166","value":" 232k/232k [00:00&lt;00:00, 954kB/s]"}},"c4f8d862871049de84348db8476117ca":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"4f79214778ca4c0887299ecc32a6feea":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"2c90677882174d9d9134e96d0a421777":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"40dd6cebc71847749dad72af14e3a7ad":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"bdee7fa9f5c44bafacc1dc514e95d755":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"580622c11c2740b399dd6bb3a7671e1b":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f4ed18979f0c46e29e518f8c8daa7166":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"5cdcce2647354471b23569b624d43448":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_d3dc9d9a97444602a1309ed26b67759b","IPY_MODEL_52ad191a40c94b96b45ff498ba069288","IPY_MODEL_f2a47e55e438440986718bd0961e669d"],"layout":"IPY_MODEL_9975e8beeb4542c48433d027a3bd32f8"}},"d3dc9d9a97444602a1309ed26b67759b":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_91f399cae7c042bb85e1f00eb519607d","placeholder":"​","style":"IPY_MODEL_b89e26af73c24b77a8c47b6755b5fcfa","value":"Downloading: 100%"}},"52ad191a40c94b96b45ff498ba069288":{"model_module":"@jupyter-widgets/controls","model_name":"FloatProgressModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_088f83d499d24e4cb0ae6120cea7128a","max":28,"min":0,"orientation":"horizontal","style":"IPY_MODEL_d929ff07ee2e42da9546841513a87d64","value":28}},"f2a47e55e438440986718bd0961e669d":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_b039e1bdeaaf4b2f9beed995dc5dd873","placeholder":"​","style":"IPY_MODEL_9bd9538f2bc1427388a0709b8a2f90db","value":" 28.0/28.0 [00:00&lt;00:00, 1.04kB/s]"}},"9975e8beeb4542c48433d027a3bd32f8":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"91f399cae7c042bb85e1f00eb519607d":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"b89e26af73c24b77a8c47b6755b5fcfa":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"088f83d499d24e4cb0ae6120cea7128a":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"d929ff07ee2e42da9546841513a87d64":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"b039e1bdeaaf4b2f9beed995dc5dd873":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"9bd9538f2bc1427388a0709b8a2f90db":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"494ecbd431bd47abbda4fa6a163cb478":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_3ed2990b9d934b928ee205c9047d9625","IPY_MODEL_7bff12caf180452b96898b43bbf5012d","IPY_MODEL_5be62fa53b614844946df72d93d6c014"],"layout":"IPY_MODEL_1dfb0493b2864f5a858f3211efd40f17"}},"3ed2990b9d934b928ee205c9047d9625":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_58ba03e50f784324bbf61934bdb232f7","placeholder":"​","style":"IPY_MODEL_f18489508b0c48daa1f57c414af03bdb","value":"Downloading: 100%"}},"7bff12caf180452b96898b43bbf5012d":{"model_module":"@jupyter-widgets/controls","model_name":"FloatProgressModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_a454d8366cd442d98607589cdabee9eb","max":570,"min":0,"orientation":"horizontal","style":"IPY_MODEL_e617b0d7591c48499b6a637eb4edf639","value":570}},"5be62fa53b614844946df72d93d6c014":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_d790b12cba5d4e8f8cbe389595895025","placeholder":"​","style":"IPY_MODEL_f42c1aaf3a564b359af6023a196be7ad","value":" 570/570 [00:00&lt;00:00, 15.0kB/s]"}},"1dfb0493b2864f5a858f3211efd40f17":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"58ba03e50f784324bbf61934bdb232f7":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f18489508b0c48daa1f57c414af03bdb":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"a454d8366cd442d98607589cdabee9eb":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"e617b0d7591c48499b6a637eb4edf639":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"d790b12cba5d4e8f8cbe389595895025":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f42c1aaf3a564b359af6023a196be7ad":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"76670e7c58654ba4a9b200b095ba5ca8":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_21a3624d48bc44f3bf0b16b03cb96354","IPY_MODEL_a22a0ad369584c4fb656870d6fc6de1a","IPY_MODEL_59bf662271534f7cb9f303a19037696f"],"layout":"IPY_MODEL_784e754470984da5ab711918a32ec40e"}},"21a3624d48bc44f3bf0b16b03cb96354":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_36fc0a9cadf34e7ba6184915e4dd0901","placeholder":"​","style":"IPY_MODEL_692a45f3f12145f1b88cfb4b5cf37b6d","value":"Downloading: 100%"}},"a22a0ad369584c4fb656870d6fc6de1a":{"model_module":"@jupyter-widgets/controls","model_name":"FloatProgressModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_0dcae4f83f20425d93eb2f294c609df9","max":536063208,"min":0,"orientation":"horizontal","style":"IPY_MODEL_3a42e9ca4ff347b0989ca7ac84349dea","value":536063208}},"59bf662271534f7cb9f303a19037696f":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_b9334cee4be947f69cd5957c9ad4618f","placeholder":"​","style":"IPY_MODEL_a42a705c850f46199983a32ae2e46259","value":" 536M/536M [00:24&lt;00:00, 18.9MB/s]"}},"784e754470984da5ab711918a32ec40e":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"36fc0a9cadf34e7ba6184915e4dd0901":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"692a45f3f12145f1b88cfb4b5cf37b6d":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"0dcae4f83f20425d93eb2f294c609df9":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"3a42e9ca4ff347b0989ca7ac84349dea":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"b9334cee4be947f69cd5957c9ad4618f":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"a42a705c850f46199983a32ae2e46259":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}}}}},"cells":[{"cell_type":"code","execution_count":null,"metadata":{"id":"MpxBud2iQZ1o"},"outputs":[],"source":["# import\n","import numpy as np\n","import tensorflow as tf\n","from tqdm import tqdm"]},{"cell_type":"code","source":["# define names\n","names = [\n","    ['George', 'Washington'],\n","    ['John', 'Adams'],\n","    ['Thomas', 'Jefferson'],\n","    ['James', 'Madison'],\n","    ['James', 'Monroe'],\n","    ['John', 'Adams'],\n","    ['Andrew', 'Jackson']]\n","names = np.asarray(names)\n","names"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"aLXodUeCQfJp","executionInfo":{"status":"ok","timestamp":1665416431899,"user_tz":240,"elapsed":6,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"8db9e08b-eea3-473b-fab0-bc9f5ff4341e"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([['David', 'Alfego'],\n","       ['Alvin', 'Koontz'],\n","       ['Eric', 'Campbell'],\n","       ['Adam', 'Sullivan'],\n","       ['Ahmed', 'Tawfik'],\n","       ['Brian', 'Munson'],\n","       ['Cassidy', 'Konzelman'],\n","       ['Carlos', 'Loza'],\n","       ['Moloy', 'Chatterjee'],\n","       ['Marc', 'Duchatelier'],\n","       ['Xia', 'Cao'],\n","       ['Lax', 'Iyer'],\n","       ['Pingsha', 'Hu'],\n","       ['Kelsey', 'McDonald'],\n","       ['Hemanth', 'Reddy'],\n","       ['William', 'Ginn'],\n","       ['Walter', 'Jessen'],\n","       ['Matt', 'Robinson'],\n","       ['Moose', 'Tomasiello'],\n","       ['Priyanshu', 'Sinha'],\n","       ['Xiaodong', 'Zhu'],\n","       ['Yiqiao', 'Yin']], dtype='<U11')"]},"metadata":{},"execution_count":2}]},{"cell_type":"code","source":["# https://huggingface.co/bert-base-uncased\n","# import\n","try:\n","    from transformers import BertTokenizer, TFBertModel\n","except:\n","    !pip install transformers\n","    from transformers import BertTokenizer, TFBertModel"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9RqFZEPfRoeX","executionInfo":{"status":"ok","timestamp":1665416445939,"user_tz":240,"elapsed":14044,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"5de0740d-cff3-49a5-f452-6f28a6e60706"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting transformers\n","  Downloading transformers-4.22.2-py3-none-any.whl (4.9 MB)\n","\u001b[K     |████████████████████████████████| 4.9 MB 7.1 MB/s \n","\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n","Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (6.0)\n","Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.21.6)\n","Collecting huggingface-hub<1.0,>=0.9.0\n","  Downloading huggingface_hub-0.10.0-py3-none-any.whl (163 kB)\n","\u001b[K     |████████████████████████████████| 163 kB 12.2 MB/s \n","\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2022.6.2)\n","Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.64.1)\n","Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n","Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (5.0.0)\n","Collecting tokenizers!=0.11.3,<0.13,>=0.11.1\n","  Downloading tokenizers-0.12.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n","\u001b[K     |████████████████████████████████| 6.6 MB 49.4 MB/s \n","\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.8.0)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.9.0->transformers) (4.1.1)\n","Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.8.1)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2022.9.24)\n","Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n","Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n","Installing collected packages: tokenizers, huggingface-hub, transformers\n","Successfully installed huggingface-hub-0.10.0 tokenizers-0.12.1 transformers-4.22.2\n"]}]},{"cell_type":"code","source":["# load models\n","tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n","model = TFBertModel.from_pretrained(\"bert-base-uncased\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":232,"referenced_widgets":["4a9d5e02f6e44367a218cc31ae7e2165","c01700362a7d4925964eb39ef4bdec42","eea9da0bb4b14de1968de294af0ae697","844ab1dfb0564ac9b064efb0da269132","c4f8d862871049de84348db8476117ca","4f79214778ca4c0887299ecc32a6feea","2c90677882174d9d9134e96d0a421777","40dd6cebc71847749dad72af14e3a7ad","bdee7fa9f5c44bafacc1dc514e95d755","580622c11c2740b399dd6bb3a7671e1b","f4ed18979f0c46e29e518f8c8daa7166","5cdcce2647354471b23569b624d43448","d3dc9d9a97444602a1309ed26b67759b","52ad191a40c94b96b45ff498ba069288","f2a47e55e438440986718bd0961e669d","9975e8beeb4542c48433d027a3bd32f8","91f399cae7c042bb85e1f00eb519607d","b89e26af73c24b77a8c47b6755b5fcfa","088f83d499d24e4cb0ae6120cea7128a","d929ff07ee2e42da9546841513a87d64","b039e1bdeaaf4b2f9beed995dc5dd873","9bd9538f2bc1427388a0709b8a2f90db","494ecbd431bd47abbda4fa6a163cb478","3ed2990b9d934b928ee205c9047d9625","7bff12caf180452b96898b43bbf5012d","5be62fa53b614844946df72d93d6c014","1dfb0493b2864f5a858f3211efd40f17","58ba03e50f784324bbf61934bdb232f7","f18489508b0c48daa1f57c414af03bdb","a454d8366cd442d98607589cdabee9eb","e617b0d7591c48499b6a637eb4edf639","d790b12cba5d4e8f8cbe389595895025","f42c1aaf3a564b359af6023a196be7ad","76670e7c58654ba4a9b200b095ba5ca8","21a3624d48bc44f3bf0b16b03cb96354","a22a0ad369584c4fb656870d6fc6de1a","59bf662271534f7cb9f303a19037696f","784e754470984da5ab711918a32ec40e","36fc0a9cadf34e7ba6184915e4dd0901","692a45f3f12145f1b88cfb4b5cf37b6d","0dcae4f83f20425d93eb2f294c609df9","3a42e9ca4ff347b0989ca7ac84349dea","b9334cee4be947f69cd5957c9ad4618f","a42a705c850f46199983a32ae2e46259"]},"id":"nVKwEZ4CTxPP","executionInfo":{"status":"ok","timestamp":1665416477725,"user_tz":240,"elapsed":31790,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"f208a042-f1fd-4931-8ad3-3e67b719d9f2"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"],"application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"4a9d5e02f6e44367a218cc31ae7e2165"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"],"application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"5cdcce2647354471b23569b624d43448"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["Downloading:   0%|          | 0.00/570 [00:00<?, ?B/s]"],"application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"494ecbd431bd47abbda4fa6a163cb478"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":["Downloading:   0%|          | 0.00/536M [00:00<?, ?B/s]"],"application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"76670e7c58654ba4a9b200b095ba5ca8"}},"metadata":{}},{"output_type":"stream","name":"stderr","text":["Some layers from the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['nsp___cls', 'mlm___cls']\n","- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n","- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n","All the layers of TFBertModel were initialized from the model checkpoint at bert-base-uncased.\n","If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.\n"]}]},{"cell_type":"code","source":["# get model\n","def bert_encoder_(text):\n","    \"\"\"\n","    The function `bert_encoder_` encodes a string of text.\n","    text: string\n","    output: float that is the sum of the output layer of the bert model using the text as input\n","    \"\"\"\n","    encoded_input = tokenizer(text, return_tensors='tf')\n","    output = model(encoded_input)\n","    return np.asarray(output[0]).sum()"],"metadata":{"id":"BEvS7ifNSFOf"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# test a word\n","bert_encoder_('hello')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"lJw1VcCbSSqm","executionInfo":{"status":"ok","timestamp":1665416477916,"user_tz":240,"elapsed":197,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"9d9efb57-6ec4-49c1-a415-700a38ca3002"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["-29.341282"]},"metadata":{},"execution_count":6}]},{"cell_type":"code","source":["# test a name\n","i=1\n","[names[i,0], names[i,1], bert_encoder_(names[i,0]), bert_encoder_(names[i,1])]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"1qA7UCnzUkcB","executionInfo":{"status":"ok","timestamp":1665416478938,"user_tz":240,"elapsed":1024,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"e7b8621c-9918-4512-9fa4-e6d1306392aa"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Alvin', 'Koontz', -28.796902, -46.784615]"]},"metadata":{},"execution_count":7}]},{"cell_type":"code","source":["%%time \n","results_ = []\n","for i in tqdm(range(names.shape[0])):\n","    results_.append([names[i,0], names[i,1], bert_encoder_(names[i,0]), bert_encoder_(names[i,1])])\n","\n","# convert to array from list\n","results_ = np.asarray(results_)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"mokrhdSAST9L","executionInfo":{"status":"ok","timestamp":1665416489597,"user_tz":240,"elapsed":10666,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"d5522e04-4a44-4ba3-f4db-f8bcf2e90608"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["100%|██████████| 22/22 [00:11<00:00,  1.99it/s]"]},{"output_type":"stream","name":"stdout","text":["CPU times: user 9.81 s, sys: 345 ms, total: 10.2 s\n","Wall time: 11.1 s\n"]},{"output_type":"stream","name":"stderr","text":["\n"]}]},{"cell_type":"code","source":["# review results\n","np.argsort(results_[:, 2])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"J-FRUiSqUP-F","executionInfo":{"status":"ok","timestamp":1665416489597,"user_tz":240,"elapsed":15,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"ed3f89cf-cf49-4cd4-95dc-0bffc8d0bbe6"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([ 3, 18,  7,  2, 13,  0,  1,  9,  4, 11,  5,  6, 15, 17, 16, 12, 10,\n","       20,  8, 21, 19, 14])"]},"metadata":{},"execution_count":9}]},{"cell_type":"code","source":["results_"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"1N1tyFc8W2XC","executionInfo":{"status":"ok","timestamp":1665416489598,"user_tz":240,"elapsed":12,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"a3de1ac4-53c1-4709-eb84-448fd893286f"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([['David', 'Alfego', '-28.721024', '-39.18271'],\n","       ['Alvin', 'Koontz', '-28.796902', '-46.784615'],\n","       ['Eric', 'Campbell', '-27.960735', '-27.948498'],\n","       ['Adam', 'Sullivan', '-27.689796', '-28.539173'],\n","       ['Ahmed', 'Tawfik', '-28.935593', '-65.328156'],\n","       ['Brian', 'Munson', '-29.464714', '-37.82363'],\n","       ['Cassidy', 'Konzelman', '-29.72972', '-47.544243'],\n","       ['Carlos', 'Loza', '-27.927849', '-35.930717'],\n","       ['Moloy', 'Chatterjee', '-43.389526', '-44.164116'],\n","       ['Marc', 'Duchatelier', '-28.89674', '-65.417206'],\n","       ['Xia', 'Cao', '-36.73168', '-34.20746'],\n","       ['Lax', 'Iyer', '-29.118505', '-36.347607'],\n","       ['Pingsha', 'Hu', '-36.683655', '-29.627468'],\n","       ['Kelsey', 'McDonald', '-28.226707', '-31.136759'],\n","       ['Hemanth', 'Reddy', '-58.88199', '-32.675293'],\n","       ['William', 'Ginn', '-29.802353', '-38.848503'],\n","       ['Walter', 'Jessen', '-32.531944', '-38.93853'],\n","       ['Matt', 'Robinson', '-30.136854', '-30.245296'],\n","       ['Moose', 'Tomasiello', '-27.813456', '-47.346333'],\n","       ['Priyanshu', 'Sinha', '-55.41825', '-43.768623'],\n","       ['Xiaodong', 'Zhu', '-39.96691', '-30.180828'],\n","       ['Yiqiao', 'Yin', '-49.851707', '-28.030106']], dtype='<U32')"]},"metadata":{},"execution_count":10}]},{"cell_type":"code","source":["results_[np.argsort(results_[:, 2]), :]"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"EsPKZssTU9rL","executionInfo":{"status":"ok","timestamp":1665416489598,"user_tz":240,"elapsed":9,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"}},"outputId":"20ac6507-3c18-4ad8-d73f-11e2f1e64bbc"},"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([['Adam', 'Sullivan', '-27.689796', '-28.539173'],\n","       ['Moose', 'Tomasiello', '-27.813456', '-47.346333'],\n","       ['Carlos', 'Loza', '-27.927849', '-35.930717'],\n","       ['Eric', 'Campbell', '-27.960735', '-27.948498'],\n","       ['Kelsey', 'McDonald', '-28.226707', '-31.136759'],\n","       ['David', 'Alfego', '-28.721024', '-39.18271'],\n","       ['Alvin', 'Koontz', '-28.796902', '-46.784615'],\n","       ['Marc', 'Duchatelier', '-28.89674', '-65.417206'],\n","       ['Ahmed', 'Tawfik', '-28.935593', '-65.328156'],\n","       ['Lax', 'Iyer', '-29.118505', '-36.347607'],\n","       ['Brian', 'Munson', '-29.464714', '-37.82363'],\n","       ['Cassidy', 'Konzelman', '-29.72972', '-47.544243'],\n","       ['William', 'Ginn', '-29.802353', '-38.848503'],\n","       ['Matt', 'Robinson', '-30.136854', '-30.245296'],\n","       ['Walter', 'Jessen', '-32.531944', '-38.93853'],\n","       ['Pingsha', 'Hu', '-36.683655', '-29.627468'],\n","       ['Xia', 'Cao', '-36.73168', '-34.20746'],\n","       ['Xiaodong', 'Zhu', '-39.96691', '-30.180828'],\n","       ['Moloy', 'Chatterjee', '-43.389526', '-44.164116'],\n","       ['Yiqiao', 'Yin', '-49.851707', '-28.030106'],\n","       ['Priyanshu', 'Sinha', '-55.41825', '-43.768623'],\n","       ['Hemanth', 'Reddy', '-58.88199', '-32.675293']], dtype='<U32')"]},"metadata":{},"execution_count":11}]}]}

docs/notebooks/ex11b - transformers.ipynb

@@ -0,0 +1,1856 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "accelerator": "GPU",
+    "widgets": {
+      "application/vnd.jupyter.widget-state+json": {
+        "a366daa8a5724ef5911a78e1e9ffd39c": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HBoxModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HBoxModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HBoxView",
+            "box_style": "",
+            "children": [
+              "IPY_MODEL_a4fc66940fee41c890cd1365ab09bb9c",
+              "IPY_MODEL_80ba5dc94bb241728e86a8650e2d72b8",
+              "IPY_MODEL_b34c9f052d334133a598c8dc3b09659d"
+            ],
+            "layout": "IPY_MODEL_f169dbb09d80415fb6b396bb3b373faf"
+          }
+        },
+        "a4fc66940fee41c890cd1365ab09bb9c": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_551f86c4cac14e4898f9a31642ffce03",
+            "placeholder": "​",
+            "style": "IPY_MODEL_1c2664c6ddcd480381b993e49fd4471f",
+            "value": "Downloading (…)olve/main/source.spm: 100%"
+          }
+        },
+        "80ba5dc94bb241728e86a8650e2d72b8": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "FloatProgressModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "FloatProgressModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "ProgressView",
+            "bar_style": "success",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_8ca1ea420b3d4537a83e7d20b7f0cd3c",
+            "max": 802397,
+            "min": 0,
+            "orientation": "horizontal",
+            "style": "IPY_MODEL_a4802e67d79d41a6a138bce8391df68d",
+            "value": 802397
+          }
+        },
+        "b34c9f052d334133a598c8dc3b09659d": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_786177d5050e4d42a057833c3396e9f2",
+            "placeholder": "​",
+            "style": "IPY_MODEL_7ef46cc8b82742e38b7881de1298119d",
+            "value": " 802k/802k [00:00<00:00, 3.88MB/s]"
+          }
+        },
+        "f169dbb09d80415fb6b396bb3b373faf": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "551f86c4cac14e4898f9a31642ffce03": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "1c2664c6ddcd480381b993e49fd4471f": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "8ca1ea420b3d4537a83e7d20b7f0cd3c": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "a4802e67d79d41a6a138bce8391df68d": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "ProgressStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "ProgressStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "bar_color": null,
+            "description_width": ""
+          }
+        },
+        "786177d5050e4d42a057833c3396e9f2": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "7ef46cc8b82742e38b7881de1298119d": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "d23405a3d6ac4ed4b8cd23f2f52616cd": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HBoxModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HBoxModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HBoxView",
+            "box_style": "",
+            "children": [
+              "IPY_MODEL_61dd52bb0c5f4f1899552afd37d0e95e",
+              "IPY_MODEL_6980a6cf10e543a78a3942da0574e7da",
+              "IPY_MODEL_5edf97ce410d45f4a982464b5de7c4ca"
+            ],
+            "layout": "IPY_MODEL_3d441fccf07148c194e76c3567b50338"
+          }
+        },
+        "61dd52bb0c5f4f1899552afd37d0e95e": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_5f43a463a28c4b62829381e9c30a173d",
+            "placeholder": "​",
+            "style": "IPY_MODEL_5ea7db53cb9e4cb88270e189da94649e",
+            "value": "Downloading (…)olve/main/target.spm: 100%"
+          }
+        },
+        "6980a6cf10e543a78a3942da0574e7da": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "FloatProgressModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "FloatProgressModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "ProgressView",
+            "bar_style": "success",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_21082a6a1fdd4170a78f8d56fca4399e",
+            "max": 778395,
+            "min": 0,
+            "orientation": "horizontal",
+            "style": "IPY_MODEL_56689cfe77064e8b973c80389eb1f6a1",
+            "value": 778395
+          }
+        },
+        "5edf97ce410d45f4a982464b5de7c4ca": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_1f0d105b89c345ada3e6c59fbf5be00c",
+            "placeholder": "​",
+            "style": "IPY_MODEL_ba4e01ebbfb54b0f98efa0d9a0735df4",
+            "value": " 778k/778k [00:00<00:00, 3.91MB/s]"
+          }
+        },
+        "3d441fccf07148c194e76c3567b50338": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "5f43a463a28c4b62829381e9c30a173d": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "5ea7db53cb9e4cb88270e189da94649e": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "21082a6a1fdd4170a78f8d56fca4399e": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "56689cfe77064e8b973c80389eb1f6a1": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "ProgressStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "ProgressStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "bar_color": null,
+            "description_width": ""
+          }
+        },
+        "1f0d105b89c345ada3e6c59fbf5be00c": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "ba4e01ebbfb54b0f98efa0d9a0735df4": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "d1f0d98e85f64836b6e21896ab342de6": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HBoxModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HBoxModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HBoxView",
+            "box_style": "",
+            "children": [
+              "IPY_MODEL_3a1146da456345a0ae2d4f6e8c27d903",
+              "IPY_MODEL_ab9b616aa5e3457b9875ce70b614ddcf",
+              "IPY_MODEL_02b247fc179643ec9aca8321855aab94"
+            ],
+            "layout": "IPY_MODEL_59f466e226bc41a8ac1ea878c7b7be1d"
+          }
+        },
+        "3a1146da456345a0ae2d4f6e8c27d903": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_46c6aab6997548cba550a8818a677748",
+            "placeholder": "​",
+            "style": "IPY_MODEL_8c76f99347134249ac888eaee23e2ad8",
+            "value": "Downloading (…)olve/main/vocab.json: 100%"
+          }
+        },
+        "ab9b616aa5e3457b9875ce70b614ddcf": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "FloatProgressModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "FloatProgressModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "ProgressView",
+            "bar_style": "success",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_5fc912b1da2f48b69de578f5015b82f9",
+            "max": 1339166,
+            "min": 0,
+            "orientation": "horizontal",
+            "style": "IPY_MODEL_e7860a8b6b3943948d9b0928d4864744",
+            "value": 1339166
+          }
+        },
+        "02b247fc179643ec9aca8321855aab94": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_fda43e7cbcf1460cbaff8e1666e2a72e",
+            "placeholder": "​",
+            "style": "IPY_MODEL_b3a4fe4f70054a9c8e2faef9644388db",
+            "value": " 1.34M/1.34M [00:00<00:00, 6.47MB/s]"
+          }
+        },
+        "59f466e226bc41a8ac1ea878c7b7be1d": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "46c6aab6997548cba550a8818a677748": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "8c76f99347134249ac888eaee23e2ad8": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "5fc912b1da2f48b69de578f5015b82f9": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "e7860a8b6b3943948d9b0928d4864744": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "ProgressStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "ProgressStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "bar_color": null,
+            "description_width": ""
+          }
+        },
+        "fda43e7cbcf1460cbaff8e1666e2a72e": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "b3a4fe4f70054a9c8e2faef9644388db": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        }
+      }
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## **INITIALIZATION:**\n",
+        "- I use these three lines of code on top of my each notebooks because it will help to prevent any problems while reloading the same project. And the third line of code helps to make visualization within the notebook."
+      ],
+      "metadata": {
+        "id": "s_rod7u5Z3Bk"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "7-c8RRG5XCK7"
+      },
+      "outputs": [],
+      "source": [
+        "#@ INITIALIZATION: \n",
+        "%reload_ext autoreload\n",
+        "%autoreload 2\n",
+        "%matplotlib inline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## **LIBRARIES AND DEPENDENCIES:**\n",
+        "- I have downloaded all the libraries and dependencies required for the project in one particular cell.\n",
+        "- Make sure to install `transformers[sentencepiece]` in order to use `pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-fr-en\")`, e.g. language translation task"
+      ],
+      "metadata": {
+        "id": "n9I8VhyfaBr7"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ INSTALLING DEPENDENCIES: UNCOMMENT BELOW: \n",
+        "!pip install transformers[sentencepiece]"
+      ],
+      "metadata": {
+        "id": "6xlT0WUMZ-62",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "outputId": "6c4a497f-1b01-4956-c2ae-5ea1bc9787be"
+      },
+      "execution_count": 2,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+            "Requirement already satisfied: transformers[sentencepiece] in /usr/local/lib/python3.9/dist-packages (4.27.4)\n",
+            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (1.22.4)\n",
+            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (0.13.3)\n",
+            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (2022.10.31)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (2.27.1)\n",
+            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (4.65.0)\n",
+            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (3.10.7)\n",
+            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (23.0)\n",
+            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (0.13.4)\n",
+            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (6.0)\n",
+            "Requirement already satisfied: protobuf<=3.20.2 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (3.20.2)\n",
+            "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.9/dist-packages (from transformers[sentencepiece]) (0.1.97)\n",
+            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers[sentencepiece]) (4.5.0)\n",
+            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->transformers[sentencepiece]) (2.0.12)\n",
+            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->transformers[sentencepiece]) (1.26.15)\n",
+            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->transformers[sentencepiece]) (2022.12.7)\n",
+            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->transformers[sentencepiece]) (3.4)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ DOWNLOADING LIBRARIES AND DEPENDENCIES:\n",
+        "import transformers\n",
+        "from transformers import pipeline"
+      ],
+      "metadata": {
+        "id": "NOL1UW_vaWtG"
+      },
+      "execution_count": 3,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## **Natural Language Processing**\n",
+        "\n",
+        "- **NLP** is a field of linguistics and machine learning focused on understanding everything related to human language. The aim of **NLP** tasks is not only to understand single words individually, but to be able to understand the context of those words. "
+      ],
+      "metadata": {
+        "id": "CECqmYGOawxR"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **SENTIMENT ANALYSIS:**\n",
+        "\n",
+        "The line of code `classifier = pipeline(\"sentiment-analysis\")` creates a sentiment analysis classifier using the Hugging Face Transformers library.\n",
+        "\n",
+        "The `pipeline()` function is a convenience method provided by the Transformers library that allows developers to quickly instantiate pre-trained models for various natural language processing (NLP) tasks, including sentiment analysis.\n",
+        "\n",
+        "In this case, the `pipeline()` function is used to create a `sentiment-analysis` pipeline, which is a pre-configured pipeline that includes a pre-trained transformer model for sentiment analysis.\n",
+        "\n",
+        "The resulting classifier object is an instance of the pipeline class, which has a convenient method called `__call__()` that allows you to use the pre-trained sentiment analysis model to analyze the sentiment of input text.\n",
+        "\n",
+        "Overall, this line of code provides a quick and easy way to create a pre-trained sentiment analysis model for use in NLP applications."
+      ],
+      "metadata": {
+        "id": "RkznUFUOcY3F"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **WHAT MODEL**\n",
+        "\n",
+        "The specific model being used by the sentiment analysis pipeline created by the code `classifier = pipeline(\"sentiment-analysis\")` depends on the default model specified by the Hugging Face Transformers library at the time the code is executed.\n",
+        "\n",
+        "However, by default, the sentiment analysis pipeline uses the pre-trained DistilBERT model, which is a smaller, faster, and lighter version of the popular BERT (Bidirectional Encoder Representations from Transformers) model. The DistilBERT model was introduced in the paper \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\" by Sanh et al. in 2020.\n",
+        "\n",
+        "The DistilBERT model was trained on a masked language modeling objective, which involves predicting masked words in text, and fine-tuned on a sentiment analysis task using the IMDb dataset, which contains movie reviews labeled as positive or negative.\n",
+        "\n",
+        "While the specific implementation of the sentiment analysis pipeline created by the code `classifier = pipeline(\"sentiment-analysis\")` may not have a dedicated research paper, the underlying pre-trained DistilBERT model has been extensively evaluated and published in the aforementioned paper. Additionally, the Hugging Face Transformers library provides extensive documentation and examples of how to use the model and pipeline for various NLP tasks.\n",
+        "\n",
+        "\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "N7rmfAbqTTAA"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF SENTIMENT ANALYSIS PIPELINE:\n",
+        "classifier = pipeline(\"sentiment-analysis\")                                 # Initializing Classifier Object. \n",
+        "classifier(\"I've started the HuggingFace course which fascinates me.\")      # Inspecting Sentiment. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "ibYIi6HBatZn",
+        "outputId": "8942c5ac-91e4-48e6-f2a4-8f9bdfc92a10"
+      },
+      "execution_count": 4,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english and revision af0f99b (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'label': 'POSITIVE', 'score': 0.9997233748435974}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 4
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF SENTIMENT ANALYSIS PIPELINE: MULTIPLE:\n",
+        "classifier(\n",
+        "    [\n",
+        "        \"I've started the HuggingFace course which fascinates me.\",\n",
+        "        \"I will no longer read it's documentation.\",\n",
+        "        \"I think the course is awesome!\"\n",
+        "    ]\n",
+        ") # Inspecting Sentiment. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "pYpdpAjXc4dH",
+        "outputId": "952ede02-025c-4cb6-ff0e-c4a0b117681f"
+      },
+      "execution_count": 5,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'label': 'POSITIVE', 'score': 0.9997233748435974},\n",
+              " {'label': 'NEGATIVE', 'score': 0.9994686245918274},\n",
+              " {'label': 'POSITIVE', 'score': 0.9998465776443481}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 5
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "#### **Pipelines**\n",
+        "\n",
+        "- The three main steps involved when we pass some text to a `pipeline` are: \n",
+        "    - The text is preprocessed into a format the model can understand.\n",
+        "    - The preprocessed inputs are passed to the model. \n",
+        "    - The predictions of the model are post-processed, so we can make sense of them. "
+      ],
+      "metadata": {
+        "id": "bp4Yx3MffEw6"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **ZERO-SHOT CLASSIFICATION:**\n",
+        "\n",
+        "- The `zero-shot-classification` pipeline is very powerful, as it allows us to specify which labels to use for the classification, so we don't have to rely on the labels of the pretrained model. This `pipeline` is called `zero-shot` because we don't need to fine-tune the model on our data to use it. It can directly return probability scores for any list of labels we want. \n",
+        "\n",
+        "The zero-shot-classification pipeline is a pre-configured pipeline that uses a pre-trained transformer model for classification tasks. Unlike traditional classification pipelines, the zero-shot classification pipeline can classify input text based on classes that are not explicitly defined in the training data.\n",
+        "\n",
+        "The pipeline works by encoding the input text and a list of candidate labels using a pre-trained transformer model. The candidate labels represent the classes that the input text may belong to, but they are not used to train the model. Instead, the model predicts the likelihood of the input text belonging to each of the candidate labels by comparing the encoded input text with the encoded candidate labels.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "natPtwAFgD9T"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF ZERO SHOT CLASSIFICATION PIPELINE: \n",
+        "classifier = pipeline(\"zero-shot-classification\")                           # Initializing Classifier Object. \n",
+        "classifier(\"This is a course about Transformers library\", \n",
+        "           candidate_labels=[\"education\", \"health\", \"programming\"])         # Inspecting Classification. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "Wt8ilH9VeW1A",
+        "outputId": "c6a08e9d-4258-472b-ff85-382107a15181"
+      },
+      "execution_count": 6,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to facebook/bart-large-mnli and revision c626438 (https://huggingface.co/facebook/bart-large-mnli).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "{'sequence': 'This is a course about Transformers library',\n",
+              " 'labels': ['education', 'programming', 'health'],\n",
+              " 'scores': [0.6901501417160034, 0.1908414661884308, 0.1190083846449852]}"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 6
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **TEXT GENERATION:**\n",
+        "\n",
+        "- The main idea here is that when we provide a prompt and the model will auto-complete it by generating the remaining text. "
+      ],
+      "metadata": {
+        "id": "HSCQbBQdylz6"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF TEXT GENERATION PIPELINE: \n",
+        "generator = pipeline(\"text-generation\")                                 # Initializing Generator Object. \n",
+        "generator(\"In this course, you will learn to\")                          # Inspecting Generated Text. "
+      ],
+      "metadata": {
+        "id": "GS1r-kAQirUE",
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "outputId": "712b29f9-d1c9-4945-cdc2-65870714ebda"
+      },
+      "execution_count": 7,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to gpt2 and revision 6c0e608 (https://huggingface.co/gpt2).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n",
+            "/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py:1201: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)\n",
+            "  warnings.warn(\n",
+            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
+            "/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py:1288: UserWarning: Using `max_length`'s default (50) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n",
+            "  warnings.warn(\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'generated_text': 'In this course, you will learn to take on each new project with real-world use cases, making it easier to follow through with your concepts. Students will go through what you need to work with in detail before you start using your new project.'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 7
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF TEXT GENERATION PIPELINE: \n",
+        "generator(\"I like python because\", num_return_sequences=2, max_length=15)     # Inspecting Generated Sequences. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "YeJPE8k5zjPL",
+        "outputId": "f637e744-863b-4dd2-baf5-98e062707c2e"
+      },
+      "execution_count": 8,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'generated_text': 'I like python because it\\'s open source,\" said Cipriano.'},\n",
+              " {'generated_text': 'I like python because of the interface. For that I found myself getting errors'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 8
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF TEXT GENERATION PIPELINE: DISTILGPT2:\n",
+        "generator = pipeline(\"text-generation\", model=\"distilgpt2\")                   # Initializing Generator Object. \n",
+        "generator(\"I want to be a programmer so that\", \n",
+        "          num_return_sequences=2, max_length=30)                              # Inspecting Generated Sequences. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "i8tULAYiz_1g",
+        "outputId": "b868156f-13c4-4c65-8b62-4eec2218bb01"
+      },
+      "execution_count": 9,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'generated_text': 'I want to be a programmer so that this will be used. It\\u202cs a real hard work to deal with. That\\u202cs a'},\n",
+              " {'generated_text': 'I want to be a programmer so that you can do anything you want, but with such a great deal of resources and time, I would love to'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 9
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **MASK FILLING:**\n",
+        "\n",
+        "- The idea of this task is to fill in the blanks in a given text. "
+      ],
+      "metadata": {
+        "id": "J0zsBIMFCwDw"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF MASK FILLING PIPELINE:\n",
+        "unmasker = pipeline(\"fill-mask\")                                        # Initilizing Mask Filling Object. \n",
+        "unmasker(\"This course will teach you all about <mask> models.\", \n",
+        "         top_k=2)                                                       # Inspecting Mask Token."
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "4c1UCi9C2kI6",
+        "outputId": "9263c813-0a48-409a-e026-7c9bcc3e3172"
+      },
+      "execution_count": 10,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to distilroberta-base and revision ec58a5b (https://huggingface.co/distilroberta-base).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'score': 0.19619806110858917,\n",
+              "  'token': 30412,\n",
+              "  'token_str': ' mathematical',\n",
+              "  'sequence': 'This course will teach you all about mathematical models.'},\n",
+              " {'score': 0.04052723944187164,\n",
+              "  'token': 38163,\n",
+              "  'token_str': ' computational',\n",
+              "  'sequence': 'This course will teach you all about computational models.'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 10
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF MASK FILLING PIPELINE:\n",
+        "unmasker = pipeline(\"fill-mask\", model=\"bert-base-cased\")               # Initilizing Mask Filling Object. \n",
+        "unmasker(\"This course will teach you all about [MASK] models.\", \n",
+        "         top_k=2)                                                       # Inspecting Mask Token."
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "kjvAPF2zHCgt",
+        "outputId": "442e5075-bc75-4efe-a079-42abac41802d"
+      },
+      "execution_count": 11,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n",
+            "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
+            "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'score': 0.25963184237480164,\n",
+              "  'token': 1648,\n",
+              "  'token_str': 'role',\n",
+              "  'sequence': 'This course will teach you all about role models.'},\n",
+              " {'score': 0.09427252411842346,\n",
+              "  'token': 1103,\n",
+              "  'token_str': 'the',\n",
+              "  'sequence': 'This course will teach you all about the models.'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 11
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **NAMED ENTITY RECOGNITION:**\n",
+        "\n",
+        "- **NER** is a task where the model has to find which parts of the input text correspond to entities such as persons, locations, or organizations. "
+      ],
+      "metadata": {
+        "id": "T1IuXZA-Fdrd"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF NER PIPELINE:\n",
+        "ner = pipeline(\"ner\", grouped_entities=True)                           # Initializing NER Object. \n",
+        "ner(\"I am Thinam Tamang and I am from Kathmandu, Nepal.\")              # Inspecting Entities. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "xkHKDYCPFEZT",
+        "outputId": "2f454b99-6a6b-4f74-88a8-274c9c3321ae"
+      },
+      "execution_count": 12,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to dbmdz/bert-large-cased-finetuned-conll03-english and revision f2482bf (https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n",
+            "/usr/local/lib/python3.9/dist-packages/transformers/pipelines/token_classification.py:157: UserWarning: `grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy=\"simple\"` instead.\n",
+            "  warnings.warn(\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'entity_group': 'PER',\n",
+              "  'score': 0.9967239,\n",
+              "  'word': 'Thinam Tamang',\n",
+              "  'start': 5,\n",
+              "  'end': 18},\n",
+              " {'entity_group': 'LOC',\n",
+              "  'score': 0.9990222,\n",
+              "  'word': 'Kathmandu',\n",
+              "  'start': 33,\n",
+              "  'end': 42},\n",
+              " {'entity_group': 'LOC',\n",
+              "  'score': 0.9997008,\n",
+              "  'word': 'Nepal',\n",
+              "  'start': 44,\n",
+              "  'end': 49}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 12
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "###**QUESTION ANSWERING:**\n",
+        "\n",
+        "- The `question-answering` pipeline answers questions using information from a given context. "
+      ],
+      "metadata": {
+        "id": "G85OQxxHIBQp"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF QUESTION ANSWERING PIPELINE: \n",
+        "question_answerer = pipeline(\"question-answering\")                          # Initialization. \n",
+        "question_answerer(\n",
+        "    question=\"What is my name?\",\n",
+        "    context=\"I am Thinam from Nepal.\")                                      # Inspecting Answer. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "F4q1loPYH3pB",
+        "outputId": "a3bab9ef-d0e3-4280-c813-f8a969f02e1a"
+      },
+      "execution_count": 13,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to distilbert-base-cased-distilled-squad and revision 626af31 (https://huggingface.co/distilbert-base-cased-distilled-squad).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "{'score': 0.8733125925064087, 'start': 5, 'end': 11, 'answer': 'Thinam'}"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 13
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **SUMMARIZATION:**\n",
+        "\n",
+        "- Summarization is the task of reducing a text into a shorter text while keeping all or most of the important aspects referenced in the text. "
+      ],
+      "metadata": {
+        "id": "uxANYti2Jzen"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF SUMMARIZATION PIPELINE:\n",
+        "summarizer = pipeline(\"summarization\")                                                  # Initializing Summarizer Object. \n",
+        "summarizer(\n",
+        "    \"\"\"\n",
+        "    America has changed dramatically during recent years. Not only has the number of \n",
+        "    graduates in traditional engineering disciplines such as mechanical, civil, \n",
+        "    electrical, chemical, and aeronautical engineering declined, but in most of \n",
+        "    the premier American universities engineering curricula now concentrate on \n",
+        "    and encourage largely the study of engineering science. As a result, there \n",
+        "    are declining offerings in engineering subjects dealing with infrastructure, \n",
+        "    the environment, and related issues, and greater concentration on high \n",
+        "    technology subjects, largely supporting increasingly complex scientific \n",
+        "    developments. While the latter is important, it should not be at the expense \n",
+        "    of more traditional engineering.\n",
+        "\n",
+        "    Rapidly developing economies such as China and India, as well as other \n",
+        "    industrial countries in Europe and Asia, continue to encourage and advance \n",
+        "    the teaching of engineering. Both China and India, respectively, graduate \n",
+        "    six and eight times as many traditional engineers as does the United States. \n",
+        "    Other industrial countries at minimum maintain their output, while America \n",
+        "    suffers an increasingly serious decline in the number of engineering graduates \n",
+        "    and a lack of well-educated engineers.\n",
+        "\"\"\"\n",
+        ")                                                                                         # Inspecting Summarized Text. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "2-jGKeWkJeVa",
+        "outputId": "b79e1d03-8b39-45d1-b098-fd22efce7fab"
+      },
+      "execution_count": 14,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "No model was supplied, defaulted to sshleifer/distilbart-cnn-12-6 and revision a4f8f3e (https://huggingface.co/sshleifer/distilbart-cnn-12-6).\n",
+            "Using a pipeline without specifying a model name and revision in production is not recommended.\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'summary_text': ' America has changed dramatically during recent years . The number of engineering graduates in the U.S. has declined in traditional engineering disciplines such as mechanical, civil,    electrical, chemical, and aeronautical engineering . Rapidly developing economies such as China and India continue to encourage and advance the teaching of engineering .'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 14
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **TRANSLATION:**"
+      ],
+      "metadata": {
+        "id": "JukLFMN6LGRq"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "#@ IMPLEMENTATION OF TRANSLATION PIPELINE: \n",
+        "translator = pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-fr-en\")        # Initializing Translator Object. \n",
+        "translator(\"Ce cours est produit par Hugging Face.\")                            # Inspecting Translation. "
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 168,
+          "referenced_widgets": [
+            "a366daa8a5724ef5911a78e1e9ffd39c",
+            "a4fc66940fee41c890cd1365ab09bb9c",
+            "80ba5dc94bb241728e86a8650e2d72b8",
+            "b34c9f052d334133a598c8dc3b09659d",
+            "f169dbb09d80415fb6b396bb3b373faf",
+            "551f86c4cac14e4898f9a31642ffce03",
+            "1c2664c6ddcd480381b993e49fd4471f",
+            "8ca1ea420b3d4537a83e7d20b7f0cd3c",
+            "a4802e67d79d41a6a138bce8391df68d",
+            "786177d5050e4d42a057833c3396e9f2",
+            "7ef46cc8b82742e38b7881de1298119d",
+            "d23405a3d6ac4ed4b8cd23f2f52616cd",
+            "61dd52bb0c5f4f1899552afd37d0e95e",
+            "6980a6cf10e543a78a3942da0574e7da",
+            "5edf97ce410d45f4a982464b5de7c4ca",
+            "3d441fccf07148c194e76c3567b50338",
+            "5f43a463a28c4b62829381e9c30a173d",
+            "5ea7db53cb9e4cb88270e189da94649e",
+            "21082a6a1fdd4170a78f8d56fca4399e",
+            "56689cfe77064e8b973c80389eb1f6a1",
+            "1f0d105b89c345ada3e6c59fbf5be00c",
+            "ba4e01ebbfb54b0f98efa0d9a0735df4",
+            "d1f0d98e85f64836b6e21896ab342de6",
+            "3a1146da456345a0ae2d4f6e8c27d903",
+            "ab9b616aa5e3457b9875ce70b614ddcf",
+            "02b247fc179643ec9aca8321855aab94",
+            "59f466e226bc41a8ac1ea878c7b7be1d",
+            "46c6aab6997548cba550a8818a677748",
+            "8c76f99347134249ac888eaee23e2ad8",
+            "5fc912b1da2f48b69de578f5015b82f9",
+            "e7860a8b6b3943948d9b0928d4864744",
+            "fda43e7cbcf1460cbaff8e1666e2a72e",
+            "b3a4fe4f70054a9c8e2faef9644388db"
+          ]
+        },
+        "id": "aEcD0rquKsMx",
+        "outputId": "c8deaac9-6cfe-4a8a-ee48-be18576b8f84"
+      },
+      "execution_count": 15,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "Downloading (…)olve/main/source.spm:   0%|          | 0.00/802k [00:00<?, ?B/s]"
+            ],
+            "application/vnd.jupyter.widget-view+json": {
+              "version_major": 2,
+              "version_minor": 0,
+              "model_id": "a366daa8a5724ef5911a78e1e9ffd39c"
+            }
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "Downloading (…)olve/main/target.spm:   0%|          | 0.00/778k [00:00<?, ?B/s]"
+            ],
+            "application/vnd.jupyter.widget-view+json": {
+              "version_major": 2,
+              "version_minor": 0,
+              "model_id": "d23405a3d6ac4ed4b8cd23f2f52616cd"
+            }
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "Downloading (…)olve/main/vocab.json:   0%|          | 0.00/1.34M [00:00<?, ?B/s]"
+            ],
+            "application/vnd.jupyter.widget-view+json": {
+              "version_major": 2,
+              "version_minor": 0,
+              "model_id": "d1f0d98e85f64836b6e21896ab342de6"
+            }
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "/usr/local/lib/python3.9/dist-packages/transformers/models/marian/tokenization_marian.py:194: UserWarning: Recommended: pip install sacremoses.\n",
+            "  warnings.warn(\"Recommended: pip install sacremoses.\")\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "[{'translation_text': 'This course is produced by Hugging Face.'}]"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 15
+        }
+      ]
+    }
+  ]
+}

docs/notebooks/ex11d - text encoder using transformers.ipynb

@@ -0,0 +1,1533 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    },
+    "widgets": {
+      "application/vnd.jupyter.widget-state+json": {
+        "973cd0e6d70c4fd8bf77b7957f4b1781": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HBoxModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HBoxModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HBoxView",
+            "box_style": "",
+            "children": [
+              "IPY_MODEL_741a005558ec4b9381cf2d5771fb3be7",
+              "IPY_MODEL_80e69e80415d4f73b1e611731f2ae436",
+              "IPY_MODEL_31fe773494ac4cf6bba8cf280714cc4e"
+            ],
+            "layout": "IPY_MODEL_94abfe4dd71844cf8636926b4b90c897"
+          }
+        },
+        "741a005558ec4b9381cf2d5771fb3be7": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_fe2ad417e96a4640a0dbfcc4a891a194",
+            "placeholder": "​",
+            "style": "IPY_MODEL_e0e92d27ef224cb28871007ea4c450df",
+            "value": "Downloading (…)solve/main/vocab.txt: 100%"
+          }
+        },
+        "80e69e80415d4f73b1e611731f2ae436": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "FloatProgressModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "FloatProgressModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "ProgressView",
+            "bar_style": "success",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_5b1fd20cb8e44abe9dd5f7e5635f5cfc",
+            "max": 231508,
+            "min": 0,
+            "orientation": "horizontal",
+            "style": "IPY_MODEL_9c7a78b88df44f629eaa5c58589bffee",
+            "value": 231508
+          }
+        },
+        "31fe773494ac4cf6bba8cf280714cc4e": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_ad7eeb030efa4b2b893b3bbd5c906d34",
+            "placeholder": "​",
+            "style": "IPY_MODEL_46bef8ac32964e87972dad9d5f5e6c42",
+            "value": " 232k/232k [00:00<00:00, 1.78MB/s]"
+          }
+        },
+        "94abfe4dd71844cf8636926b4b90c897": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "fe2ad417e96a4640a0dbfcc4a891a194": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "e0e92d27ef224cb28871007ea4c450df": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "5b1fd20cb8e44abe9dd5f7e5635f5cfc": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "9c7a78b88df44f629eaa5c58589bffee": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "ProgressStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "ProgressStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "bar_color": null,
+            "description_width": ""
+          }
+        },
+        "ad7eeb030efa4b2b893b3bbd5c906d34": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "46bef8ac32964e87972dad9d5f5e6c42": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "cecff255767f46af827aed7d2c9e7700": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HBoxModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HBoxModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HBoxView",
+            "box_style": "",
+            "children": [
+              "IPY_MODEL_dab1d13daabb4e4286b1fa72ee058ff9",
+              "IPY_MODEL_2b859150fcce447caf1ceb59cd28e8b1",
+              "IPY_MODEL_5f9c860b9bd34cce9ffafd3d9ba7b7f8"
+            ],
+            "layout": "IPY_MODEL_00eea702dfda42258a81193222fc6bdd"
+          }
+        },
+        "dab1d13daabb4e4286b1fa72ee058ff9": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_8143eb3cf4b24c58b4a661ecd7263441",
+            "placeholder": "​",
+            "style": "IPY_MODEL_3903fce6beab4707b76b98d27ec655b6",
+            "value": "Downloading (…)okenizer_config.json: 100%"
+          }
+        },
+        "2b859150fcce447caf1ceb59cd28e8b1": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "FloatProgressModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "FloatProgressModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "ProgressView",
+            "bar_style": "success",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_f8c8235f18fb4ed285f22b1b1aef7c4c",
+            "max": 28,
+            "min": 0,
+            "orientation": "horizontal",
+            "style": "IPY_MODEL_44ee255332424b7c9ba9cd09e092af77",
+            "value": 28
+          }
+        },
+        "5f9c860b9bd34cce9ffafd3d9ba7b7f8": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_a52444f1de5a4b7caaef280ebac9885f",
+            "placeholder": "​",
+            "style": "IPY_MODEL_0e27568aa5e245bc85138fc4523c6abd",
+            "value": " 28.0/28.0 [00:00<00:00, 504B/s]"
+          }
+        },
+        "00eea702dfda42258a81193222fc6bdd": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "8143eb3cf4b24c58b4a661ecd7263441": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "3903fce6beab4707b76b98d27ec655b6": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "f8c8235f18fb4ed285f22b1b1aef7c4c": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "44ee255332424b7c9ba9cd09e092af77": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "ProgressStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "ProgressStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "bar_color": null,
+            "description_width": ""
+          }
+        },
+        "a52444f1de5a4b7caaef280ebac9885f": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "0e27568aa5e245bc85138fc4523c6abd": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "9c17e91736eb43b9b26d8dfe2cf78519": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HBoxModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HBoxModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HBoxView",
+            "box_style": "",
+            "children": [
+              "IPY_MODEL_e2aa7bb9516e427eb3370c09dc3932ee",
+              "IPY_MODEL_3dc6222ba6bc4f7b9b20c1ec9bb69876",
+              "IPY_MODEL_96be2e330b2446828c4f006dec4db56e"
+            ],
+            "layout": "IPY_MODEL_e2dd76912b434e4d8c384fc5fdbc47be"
+          }
+        },
+        "e2aa7bb9516e427eb3370c09dc3932ee": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_449f3587ecd0498ea33a57d68eb7e8b4",
+            "placeholder": "​",
+            "style": "IPY_MODEL_61260c8f08dd47e497c7954acbe067b9",
+            "value": "Downloading (…)lve/main/config.json: 100%"
+          }
+        },
+        "3dc6222ba6bc4f7b9b20c1ec9bb69876": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "FloatProgressModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "FloatProgressModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "ProgressView",
+            "bar_style": "success",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_071376d49f954f1fbbd117e84971b0ca",
+            "max": 570,
+            "min": 0,
+            "orientation": "horizontal",
+            "style": "IPY_MODEL_40b3f219ee5d48819a2471db0410a94c",
+            "value": 570
+          }
+        },
+        "96be2e330b2446828c4f006dec4db56e": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "HTMLModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_dom_classes": [],
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "HTMLModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/controls",
+            "_view_module_version": "1.5.0",
+            "_view_name": "HTMLView",
+            "description": "",
+            "description_tooltip": null,
+            "layout": "IPY_MODEL_2fb8547dc20e477aacfcc7f0172cb000",
+            "placeholder": "​",
+            "style": "IPY_MODEL_675700ceb432426d87e22efabb39ea5a",
+            "value": " 570/570 [00:00<00:00, 17.2kB/s]"
+          }
+        },
+        "e2dd76912b434e4d8c384fc5fdbc47be": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "449f3587ecd0498ea33a57d68eb7e8b4": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "61260c8f08dd47e497c7954acbe067b9": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        },
+        "071376d49f954f1fbbd117e84971b0ca": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "40b3f219ee5d48819a2471db0410a94c": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "ProgressStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "ProgressStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "bar_color": null,
+            "description_width": ""
+          }
+        },
+        "2fb8547dc20e477aacfcc7f0172cb000": {
+          "model_module": "@jupyter-widgets/base",
+          "model_name": "LayoutModel",
+          "model_module_version": "1.2.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/base",
+            "_model_module_version": "1.2.0",
+            "_model_name": "LayoutModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "LayoutView",
+            "align_content": null,
+            "align_items": null,
+            "align_self": null,
+            "border": null,
+            "bottom": null,
+            "display": null,
+            "flex": null,
+            "flex_flow": null,
+            "grid_area": null,
+            "grid_auto_columns": null,
+            "grid_auto_flow": null,
+            "grid_auto_rows": null,
+            "grid_column": null,
+            "grid_gap": null,
+            "grid_row": null,
+            "grid_template_areas": null,
+            "grid_template_columns": null,
+            "grid_template_rows": null,
+            "height": null,
+            "justify_content": null,
+            "justify_items": null,
+            "left": null,
+            "margin": null,
+            "max_height": null,
+            "max_width": null,
+            "min_height": null,
+            "min_width": null,
+            "object_fit": null,
+            "object_position": null,
+            "order": null,
+            "overflow": null,
+            "overflow_x": null,
+            "overflow_y": null,
+            "padding": null,
+            "right": null,
+            "top": null,
+            "visibility": null,
+            "width": null
+          }
+        },
+        "675700ceb432426d87e22efabb39ea5a": {
+          "model_module": "@jupyter-widgets/controls",
+          "model_name": "DescriptionStyleModel",
+          "model_module_version": "1.5.0",
+          "state": {
+            "_model_module": "@jupyter-widgets/controls",
+            "_model_module_version": "1.5.0",
+            "_model_name": "DescriptionStyleModel",
+            "_view_count": null,
+            "_view_module": "@jupyter-widgets/base",
+            "_view_module_version": "1.2.0",
+            "_view_name": "StyleView",
+            "description_width": ""
+          }
+        }
+      }
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "In handling language data, try to think of using packages before you code from scratch.\n",
+        "\n",
+        "- *OpenAI*\n",
+        "- *HuggingFace*"
+      ],
+      "metadata": {
+        "id": "TofAx86oAE-d"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!pip install transformers"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "vaIXK5MK9QcN",
+        "outputId": "0036b9da-4398-4a10-8b93-3d565e8266c9"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
+            "Collecting transformers\n",
+            "  Downloading transformers-4.27.4-py3-none-any.whl (6.8 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m28.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.9/dist-packages (from transformers) (1.22.4)\n",
+            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from transformers) (6.0)\n",
+            "Collecting huggingface-hub<1.0,>=0.11.0\n",
+            "  Downloading huggingface_hub-0.13.4-py3-none-any.whl (200 kB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m200.1/200.1 KB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.9/dist-packages (from transformers) (4.65.0)\n",
+            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers) (3.10.7)\n",
+            "Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from transformers) (2.27.1)\n",
+            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers) (2022.10.31)\n",
+            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1\n",
+            "  Downloading tokenizers-0.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
+            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m47.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from transformers) (23.0)\n",
+            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers) (4.5.0)\n",
+            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->transformers) (3.4)\n",
+            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->transformers) (2022.12.7)\n",
+            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->transformers) (1.26.15)\n",
+            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->transformers) (2.0.12)\n",
+            "Installing collected packages: tokenizers, huggingface-hub, transformers\n",
+            "Successfully installed huggingface-hub-0.13.4 tokenizers-0.13.3 transformers-4.27.4\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WSYoFFdU82vh"
+      },
+      "outputs": [],
+      "source": [
+        "import numpy as np\n",
+        "from transformers import BertTokenizer"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def greeting1(name: str) -> str:\n",
+        "    return 'Hello ' + name\n"
+      ],
+      "metadata": {
+        "id": "FyjYUgPg90L9"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "greeting1('dlkfsdlkfj')"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 36
+        },
+        "id": "kgT6Jp0C92rs",
+        "outputId": "b558d343-86c1-4fb2-96b7-0035fa7c37b1"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "'Hello dlkfsdlkfj'"
+            ],
+            "application/vnd.google.colaboratory.intrinsic+json": {
+              "type": "string"
+            }
+          },
+          "metadata": {},
+          "execution_count": 12
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def greeting2(name):\n",
+        "    return 'Hello ' + name\n"
+      ],
+      "metadata": {
+        "id": "N4-9ugpm90nc"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "greeting2"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "rTtEr11599h8",
+        "outputId": "7039cc17-176f-4ca3-98f3-a7c79c8544ce"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "<function __main__.greeting2(name)>"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 10
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "def tokenize_string(x: str) -> np.array:\n",
+        "    # Load the tokenizer\n",
+        "    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
+        "    \n",
+        "    # Tokenize the string\n",
+        "    tokens = tokenizer.encode(x, add_special_tokens=True)\n",
+        "    \n",
+        "    # Convert the tokens to a NumPy array\n",
+        "    token_array = np.array(tokens)\n",
+        "    \n",
+        "    return token_array\n"
+      ],
+      "metadata": {
+        "id": "EFWfh9g59M9q"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "tokenize_string(\"1, this sentiment is negative because the person doesn't like negative growth rate\")"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 148,
+          "referenced_widgets": [
+            "973cd0e6d70c4fd8bf77b7957f4b1781",
+            "741a005558ec4b9381cf2d5771fb3be7",
+            "80e69e80415d4f73b1e611731f2ae436",
+            "31fe773494ac4cf6bba8cf280714cc4e",
+            "94abfe4dd71844cf8636926b4b90c897",
+            "fe2ad417e96a4640a0dbfcc4a891a194",
+            "e0e92d27ef224cb28871007ea4c450df",
+            "5b1fd20cb8e44abe9dd5f7e5635f5cfc",
+            "9c7a78b88df44f629eaa5c58589bffee",
+            "ad7eeb030efa4b2b893b3bbd5c906d34",
+            "46bef8ac32964e87972dad9d5f5e6c42",
+            "cecff255767f46af827aed7d2c9e7700",
+            "dab1d13daabb4e4286b1fa72ee058ff9",
+            "2b859150fcce447caf1ceb59cd28e8b1",
+            "5f9c860b9bd34cce9ffafd3d9ba7b7f8",
+            "00eea702dfda42258a81193222fc6bdd",
+            "8143eb3cf4b24c58b4a661ecd7263441",
+            "3903fce6beab4707b76b98d27ec655b6",
+            "f8c8235f18fb4ed285f22b1b1aef7c4c",
+            "44ee255332424b7c9ba9cd09e092af77",
+            "a52444f1de5a4b7caaef280ebac9885f",
+            "0e27568aa5e245bc85138fc4523c6abd",
+            "9c17e91736eb43b9b26d8dfe2cf78519",
+            "e2aa7bb9516e427eb3370c09dc3932ee",
+            "3dc6222ba6bc4f7b9b20c1ec9bb69876",
+            "96be2e330b2446828c4f006dec4db56e",
+            "e2dd76912b434e4d8c384fc5fdbc47be",
+            "449f3587ecd0498ea33a57d68eb7e8b4",
+            "61260c8f08dd47e497c7954acbe067b9",
+            "071376d49f954f1fbbd117e84971b0ca",
+            "40b3f219ee5d48819a2471db0410a94c",
+            "2fb8547dc20e477aacfcc7f0172cb000",
+            "675700ceb432426d87e22efabb39ea5a"
+          ]
+        },
+        "id": "ZHbMMXpf9Zkx",
+        "outputId": "f0d7af5a-84c4-44e2-cd79-8778a10a44ed"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
+            ],
+            "application/vnd.jupyter.widget-view+json": {
+              "version_major": 2,
+              "version_minor": 0,
+              "model_id": "973cd0e6d70c4fd8bf77b7957f4b1781"
+            }
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "Downloading (…)okenizer_config.json:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
+            ],
+            "application/vnd.jupyter.widget-view+json": {
+              "version_major": 2,
+              "version_minor": 0,
+              "model_id": "cecff255767f46af827aed7d2c9e7700"
+            }
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "Downloading (…)lve/main/config.json:   0%|          | 0.00/570 [00:00<?, ?B/s]"
+            ],
+            "application/vnd.jupyter.widget-view+json": {
+              "version_major": 2,
+              "version_minor": 0,
+              "model_id": "9c17e91736eb43b9b26d8dfe2cf78519"
+            }
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "array([  101,  1015,  1010,  2023, 15792,  2003,  4997,  2138,  1996,\n",
+              "        2711,  2987,  1005,  1056,  2066,  4997,  3930,  3446,   102])"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 14
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# from transformers import BertTokenizer\n",
+        "\n",
+        "# Instantiate a BERT tokenizer object\n",
+        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
+        "\n",
+        "# Get the ID of the [SEP] token\n",
+        "sep_token_id = tokenizer.sep_token_id\n",
+        "sep_token_id"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "rctLhI7Z-cgR",
+        "outputId": "78b2df8f-784f-40ed-d422-ec8b7298652c"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "102"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 15
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# import numpy as np\n",
+        "# from transformers import BertTokenizer\n",
+        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
+        "\n",
+        "def tokenize_string(string: str, max_length: int) -> np.array:\n",
+        "    # Load the tokenizer\n",
+        "    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
+        "    \n",
+        "    # Tokenize the string\n",
+        "    tokens = tokenizer.encode(string, add_special_tokens=True)\n",
+        "    \n",
+        "    # Limit the length of the token array\n",
+        "    if len(tokens) > max_length - 2:\n",
+        "        tokens = tokens[:max_length - 2] + [tokenizer.sep_token_id]\n",
+        "    else:\n",
+        "        tokens += [tokenizer.sep_token_id]\n",
+        "    \n",
+        "    # Pad the token array with zeros\n",
+        "    token_array = pad_sequences([tokens], maxlen=max_length, dtype='int32', padding='post', truncating='post', value=0)\n",
+        "    \n",
+        "    return np.asarray(token_array[0])\n"
+      ],
+      "metadata": {
+        "id": "jf4xYkYC_wEc"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "y1 = tokenize_string(\"1, I don't know this finance.\")\n",
+        "len(y1)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "7735fsfh_z7L",
+        "outputId": "18512fe3-d181-4764-f698-6c7f138f308b"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "128"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 19
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "y2 = tokenize_string(\"1, I don't know this finance. Because the company has negative earnings.\")\n",
+        "len(y2)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "dlcMWWZr_1fv",
+        "outputId": "e345f7d6-9936-41ef-97a1-c617ab2c4af2"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "128"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 20
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import numpy as np\n",
+        "import tensorflow as tf\n",
+        "from transformers import RobertaTokenizer, TFRobertaModel\n",
+        "\n",
+        "def encode_texts(texts: list[str, str, ...]) -> np.array:\n",
+        "    # Load the tokenizer and model\n",
+        "    tokenizer = RobertaTokenizer.from_pretrained('roberta-base')\n",
+        "    model = TFRobertaModel.from_pretrained('roberta-base')\n",
+        "    \n",
+        "    # Tokenize the texts\n",
+        "    tokenized_texts = [tokenizer.encode(text, add_special_tokens=True) for text in texts]\n",
+        "    \n",
+        "    # Pad the tokenized texts to a fixed length\n",
+        "    max_length = max([len(text) for text in tokenized_texts])\n",
+        "    padded_texts = np.array([text + [tokenizer.pad_token_id] * (max_length - len(text)) for text in tokenized_texts])\n",
+        "    \n",
+        "    # Encode the padded texts using the RoBERTa model\n",
+        "    input_ids = tf.constant(padded_texts)\n",
+        "    outputs = model(input_ids)\n",
+        "    vectors = outputs[0][:, 0, :].numpy()\n",
+        "    \n",
+        "    return vectors\n"
+      ],
+      "metadata": {
+        "id": "_HBdyZ3m_5Xo"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "output1 = encode_texts(\n",
+        "    [\n",
+        "        \"0, i don't like this financials\",\n",
+        "        \"1, i don't understand financial data\",\n",
+        "        \"2, i like this company and i think the stock price goes up\"\n",
+        "    ]\n",
+        ")"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "fQFDQzZNC9PA",
+        "outputId": "c596fe16-33d3-4f48-bfa2-3de5c9aa34fd"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "Some layers from the model checkpoint at roberta-base were not used when initializing TFRobertaModel: ['lm_head']\n",
+            "- This IS expected if you are initializing TFRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
+            "- This IS NOT expected if you are initializing TFRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
+            "All the layers of TFRobertaModel were initialized from the model checkpoint at roberta-base.\n",
+            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFRobertaModel for predictions without further training.\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "output1.shape"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "B1cn0As2DUFf",
+        "outputId": "a77179d9-60a5-4af3-dc24-6f14bebb426d"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "(3, 768)"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 26
+        }
+      ]
+    }
+  ]
+}

docs/notebooks/ex11e - attention layer sample.ipynb

@@ -0,0 +1,366 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Source\n",
+        "\n",
+        "- The source code comes from [here](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention).\n",
+        "\n",
+        "- More detailed explanation can be read from [here](https://machinelearningmastery.com/the-luong-attention-mechanism/)."
+      ],
+      "metadata": {
+        "id": "Exu3AGMEkCjL"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Data\n",
+        "\n",
+        "The code snippet is responsible for generating synthetic data to train a neural network model. The `batch_size` variable defines how many examples will be processed at once during training, which is set to 32. The `max_sequence_length` is a hyperparameter that specifies the length of the input sequences, and it's set to 10; this value can be adjusted based on the specific requirements of the dataset or the model architecture. `num_samples` is the total number of training examples to be generated, here equal to 1000. Using NumPy's `random.randint` function, two datasets `fake_query_data` and `fake_value_data` are created, each consisting of 1000 samples where each sample is a sequence of 10 integers (representing tokens), and each integer can range from 0 to 999. Similarly, `fake_labels` is created as a binary target variable for each sample, suitable for a binary classification task.\n"
+      ],
+      "metadata": {
+        "id": "0DRCdMW8j3y7"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "from tqdm import tqdm"
+      ],
+      "metadata": {
+        "id": "opBhRqAjnUfK"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Generate fake data\n",
+        "batch_size = 32\n",
+        "max_sequence_length = 64  # You can set the sequence length to what makes sense for your problem\n",
+        "num_samples = 1000  # Number of samples in fake data\n",
+        "\n",
+        "# Create fake data with the correct dimensions\n",
+        "fake_query_data = np.random.randint(1000, size=(num_samples, max_sequence_length))\n",
+        "fake_value_data = np.random.randint(1000, size=(num_samples, max_sequence_length))\n",
+        "# fake_labels = np.random.randint(2, size=(num_samples, 1))\n",
+        "module_1 = (fake_query_data[:, 0] + fake_query_data[:, 1]) % 2\n",
+        "module_2 = (fake_query_data[:, 2] + fake_query_data[:, 3]) % 2\n",
+        "\n",
+        "fake_labels = []\n",
+        "for i in tqdm(range(num_samples)):\n",
+        "    if np.random.randint(0, 2):\n",
+        "        fake_labels.append(module_1[i])\n",
+        "    else:\n",
+        "        fake_labels.append(module_2[i])"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "UMIRmUt8jaVO",
+        "outputId": "8c794a82-3bd4-4b08-f4f8-4e191adf35f4"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stderr",
+          "text": [
+            "100%|██████████| 1000/1000 [00:00<00:00, 158875.15it/s]\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "fake_labels = np.array(fake_labels)\n",
+        "fake_labels[:20]"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "dmgj_GRAm7w_",
+        "outputId": "9c96b323-e0fa-4ec3-85e2-31c926b18d3a"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0])"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 56
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "print(fake_query_data.shape)\n",
+        "print(fake_value_data.shape)\n",
+        "print(fake_labels.shape)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "rZkdP7Fsjh4a",
+        "outputId": "de71e383-0729-40d9-8958-e232c94c78d0"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "(1000, 64)\n",
+            "(1000, 64)\n",
+            "(1000,)\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Model Architecture Explanation\n",
+        "\n",
+        "The given code defines a neural network model using TensorFlow, particularly with its high-level API, Keras. The model is designed to process sequences of integers, which could represent words or tokens in a natural language processing context.\n",
+        "\n",
+        "First, the code imports the required TensorFlow and NumPy libraries. It then defines two input layers: `query_input` and `value_input`. Both are expected to receive sequences of integers of variable length, as denoted by `shape=(None,)`, and the data type specified is `int32`.\n",
+        "\n",
+        "An embedding layer, `token_embedding`, is then declared, which is capable of mapping each integer in the sequences to a 64-dimensional vector space. The input dimension is set to 1000, meaning the layer can handle 1000 unique tokens. The `query_embeddings` and `value_embeddings` are generated by passing `query_input` and `value_input` through this embedding layer, effectively converting the token sequences into dense vectors that capture semantic information.\n",
+        "\n",
+        "Following the embeddings, a 1D convolutional neural network (CNN) layer is applied to both sets of embeddings. This `cnn_layer` uses 100 filters and a kernel size of 4, with 'same' padding to ensure the output length is equal to the input length. The result of this operation is two sets of encoded sequences, `query_seq_encoding` and `value_seq_encoding`.\n",
+        "\n",
+        "The code then employs an attention mechanism, specifically using the `Attention` layer, to compare the query and value encodings and generate a sequence that represents the weighted importance of the values with respect to the query.\n",
+        "\n",
+        "Global average pooling is applied to both the query encoding and the combined query-value attention sequence. This operation reduces the sequence dimension, averaging out the features across the time steps, resulting in two fixed-size vectors.\n",
+        "\n",
+        "These vectors are then concatenated to form a single input layer, `input_layer`, which serves as the combined representation of both the query and value information.\n",
+        "\n",
+        "Next, the code defines a dense neural network layer with 64 units and a ReLU activation function to introduce non-linearity into the model. This is followed by the output layer, which is a single neuron with a sigmoid activation function, indicating the model is designed for binary classification.\n",
+        "\n",
+        "Finally, the `model` is constructed by specifying the inputs and output layer. This model can now be compiled and trained on suitable data for tasks such as document retrieval, question answering, or any other scenario where the relationship between two sequences needs to be modeled.\n"
+      ],
+      "metadata": {
+        "id": "wXVBH7hxkeKT"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DO_DwbcijQij"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "import numpy as np\n",
+        "\n",
+        "# Define the inputs\n",
+        "query_input = tf.keras.Input(shape=(None,), dtype='int32')\n",
+        "value_input = tf.keras.Input(shape=(None,), dtype='int32')\n",
+        "\n",
+        "# Embedding lookup\n",
+        "token_embedding = tf.keras.layers.Embedding(input_dim=1000, output_dim=64)\n",
+        "# Query and value embeddings\n",
+        "query_embeddings = token_embedding(query_input)\n",
+        "value_embeddings = token_embedding(value_input)\n",
+        "\n",
+        "# CNN layer\n",
+        "cnn_layer = tf.keras.layers.Conv1D(filters=100, kernel_size=4, padding='same')\n",
+        "# Encoded sequences\n",
+        "query_seq_encoding = cnn_layer(query_embeddings)\n",
+        "value_seq_encoding = cnn_layer(value_embeddings)\n",
+        "\n",
+        "# Attention layer\n",
+        "query_value_attention_seq = tf.keras.layers.Attention()([query_seq_encoding, value_seq_encoding])\n",
+        "\n",
+        "# Global Average Pooling\n",
+        "query_encoding = tf.keras.layers.GlobalAveragePooling1D()(query_seq_encoding)\n",
+        "query_value_attention = tf.keras.layers.GlobalAveragePooling1D()(query_value_attention_seq)\n",
+        "\n",
+        "# Concatenate layer\n",
+        "input_layer = tf.keras.layers.Concatenate()([query_encoding, query_value_attention])\n",
+        "\n",
+        "# Add DNN layers\n",
+        "dense_layer = tf.keras.layers.Dense(64, activation='relu')(input_layer)\n",
+        "output_layer = tf.keras.layers.Dense(1, activation='sigmoid')(dense_layer)\n",
+        "\n",
+        "# Create the model\n",
+        "model = tf.keras.Model(inputs=[query_input, value_input], outputs=output_layer)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "tf.keras.utils.plot_model(model)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 607
+        },
+        "id": "SqMVuadensV9",
+        "outputId": "6baa0d27-4465-47cc-d48d-e182a96400f1"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "data": {
+            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6gAAALlCAIAAABGklx5AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdZ0BUV8LG8TMwzNCkGRQ3FAUL9hKwoEZiRyNroVmSaCzEmAQ1JvDaoqsmagoxKiaWGI0NQUVNxG6sgDUW7B27ooIUZYD7fphdlrUgKHAH7v/3aefOzLkPZy/45HLmoJIkSQAAAADlnZHcAQAAAIDSQPEFAACAIlB8AQAAoAgUXwAAACiCWu4AgCGKi4v74Ycf5E6BMmnkyJEtWrR4zUH8/f2LJQyUpkWLFiNHjpQ7BWC4uOMLPEdSUlJ0dLTcKWQWHx8fHx8vd4oyJjo6OikpqVjGuXbt2uuPU3Zdu3aN78Giio+Pj4uLkzsFYNC44wu8UFRUlNwR5KS/6ajwSSgqlUpVXEONGDEiICCguEYrc1auXBkYGMjlVyT8ogB4Ke74AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii/w6jZs2GBtbb1+/Xq5g/xbbm5ueHi4l5dX/oPTpk1zd3c3MzOzsLBwd3cfN25campqcZ0xPj6+du3aRkZGKpWqcuXKkydPLq6RX2TVqlWurq4qlUqlUjk4OPTr16+kz2iwysTll9/jx4/d3d3Hjh1bXGfk8gNQVGq5AwBlmCRJckf4r3Pnzg0YMGDv3r0NGzbMf3z37t2DBw9+//33zczMYmNj+/btm5CQsHnz5mI5afPmzU+dOtW5c+dNmzadOXPGxsamWIYtQK9evXr16lW9evV79+7dunWrpE9nyMrE5ZffmDFjzpw5U4wn5fIDUFTc8QVeXdeuXVNSUrp161ZC42dmZhZw/yy/o0ePhoWFDR06tFGjRk89pdFohg0bZm9vb2lp6e/v37179y1btty8ebME8paUws+DopSJyy/Pvn37Tpw4UXzpSg+XH1CeUHwBw7VgwYI7d+4U5pUNGzZctWpV3759tVrtU0+tXr3a1NQ07+Gbb74phEhLSyvGnCWt8POAYlQsl59eZmbmF1988eOPPxZrwFLC5QeUJxRf4BXt2bPH2dlZpVLNmjVLCBEREWFhYWFubr527VofHx8rKytHR8fly5cLIX766SdTU9NKlSp99NFHVapUMTU19fLySkhIEEJ89tlnGo3GwcFBP+awYcMsLCxUKtW9e/eGDx/++eefX7hwQaVSVa9evbhinzt3zsbGxsXFpbgGfIohzMPu3bvr1KljbW1tampav379TZs2CSEGDRqkX5rp5uZ25MgRIcSAAQPMzc2tra3XrVuXk5Mzfvx4Z2dnMzOzBg0aREZGCiGmT59ubm5eoUKFO3fufP7552+++Wbx/qb+dZSty2/MmDH6Xzu85jgvZQjzoITLDyjDJADP0P/D89KXJSUlCSFmzpypfzhmzBghxLZt21JSUu7cudO6dWsLC4usrCxJkoKDgy0sLE6ePPn48ePExERPT88KFSpcvXpVkqS+fftWrlw5b8xvv/1WCHH37l1Jknr16uXm5lak5M2aNWvYsOGzx7Oysq5duzZz5kytVvv7778XZig/Pz8/P7/CvLJTp05CiAcPHugflsI8uLm5WVtbvyhPVFTUhAkT7t+/n5yc3Lx584oVK+qP9+rVy9jY+Pr163mv7NOnz7p16yRJGjVqlFarjY6OfvDgwejRo42MjA4cOJD3tYSEhMycObNnz56nTp0qeCqEEJGRkYWZtNcfp6xcfnv27PH19ZUk6e7du0KIMWPGFGaoQn4PSlx++RT+exZQLO74AsXMy8vLysrK3t4+KCgoPT396tWr+uNqtbp27dparbZOnToRERGPHj1auHBhqaVycnJydHScMGHC9OnTAwMDS+GMMs6Dn5/fV199ZWtra2dn5+vrm5ycrK9cQ4cOzcnJyTtdamrqgQMHunTp8vjx44iIiB49evTq1cvGxmbs2LEmJib5U02dOvWTTz5ZtWqVu7t78UYtdoZ2+WVmZg4fPjwiIqIUzpUflx+A56L4AiVFo9EIIXQ63bNPeXh4mJubnz59utTCJCUl3blzZ9myZYsWLWrcuHFprlmUdx5MTEyEEDk5OUKItm3b1qxZ89dff5UkSQixYsWKoKAgY2PjM2fOZGRk1KtXT/8WMzMzBweH0vx/pyQYyOU3evToIUOG6FeWy4LLD0B+FF9AHlqtVn8fqHSYmJjY29t37NhxxYoViYmJX3/9damdumAlMQ9//vmnt7e3vb29Vqv98ssv846rVKqPPvro4sWL27ZtE0IsXrx44MCBQoj09HQhxNixY1X/ceXKlYyMjOJNZVBK5/Lbs2fP8ePHBw0aVNInemVcfoDSUHwBGeh0uocPHzo6Opb+qatXr25sbJyYmFj6p35W8c7Drl27wsPDr1692qNHDwcHh4SEhJSUlGnTpuV/Tf/+/U1NTefPn3/mzBkrKyv9h/z0H7oKDw/Pvw4sLi6uWFIZoFK7/BYsWLBt2zb9H5hQqVT6eZ4yZYpKpTp48GBJn/2luPwABaL4AjL466+/JElq3ry5EEKtVj/397DFIjk5uU+fPvmPnDt3Licnx8nJqYTOWCTFOw+HDh2ysLA4fvy4Tqf7+OOPXV1dTU1NVSpV/tfY2toGBgbGxMR89913gwcP1h90cnIyNTX9+++/X+fsZUipXX4LFy7MX+byf7jNw8OjhE5aeFx+gAJRfIFSkpub++DBg+zs7GPHjg0fPtzZ2bl///5CiOrVq9+/fz8mJkan0929e/fKlSt5b7Gzs7tx48bly5cfPXr0av8qW1hYbN68efv27ampqTqd7siRIx988IGFhcXIkSOL6+sqqpKYB51Od/v27b/++svCwsLZ2VkIsXXr1sePH587d06/X1V+Q4cOffLkyR9//JH3px9MTU0HDBiwfPnyiIiI1NTUnJyca9eula2/8fFSslx+BojLD1C6ktwyAiirCrOV0syZM/Ubf5qbm/v6+s6ePdvc3FwIUaNGjQsXLsydO9fKykoI4eLicvbs2eDgYBMTkzfffFOtVltZWXXv3v3ChQv6cZKTk9955x1TU9Nq1ap9+umnX3zxhRCievXqV69ePXz4sIuLi5mZWatWrW7dulVAmLi4uJYtW1apUkX/fe3g4ODl5bVz505Jknx9fatVq2ZpaanVat3c3IKCgo4fP16YSSjM1kjx8fF169Y1MjLSn3TKlCklPQ9z5sxxc3N70Q+01atXS5IUGhpqZ2dnY2Pj7++v3+bWzc1Nv2uVXuPGjf/v//4v/xfy5MmT0NBQZ2dntVptb2/fq1evxMTEadOmmZmZCSGcnJwKuQecKK3tzMrK5ZdfsW9nxuX3FLYzA16K4gs8R+H3EC2k4OBgOzu7YhywFJTEP6IGMg9dunS5ePFiSYxcasW3SAxk2ouk2L8HJYOZh5K7/Ci+wEux1AEoJfotjSDXPOT9kvrYsWP6G3uyxJALl58elx+gcBRfoAw4ffq06sWCgoLkDlgGhIaGnjt37uzZswMGDJg0aZLcccoSLr/Xx+UHGAiKL1DiRo8evXDhwpSUlGrVqkVHR7/CCO7u7gX84mbFihXFnrkkvP48vA5zc3N3d/f27dtPmDChTp06pXx2GXH56XH5ARBCqCRJkjsDYHBWrlwZGBio8O8Of39/IURUVJTcQcoSlUoVGRkZEBBgIOOUXXwPvgK+Z4GX4o4vAAAAFIHiCwAAAEWg+AIAAEARKL4AAABQBIovAAAAFIHiCwAAAEWg+AIAAEARKL4AAABQBIovAAAAFIHiCwAAAEWg+AIAAEARKL4AAABQBIovAAAAFEEtdwDAcPn7+8sdQU7x8fGiVCZBp9OZmJiU9FnKnPDw8KioKLlTFIPc3Nzc3Fy1umj/3Fy7dk0o/nuwqOLj45s3by53CsCgGU+YMEHuDIDBSU1NTUlJkTuFzBwdHR0dHUv6LNeuXdu7d6+9vb2ZmVlJn6sU1KlTp3Pnzk5OTq85TmJiopWVVbFEkld2dva+fftu375d1GvJysqqTp06JZSqvHJ0dGzRokWLFi3kDgIYLpUkSXJnAKBcaWlpgYGBO3fujIyM7Nq1q9xxUJxu3brVtWvXGzdu/PHHH2+99ZbccQCANb4AZGVpabl27do+ffr885//nDt3rtxxUGwuXLjQunXrlJSUXbt20XoBGAiWOgCQmZGR0bvvvitJ0qhRox4/fty+fXu5E+F1JSQktGvX7h//+Mf27dtff+EHABQXPtwGQH4qlWrChAnOzs7BwcF37tyZO3duUT8LBcOxdu3a3r17d+zYcdmyZebm5nLHAYD/YqkDAEPx4Ycfrlq1Sr/Y99GjR3LHwauYP3++n59f7969o6Ojab0ADA3FF4AB8fX13bFjx99//92uXbs7d+7IHQdFIEnShAkThgwZMmbMmAULFnDPHoABYlcHAAbn4sWLnTt3zsnJiY2NrVmzptxx8HJZWVkDBw5csWLFnDlzBg0aJHccAHg+7vgCMDiurq5xcXGVK1f28vLat2+f3HHwEmlpaf/85z9jYmLWrVtH6wVgyCi+AAxRxYoVt2zZ0rx5844dO/7xxx9yx8EL3bx58+233/7777937tzp4+MjdxwAKAjFF4CBsrCwWLt2bb9+/bp37/7zzz/LHQfPcerUqRYtWjx69Gj37t1NmjSROw4AvAT7+AIwXPotfrVa7YgRIx4/ftyuXTuVSiV3KPxbfHx8x44dXVxctm3bVgp/3RoAXh+fugVg6EJDQytXrjxkyJBbt27NmzfPxMRE7kQQa9as6du3b6dOnZYtW2ZmZiZ3HAAoFJY6ACgD+vfv/+eff65evZotfg3BzJkz/fz8Bg4cuGrVKlovgDKE4gugbOjQocO2bduOHj3atm3b27dvyx1HofSb9YaEhIwbN27mzJlGRvwjAqAsYR9fAGXJxYsXfXx8dDpdbGxsrVq15I6jLFlZWQMGDIiOjv7111/79u0rdxwAKDL+Yx1AWeLq6rpv374qVap4eXnt3btX7jgK8ujRI19f33Xr1q1du5bWC6CMovgCKGMqVqy4efNmLy+v9u3bR0dHyx1HEfSb9R47dmzXrl2dO3eWOw4AvCKKL4Cyx8LCIiYmpn///kFBQXPmzJE7TjmXmJjYvHnzrKys+Pj4xo0byx0HAF4d25kBKJOMjY3nzJlTtWrVYcOGnT59+scff2SL35IQFxfXrVu3mjVrrlu37o033pA7DgC8FoovgDIsNDTUwcFh8ODBDx8+nD9/Plv8Fq/Vq1f37dvXx8dn6dKlbFsGoBxgqQOAsu2DDz7YsGFDTEyMj49Pamqq3HHKjxkzZvj7+w8ZMiQ6OprWC6B8oPgCKPPat2+/e/fu06dPt2rV6vr163LHKfMkSQoLCxsxYsS4ceNmzJjBZr0Ayg328QVQTly6dMnHx+fJkyexsbHu7u5yxymrnjx50r9//9WrVy9cuLBPnz5yxwGA4sR/xwMoJ6pVq7Zv3z5HR8eWLVvu2bNH7jhl0sOHDzt16vTnn3+uX7+e1gug/KH4Aig/7OzstmzZ0rZt2/bt269cuVLuOGXMjRs3vL29z549u2vXro4dO8odBwCKH8UXQLliamq6YsWKgQMH9unTZ/bs2XLHKTP0m/VmZ2fHx8c3atRI7jgAUCKMJ0yYIHcGAChORkZGXbt2tba2DgkJuX//fqdOndjit2A7duzo2LFjnTp1Nm/eXLlyZbnjAEBJYR9fAOVTSEiIra3toEGD7t+/v2DBAo1GI3ciA7Vq1ap+/fp17dp1yZIlpqamcscBgBLEUgcA5db7778fGxu7bt06Hx+flJQUueMYohkzZgQEBAwZMmTlypW0XgDlHtuZASjnjh8/3qVLFxsbm9jYWEdHR7njGApJkkJDQ7/77rupU6d++eWXcscBgNJA8QVQ/l2+fNnHx+fRo0cbNmxo0KCB3HHk9+TJkw8++CAmJua3334LCgqSOw4AlBKWOgAo/6pWrbpv3z5XV1dvb+9du3bJHUdmDx486Nix4+bNmzdt2kTrBaAoFF8AimBra7tly5YOHTp07NgxMjJS7jiyuX79ure39/nz53fs2NGmTRu54wBAqaL4AlAKrVa7bNmyQYMG9e7d+9tvv5U7jgxOnDjRvHnz3Nzc+Pj4hg0byh0HAEob+/gCUBAjIyP9B92++OKLF23x++TJE7W6DG/1mJOTk5uba2T09H2N7du3d+rUqV69eps3b65UqZIs2QBAXtzxBaA4ISEhkZGRc+fODQwMfPz4cf6n/vjjDz8/P7mCFYtFixZ99NFHTx1csmSJj49Phw4dNmzYYG1tLUswAJAduzoAUKjt27f37NmzSZMma9as0XfB/fv3t2nT5vHjx5s2berYsaPcAV9Fenp6tWrV7t69+9VXX+X9Qm/GjBkjR4785JNPwsPDn70TDADKQfEFoFwnTpzw8fGxsbHZsGHDkydPmjZtmpqaKoSoWbPm8ePHjY2N5Q5YZBMnTpw8eXJ2drYQYt68eQMGDAgJCYmIiJg+ffqoUaPkTgcAMqP4AlC0K1eu+Pj4pKWlGRkZ3bhxQ6fTCSGMjIzmzp07cOBAudMVzY0bN9zc3PIWbxgZGXXo0GHnzp1Lly7t2bOnvNkAwBBQfAEo3fXr199+++2kpCR96xVCqFQqW1vby5cvV6hQQd5sRfLhhx8uWbIk/1ehVqt/+umnZ5f8AoAysdgLgKLl5OR89NFHV69ezeuLQghJklJTU7///nsZgxXVsWPHFi1a9NRXkZub+3//939nz56VMRgAGA7u+AJQtI8//nju3Lk5OTnPPqXRaM6fP+/k5FT6qV5Bu3btdu/enb/46qnV6ipVqhw4cKBy5cqyBAMAw8EdXwDKNXHixDlz5jy39QohJEkaP358KUd6NRs3bty+ffuzrVcIkZ2dfe3atS5duqSnp5d+MAAwKBRfAAqVm5trZWVVrVo1IYSJicmzL9DpdIsXLz569GipRyuanJyc4cOHP3cPCpVKZWRkZGZm1rx58wcPHpR+NgAwKCx1AKB0e/bs+fHHH2NiYlQqlX4jsDwmJibNmzfftWuXXNkK45dffvn4449zc3PzHzQxMdHpdNWrV//kk08GDhxoaWkpVzwAMBwUXwAQQohbt24tWrTop59+unnzplqtzr9sYMOGDT4+PjJmK0BaWlq1atWSk5Pzfpir1WqVSuXr6/vRRx+1b99e3ngAYFAovgDwX9nZ2evXr581a9aOHTv09dfIyKh69eonT540zL9nMW7cuG+++SYnJ0etVmdnZzs7O3/22Wf9+/evWLGi3NEAwOBQfAG80MqVK+WOIJtbt25t3bp127ZtGRkZQojBgwcb4N3T5OTkkJAQfTtv3Lhx586d69evr1Kp5M4lDy8vL0dHR7lTADBoFF8AL6TYCoWyKDIyMiAgQO4UAAyaWu4AAAwaZULv6NGjkiQ1atSohMb39/cXQkRFRRX+Lenp6Vu2bHn33XfVan6S8x9pAAqFH5cA8HINGzaUO8LTLCwsunfvLncKAChL2McXAAAAikDxBQAAgCJQfAEAAKAIFF8AAAAoAsUXAAAAikDxBQAAgCJQfAEAAKAIFF8AAAAoAsUXAAAAikDxBQAAgCJQfAEAAKAIFF8AAAAoAsUXAAAAikDxBSAzT09PY2PjRo0avfIIgwYNqlChgkql+vvvvwt+asOGDdbW1uvXr3+txAXS6XRff/119erVNRqNjY1NvXr1Ll++XCwjr1q1ytXVVfU8VatWLepo5WzaAaAwKL4AZHbgwIF33nnndUaYP3/+vHnzCvOUJEmvc6LCCAwMXLx48dKlSzMyMk6dOuXm5paWllYsI/fq1evixYtubm7W1taSJEmSlJ2dnZGRcfv2bXNz86KOVs6mHQAKQy13AAAQQgiVSlUKZ+natWtKSkrJjb9ixYqYmJijR4/Wr19fCFGlSpW1a9eW3OmMjY3NzMzMzMxq1qz5aiOUj2kHgELiji8Ag2BiYvI6by+gwBVLt5MkKSoqau7cuQW/bM6cOU2aNNG33tIUExPzam8sH9MOAIVE8QXwWnJycsaPH+/s7GxmZtagQYPIyEghxI8//mhhYWFkZPTWW29VrlzZxMTEwsKiSZMmrVu3dnJyMjU1tbGx+fLLL/OPc/78eXd3dwsLCzMzs9atW+/Zs6eA8YUQkiR9++23tWrV0mq11tbWX3zxRd5QL3pqz549zs7OKpVq1qxZQoiIiAgLCwtzc/O1a9f6+PhYWVk5OjouX74877xff/11rVq1zMzM3njjjWrVqn399dcBAQEFTEVWVlZ8fPzrrJp9TcqcdgAoAgkAXkAIERkZWfBrRo0apdVqo6OjHzx4MHr0aCMjowMHDkiS9NVXXwkhEhIS0tPT792717lzZyHEn3/+effu3fT09M8++0wI8ffff+sHadeunaur66VLl3Q63YkTJ5o1a2Zqanr27NkCxh8zZoxKpfr+++8fPHiQkZExe/ZsIcSRI0cKfiopKUkIMXPmTP15x4wZI4TYtm1bSkrKnTt3WrdubWFhkZWVJUnSlClTjI2N165dm5GRcejQocqVK3t7exc8FZcuXRJCNGrUyNvb28HBQavVuru7z5o1Kzc396VT7efn5+fn99KXSZKUf42vJEkhISHHjx/Pe6jAadcrzLUKABRfAC/00jKRmZlpbm4eFBSkf5iRkaHVaj/++GPpPw3s0aNH+qcWLVokhMiraPv37xdCrFixQv+wXbt2DRs2zBv22LFjQohRo0a9aPyMjAxzc/MOHTrkvUV/y/DIkSMFPCW9oIFlZmbqH+q72vnz5yVJ8vT0bNq0ad4gQ4YMMTIyevLkSQGzcfz4cSFEhw4d9u7dm5yc/PDhw7CwMCHEkiVLCniXXpGK71P3L54tvoqadj2KL4DCYKkDgFd35syZjIyMevXq6R+amZk5ODicPn362VdqNBohRHZ2tv6hfmmpTqd77rD169e3trY+duzYi8Y/f/58RkZGu3btnn1vAU+9lD6kPtXjx4+lfHsR5OTkmJiYGBsbF/B2rVYrhKhbt66Xl5ednZ21tfXEiROtra2LfYnqU3d8C3ilEqYdAAqP4gvg1aWnpwshxo4dm7eh7JUrVzIyMl5/ZBMTE51O96Lxr127JoSwt7d/9o0FPFUkXbp0OXTo0Nq1azMzMw8ePBgTE/Puu+8W3MCqVKkihLh3717eEY1G4+LicuHChdcMU4Aff/wxr6G+vrI47QBQeBRfAK9OX3TCw8Pz/yIpLi7uNYfNzs6+f/++s7Pzi8Y3NTUVQjx58uTZ9xbwVJFMmDChbdu2/fv3t7Ky6tmzZ0BAwIv2rM1jaWlZo0aNkydPPvW1WFtbv2aY0lFGpx0ACo/iC+DV6fcKePYPd72mHTt25ObmNmnS5EXj16tXz8jIaOfOnc++t4CniiQxMfHChQt3797V6XRXr16NiIiwtbV96bsCAwOPHDly8eJF/cOMjIwrV66Uwu5mN2/eHDBgwGsOUnanHQAKieIL4NWZmpoOGDBg+fLlERERqampOTk5165du3nz5isMlZWVlZKSkp2dffjw4c8++8zFxaV///4vGt/e3t7Pzy86OnrBggWpqanHjh3LW0dbwFNF8sknnzg7Oxf1j66NHDlSn/zq1avJycmhoaGZmZn6j7iVEEmSMjMzV61aZWVl9QpvLx/TDgCFVXKfmwNQ1olCfFL+yZMnoaGhzs7OarXa3t6+V69eiYmJP/74o/6P6FatWnX37t1Tp07V/7q/cuXKS5cuXbFiReXKlYUQtra2y5cvlyRp4cKF77zzTqVKldRqdcWKFXv37n3lypUCxpck6dGjR4MHD65YsaKlpWWrVq3Gjx8vhHB0dDx69OiLnho8eLCDg4MQwtzc3NfXd/bs2fqQNWrUuHDhwty5c/Xd0cXF5ezZs9u3b69YsWLej0oTE5PatWuvWrXqpZOWlJTUu3dvW1tbrVbbtGnT2NjYwkx1YXZ1WL169bNbOuQZO3askqe9MNcqAKgk/oQ6gBdQqVSRkZHK/PMBERER586dCw8P1z/MysoKCwuLiIh48OCBmZlZsZ/O399fCBEVFVXsI5ctrzztSr5WARSeWu4AAGBwbt269dlnn+Vf5KrRaJydnXU6nU6nK4niC8G0Ayh5rPEFgKeZmZmZmJgsWLDg9u3bOp3uxo0b8+fPHz9+fKNGjaytrVUvEBQUJHfwsu1F0x4UFPRqK5gB4Cnc8QWAp1lbW2/evPlf//pXzZo109PTLS0t69atO3Xq1CFDhqjV/NgsKQVMu9zRAJQT/AQHgOdo3br1li1b5E6hOEw7gBLFUgcAAAAoAsUXAAAAikDxBQAAgCJQfAEAAKAIFF8AAAAoAsUXAAAAikDxBQAAgCJQfAEAAKAIFF8AAAAoAsUXAAAAikDxBQAAgCJQfAEAAKAIFF8AAAAoglruAAAMWlxcnNwRFOHatWtCiJUrV8odBADKM5UkSXJnAGCgVCqV3BGAwoqMjAwICJA7BQCDRvEFgFKlUqmoaAAgC9b4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBEovgAAAFAEii8AAAAUgeILAAAARaD4AgAAQBHUcgcAgHJu+fLljx49yn9k69atDx8+zHvYvXv3SpUqlXouAFAclSRJcmcAgPLsgw8+WLx4sYmJif5hbm6uSqVSqVRCiJycHOd5J+kAACAASURBVAsLi7t372q1WlkzAoAisNQBAEpW7969hRC6/8jJycnOztb/b2NjY39/f1ovAJQO7vgCQMnKzs6uXLny/fv3n/vs1q1b27VrV8qRAECZuOMLACVLrVb37t07b6lDfhUrVvT29i71RACgUBRfAChxvXv31ul0Tx3UaDTvvfeesbGxLJEAQIFY6gAAJU6SJEdHxxs3bjx1PCEhoWnTprJEAgAF4o4vAJQ4lUr1/vvvP7XawcnJydPTU65IAKBAFF8AKA1PrXYwMTHp37+/flMzAEDpYKkDAJQSd3f3M2fO5D08ceJE3bp1ZcwDAErDHV8AKCXvvfde3mqHOnXq0HoBoJRRfAGglPTu3Ts7O1sIYWJi8sEHH8gdBwAUh6UOAFB6PDw8Dh8+LIS4dOmSi4uL3HEAQFm44wsApef999+XJKlp06a0XgAofdzxBVBK2MEAT4mMjAwICJA7BQAFUcsdAICCDB8+vEWLFnKnkNk333zz8ccfW1tb6x+Gh4cLIUaMGCFrKBkEBgbKHQGA4lB8AZSeFi1acIevcePGNWrUyHsYFRUlhFDgtFB8AZQ+1vgCQKnK33oBAKWJ4gsAAABFoPgCAABAESi+AAAAUASKLwAAABSB4gsAAABFoPgCAABAESi+AAAAUASKLwAAABSB4gsAAABFoPgCAABAESi+AAAAUASKLwAAABSB4gsAAABFoPgCUK7c3Nzw8HAvL68XvWDQoEEVKlRQqVR///33S0fT6XRff/119erVNRqNjY1NvXr1Ll++XJxxhThz5synn35at27dChUqqNVqa2vrmjVrdu3aNS4urnhPlN+zs7Rq1SpXV1dVPhqNplKlSt7e3t9+++2DBw9KLgwAvA6KLwCFOnfu3Ntvvz1y5MiMjIwXvWb+/Pnz5s0r5ICBgYGLFy9eunRpRkbGqVOn3Nzc0tLSiimsEEIsWLCgfv36x44d++GHH5KSktLT048cOTJp0qSHDx8eP368GE+U33NnqVevXhcvXnRzc7O2tpYkKTc3986dOytXrqxWrVpoaGjdunUPHjxYQnkA4HWo5Q4AADI4evTov/71r6FDh6anp0uS9PoDrlixIiYm5ujRo/Xr1xdCVKlSZe3ata8/bJ74+Pjg4OA2bdps2rRJrf73j25XV1dXV1cbG5tz584V47nyFHKWVCqVjY2Nt7e3t7d3165dAwMDu3btevbsWWtr65JIBQCvjDu+AJSoYcOGq1at6tu3r1arLfiVKpWqMAPOmTOnSZMm+tZbEiZPnpyTk/PNN9/ktd48nTp1+uSTT0ripIWfpTx+fn79+/e/c+fOzz//XBKRAOB1UHwBGJbff//dw8PD1NTUwsKiatWqkyZNEkJIkvTDDz/Url1bq9Xa2tp279799OnTQoiIiAgLCwtzc/O1a9f6+PhYWVk5OjouX75cCFG7dm2VSmVkZPTWW2/pf03/5ZdfWltbm5qa/vbbbwUEkCTp22+/rVWrllartba2/uKLL16aOSsrKz4+vlGjRsUyA88df9u2bRUrVmzatGkBLyvNWSpA//79hRCxsbGv9nYAKEESAJQKIURkZGTBrwkPDxdCfPPNN8nJyffv3//ll1/69u0rSdL48eM1Gs3vv//+8OHDY8eONWnS5I033rh165YkSWPGjBFCbNu2LSUl5c6dO61bt7awsMjKysrOzq5ataqzs3N2dnbe+CNGjAgPD89/xmbNmjVs2DD/kTFjxqhUqu+///7BgwcZGRmzZ88WQhw5cqSA2JcuXRJCNGrUyNvb28HBQavVuru7z5o1Kzc396XT4ufn5+fnV/Brzp49K4Ro3rx5wS8rzVmSJClvje9TUlNThRBOTk4Fpy3M9QAAxYs7vgAMhU6nmzhx4jvvvBMWFmZnZ2draztw4EBPT8/MzMwffvihZ8+e/fr1s7a2rl+//s8//3zv3r25c+fmvdfLy8vKysre3j4oKCg9Pf3q1avGxsYhISFXr15dvXq1/jUZGRmrVq368MMPC8iQmZkZHh7evn37kSNH2tjYmJmZ2dnZvTS5/kNs9vb2U6ZMSUxMvH37dvfu3T/55JNly5a93pT8m75KWlpaFpy81GapYPp9MB49evTKIwBACaH4AjAUx44de/jwYadOnfKO6GtZYmJiWlqah4dH3nFPT0+NRpOQkPDsIBqNRgih0+mEEIMGDbK2tv7xxx/1Ty1ZsqR79+5WVlYFZDh//nxGRka7du2KlFy/BLZu3bpeXl52dnbW1tYTJ060trbOXzpfh77yFrD7hBCiNGepYPpPwr3OCABQQii+AAyF/r6mjY3NU8cfPnwonrnfaWNj89J7ipaWlkOGDNm3b9/+/fuFEHPmzPnss88Kfsu1a9eEEPb29kVKXqVKFSHEvXv38o5oNBoXF5cLFy4UaZwXqVq1qqmpqX7Bw4uU5iwVTJ/T3d39dQYBgJJA8QVgKP7xj3+I/62Pevoq/FSBe/jwoaOj40vH/Oyzz0xMTMLDw3ft2uXk5OTm5lbw601NTYUQT548KVJyS0vLGjVqnDx5Mv/B7Ozs4trPS6vVdurU6d69e3v37n322fv37w8aNKg0Z6lgGzduFEL4+Pi8ziAAUBIovgAMRdWqVe3s7DZv3vzU8Xr16llaWub/mwgJCQlZWVlvvfXWS8d0dHQMCAiIjo4eN27c8OHDX/r6evXqGRkZ7dy5s6jhAwMDjxw5cvHiRf3DjIyMK1euFOPuZhMmTNBqtSNHjszMzHzqqRMnTqjV6tKcpQLcunUrPDzc0dHxdVYJA0AJofgCMBRarXb06NG7du367LPPrl+/npub++jRo5MnT5qamn7++eerV69esmRJamrq8ePHhw4dWqVKleDg4MIM+/nnn2dnZz948KBt27YvfbG9vb2fn190dPSCBQtSU1OPHTtWyHW6I0eOdHFx6d+//9WrV5OTk0NDQzMzM8PCwgrz3sJo1KjR0qVLT5w40bp16w0bNqSkpOh0ukuXLs2bN2/gwIEmJialOUt5JElKS0vTb15x9+7dyMjIli1bGhsbx8TEsMYXgCGSeVcJAIohCrd91axZs+rXr29qampqatq4cePZs2dLkpSbm/vtt9/WqFHDxMTE1ta2R48eZ86ckSRp9uzZ5ubmQogaNWpcuHBh7ty5+r7l4uJy9uzZvDHfeeed+fPn5z9LXFxcy5Yt9WtzhRAODg5eXl47d+6UJOnRo0eDBw+uWLGipaVlq1atxo8fL4RwdHQ8evRowcmTkpJ69+5ta2ur1WqbNm0aGxtbmGkpzHZmea5evTpq1Kj69etbWloaGxvb2Ng0btx44MCBe/fuLc1ZWrduXYMGDczNzTUajZGRkfjPH29r2rTpv/71r+Tk5MJ8LYW8HgCgGKmk4vhbnQDwUiqVKjIyMiAgQO4ghsXf318IERUVJXeQ0sb1AKD0sdQBAAAAikDxBYCXO336tOrFgoKC5A4IAHg5tdwBAKAMcHd3Z2EYAJR13PEFAACAIlB8AQAAoAgUXwAAACgCxRcAAACKQPEFAACAIlB8AQAAoAgUXwAAACgCxRcAAACKQPEFAACAIlB8AQAAoAgUXwAAACgCxRcAAACKQPEFAACAIlB8AQAAoAgqSZLkzgBAEVQqldwRYFgiIyMDAgLkTgFAQdRyBwCgFJGRkXJHKH7Tp0+/ffv2d999J2+tT0pKmjhxop2d3ahRoypVqiRjkiLx8vKSOwIAZeGOLwC8op9//nnYsGHbtm3z9vaWO4tISkrq2bPnpUuXli9f3qFDB7njAIAhYo0vALyKCxcufPHFF2FhYYbQeoUQTk5Ou3fv7tq1q4+Pz7Rp0+SOAwCGiDu+AFBk2dnZrVq10ul0cXFxGo1G7jj/Y+7cucOGDfP3958/f765ubnccQDAgHDHFwCKbMKECcePH1+6dKmhtV4hxJAhQ/7444+NGze2bNny8uXLcscBAANC8QWAotm7d+/UqVO/++47d3d3ubM8X6dOnQ4cOJCdne3p6bl9+3a54wCAoWCpAwAUQVpaWuPGjd3c3GJjYw18g7a0tLQBAwbExMRMnjw5NDRU7jgAID/u+AJAEQwbNiwlJeW3334z8NYrhLC0tFy5cuXkyZNHjx7dr1+/zMxMuRMBgMy44wsAhbV69Wo/P7+1a9d269ZN7ixFEBsb26dPH1dX1zVr1jg7O8sdBwBkQ/EFgEK5fv16w4YN/f3958yZI3eWIjt37lz37t3v3r27cuVKA9l/DQBKH0sdAODlJEkaPHiwjY3Nt99+K3eWV1GjRo34+PhWrVp16NDhp59+kjsOAMiD4gsALxceHr5ly5alS5daWlrKneUVVahQYdWqVZMnTx4xYkRwcHBWVpbciQCgtLHUAQBeIjEx0cPDY+zYsWPGjJE7SzH4448/+vXrV7du3ejo6CpVqsgdBwBKD8UXAAry5MmTpk2bVqhQYefOncbGxnLHKR5nzpzp0aNHSkrK6tWrmzVrJnccACglLHUAgIKEhYVdvHjxt99+KzetVwhRq1at+Ph4T0/PNm3a/Prrr3LHAYBSQvEFgBfaunXrjBkzZs+eXb16dbmzFDMrK6s1a9ZMnDhx0KBBwcHBOp1O7kQAUOJY6gAAz/fw4cMGDRp4enquWrVK7iwlaOXKlR9++GHjxo2jo6MrV64sdxwAKEEUXwB4vqCgoD179hw9erRixYpyZylZx44d69GjR1ZW1urVqz09PeWOAwAlhaUOAPAcv/32W1RU1KJFi8p96xVCNGjQ4MCBA7Vr13777bd/++03ueMAQEmh+ALA0y5duhQSEjJixIh27drJnaWU2NnZxcbGhoSEDBgwgCW/AMorljoAwP/Izc1t27btvXv3Dh48aGpqKnec0rZixYqBAwd6eHhERUVVqlRJ7jgAUJy44wsA/2PKlCnx8fHLli1TYOsVQgQFBe3du/fq1aseHh6HDh2SOw4AFCeKLwD816FDhyZNmjR16tQGDRrInUU2jRo1OnDgQM2aNVu1avX777/LHQcAig1LHQDg39LT09966y0nJ6fNmzerVCq548gsOzt77Nix06dP//LLL6dMmVKe/n4HAMWi+ALAvwUHB69cufLYsWNOTk5yZzEUS5cuHTx4cJs2bZYtW2Zrayt3HAB4LRRfABBCiNjY2K5duy5fvjwwMFDuLIbl8OHDPXr00Gg0a9asqVevntxxAODVscYXAMSdO3cGDBjQv39/Wu+zmjRpcvDgQUdHxxYtWpTvP2IHoNwznjBhgtwZAEBOkiT17t374cOHMTExWq1W7jiGyMLCol+/fmlpacOHD8/MzGzbti1roAGURWq5AwCAzCIiIjZs2LB9+3YrKyu5sxgutVo9depUV1fXTz/99Pjx40uXLrWxsZE7FAAUDWt8ASjaqVOnPDw8Ro0aNXHiRLmzlA379u3z8/OztLSMiYmpU6eO3HEAoAgovgCUS6fTtWzZMjc3Ny4uzsTERO44ZcaNGzd69eqVmJi4ePHi7t27yx0HAAqLD7cBUK7x48efPHly6dKltN4i+cc//vHXX38FBAT07NkzLCwsNzdX7kQAUCjc8QWgUHv27PH29p4zZ87gwYPlzlJWzZ0795NPPuncufOSJUtYIQ3A8FF8AShRSkpKw4YNGzVqFBMTI3eWsm3Pnj1+fn42NjYxMTHu7u5yxwGAgrDUAUD5t3z58rt37+Y/MmzYsMzMzF9++UWuSOVGq1atDh48aGVl1axZs/Xr18sdBwAKQvEFUP598cUXtWvX3rhxo/5hdHT0smXLFixYULlyZXmDlQ+Ojo67du3q2bNnjx49pk2bxi8SARgsljoAKOdOnTpVp04dlUolSdKwYcOGDx/erFmzPn36zJw5U+5o5Y1+yW/Pnj0XLFhgYWEhdxwAeBrFF0A598MPP4SGhmZnZwsh1Gp1hQoVbGxsTpw4YW5uLne0cmjXrl3+/v5VqlRZs2ZNtWrV5I4DAP+DpQ4Ayrk//vgjb7+t7OzsR48eXblyZfr06Tk5OfIGK5fefvvtgwcPqtVqT0/PrVu3yh0HAP4HxRdAeZaenr5nz578G81mZ2fn5uZOmjSpRYsWFy9elDFbeeXk5LRnz54uXbp07tx52rRpcscBgP+i+AIoz3bs2KHT6Z49npube+jQoUaNGu3YsaP0U5V7pqamixYtmjJlypgxY/r06ZORkSF3IgAQguILoHzbuHGjRqN59riRkVFubu6gQYO8vLxKP5USqFSq0NDQP//8c+PGja1atbpy5YrciQCAD7cBKNecnZ2TkpKeOmhiYmJlZbV48eIuXbrIkkpRzp8/36NHj1u3bkVGRrZt21buOAAUjTu+AMqtM2fOPNt6VSpVmzZtTpw4QestHdWrV4+Li2vTpk2nTp1Y8gtAXhRfAOXWxo0b1Wp13kMTExONRhMeHr5582YHBwcZgymNpaVlVFTU5MmTR48e/d5772VmZsqdCIBCsdQBQLnVoUOH7du367d0UKvVrq6uUVFRDRo0kDuXcm3YsKFv376urq5r1qxxdnaWOw4AxeGOL4DyKTMzc9euXbm5uUZGRkKIoUOHHjt2jNYrry5duuzfv//x48ceHh47d+6UOw4AxaH4Aiif/vrrr6ysLCMjIzs7u02bNv30009arVbuUBA1atSIj49v2bJlp06d5s2bJ3ccAMrCUgfAIMTFxf3www9ypyhX/v777/Pnzzs4OHh6eiqn8o4cObJFixZyp3g5SZKmT58+evToQYMGzZw587lbzgFAseOOL2AQkpKSoqOj5U5RxsTHx8fHx7/o2bt37zZu3LhVq1bKab3R0dHP7mJhmPS7/K5du1a/x9mtW7fkTgRAEdQvfwmA0hIVFSV3hLLE399fvGDS0tPTr1y5UqdOnVIPJSeVSiV3hKJ59913ExISunfv7uHhsWrVqmbNmsmdCEA5xx1fAOWQhYWF0lpvGVWrVq2EhIS33nqrTZs2v/76q9xxAJRzFF8AgJysrKxWr149fPjwQYMGBQcH63Q6uRMBKLcovgAAmRkbG0+dOnX58uVLly5t167d7du35U4EoHyi+AIADEJgYOC+ffuuXbvm4eFx4MABueMAKIcovgAAQ9GgQYMDBw64u7u//fbbixYtkjsOgPKG4gsAMCAVK1bcuHFjSEhI//79g4ODs7Oz5U4EoPyg+AIADIt+ye+yZcuWLFnSvn37u3fvyp0IQDlB8QUAGKLevXvv3bv3ypUrLVq0OH78uNxxAJQHFF8AgIFq1KjRgQMHXFxcvLy8+NOGAF4fxRcAYLjeeOONTZs2DRs2LCAgICwsLCcnR+5EAMowii8AwKCp1eqpU6cuXrz4p59+evfddx88eCB3IgBlFcUXAFAG9OvXb8+ePSdPnmzatGliYqLccQCUSRRfAEDZ0KRJk4MHDzo6OjZv3nz16tVyxwFQ9lB8AQBlhr29/aZNm3r37u3n5xcWFpabmyt3IgBlCcUXwMtt2LDB2tp6/fr1JXcKnU739ddfV69eXaPR2NjY1KtX7/Lly8V+lmXLlqlUKi8vr2If+aVKYQ4VQqPRzJ079+effw4PD/f19U1JSZE7EYAyg+IL4OUkSSrpUwQGBi5evHjp0qUZGRmnTp1yc3NLS0sr9rMsW7bMzc0tLi7u/PnzxT54wUphDhVlyJAh27dvP3ToUNOmTU+dOiV3HABlA8UXKA8yMzPz38V86uHrD9i1a9eUlJRu3bq9zpgFWLFiRUxMTFRUVLNmzdRqdZUqVdauXVuvXr3iPUtycvLJkycnTpwohFi8ePFTz5b1OVSgli1bHjx40NbWtlmzZmvXrpU7DoAygOILlAcLFiy4c+fOix6+/oAlbc6cOU2aNKlfv36JnmXlypVdu3b19fU1NTX9/fffn7oFW9bnUJnefPPNnTt3+vv79+jRgyW/AF5OAmAAIiMjC/P9uGvXrtq1a1tZWWm12nr16m3cuFGSpJCQEI1Go/+OdnNze+qhJEnZ2dnjxo1zcnIyNTWtX7/+ihUrJEmaPXu2ubm5mZlZTExM586dK1So8Oabby5btuzZAXfv3u3k5CSEmDlzpj5Gbm7u999/7+7url+P+89//vPUqVMFj1mAJ0+eaDSagQMHFnXS/Pz8/Pz8Cv/6Vq1abd++XZIkX19fIcTOnTvznirrc6gnhIiMjCz8hJQnv/zyi4mJSbdu3VJSUuTOAsBwUXwBg1DI4hsVFTVhwoT79+8nJyc3b968YsWK+uO9evXS97PnPhw1apRWq42Ojn7w4MHo0aONjIwOHDggSdKYMWOEENu2bUtJSblz507r1q0tLCyysrKeHSEpKSl/aRs/frxGo/n9998fPnx47NixJk2avPHGG7du3Sp4zBe5dOmSEKJRo0be3t4ODg5ardbd3X3WrFm5ubkFz0aRiu+VK1fs7e2zs7MlSfr999+FEE9V7TI9h3pKLr6SJO3ataty5cr169e/cOGC3FkAGCiWOgBliZ+f31dffWVra2tnZ+fr65ucnHz37t2C3/L48eOIiIgePXr06tXLxsZm7NixJiYmCxcuzHuBl5eXlZWVvb19UFBQenr61atXCx4wMzPzhx9+6NmzZ79+/aytrevXr//zzz/fu3dv7ty5rzam/kNs9vb2U6ZMSUxMvH37dvfu3T/55JNly5a9fEYKbdmyZe+++66xsbEQwtfXV6vVRkVFZWZmFua9hj+HEEK0bt364MGDpqamnp6emzdvljsOAENE8QXKKhMTEyFETk5OwS87c+ZMRkZG3gfFzMzMHBwcTp8+/ewr9b+a1+l0BQ+YmJiYlpbm4eGRd8TT01Oj0SQkJLzamFqtVghRt25dLy8vOzs7a2vriRMnWltb52+Br2/ZsmU9e/bU/28rK6uOHTumpqYW8hNRhj+H0HN0dNy1a5evr2+XLl2mTZsmsZMGgP9F8QXKkj///NPb29ve3l6r1X755ZeFeUt6eroQYuzYsar/uHLlSkZGxitnePjwoRDC0tIy/0EbG5tHjx692oBVqlQRQty7dy/viEajcXFxuXDhwiuHfMqJEyeOHz/erVu3vEnQ76f77N4Oz2X4c4g8pqamCxcujIiIGDduXJ8+fV7n/yYA5Q/FFygzrl692qNHDwcHh4SEhJSUlGnTphXmXfb29kKI8PDw/Iuc4uLiXjmGjY2NEOKpivbw4UNHR8dXG9DS0rJGjRonT57MfzA7O9va2vqVQz5l6dKlvXv3zj8D9+/fNzMz27x5861bt176dsOfQzxlyJAhW7Zs2b59u5eXl34ROQAIii9Qhhw/flyn03388ceurq6mpqYqlaow79JvRPD3338XV4x69epZWloePHgw70hCQkJWVtZbb731ymMGBgYeOXLk4sWL+ocZGRlXrlwprt3NJElasWLFsGHD8h+0tbX19/fPyckpzEriMjGHeEqbNm0OHjyoVqs9PT23bdsmdxwABoHiC5QZzs7OQoitW7c+fvz43Llz+deD2tnZ3bhx4/Lly48ePdLpdPkfGhsbDxgwYPny5REREampqTk5OdeuXbt582bB53pqwPxPmZqafv7556tXr16yZElqaurx48eHDh1apUqV4ODgV/7SRo4c6eLi0r9//6tXryYnJ4eGhmZmZoaFhb3ygPnt27fPysqqZcuWTx0fOnSoyLfaoazPIZ7l5OS0c+fOtm3bdu7cuZC/IQFQzpX8xhEAXq6Q25mFhoba2dnZ2Nj4+/vPmjVLCOHm5nb16tXDhw+7uLiYmZm1atXq1q1bTz188uRJaGios7OzWq22t7fv1atXYmKifr9YIUSNGjUuXLgwd+5cKysrIYSLi8vZs2fzjzB27FgHBwchhLm5ua+vryRJubm53377bY0aNUxMTGxtbXv06HHmzBnpP3vQvmjMgr+0pKSk3r1729raarXapk2bxsbGvnQ2CrOd2cCBAy0sLNRqdcOGDQ8fPpx3fNKkSfq1xUKIN998c/bs2eVgDoWytzN7kdzc3KlTpxobG+uX/ModB4CcVBIfegUMwMqVKwMDA/l+LBJ/f38hRFRUlNxBDIVKpYqMjAwICJA7iCHauHFjnz59qlatumbNGhcXF7njAJAHSx0AAOVf586d9+/fn5WV5eHhsWPHDrnjAJAHxRdAiTt9+rTqxYKCguQOCEWoXr16fHz822+/3bFjR5b8AsqkljsAgPLP3d2dVRwwBJaWltHR0dOnTx89enRiYuIvv/xiZmYmdygApYc7vgAABVGpVKGhoevWrVu/fn27du1eujsHgPKE4gsAUJyuXbsmJCQ8fPjQw8MjPj5e7jgASgnFFwCgRDVr1kxISGjatKm3t/f8+fPljgOgNFB8AQAKVaFChdWrV0+cODE4ODg4ODgrK0vuRABKFsUXAKBc+iW/MTExK1asaNu27a1bt+ROBKAEUXwBAErXrVu3/fv3Jycne3h47N+/X+44AEoKxRcAAFGrVq29e/fWrVv37bffXrhwodxxAJQIii8AAEIIYWdnt2HDhuHDhw8cODA4OFin08mdCEAxo/gCAPBvxsbGU6dOXb58+ZIlS9q3b3/nzh25EwEoThRfAAD+R2Bg4L59+5KSkjw8PA4ePCh3HADFhuILAMDTGjZseODAgVq1arVu3Xrx4sVyxwFQPCi+AAA8R8WKFTdu3BgSEvLBBx8EBwdnZ2fLnQjA66L4AgDwfPolv0uXLl2yZMm777774MEDuRMBeC0UXwAACtKnT589e/acPn3a09PzxIkTcscB8OrUcgcA8F/+/v5yR3iJ5ORkjUZToUIFuYMIIUR8fLwoC5OGcqBx48YHDx4MCAho0aLFwoUL/fz85E4E4FUYT5gwQe4MAERqampKSorcKQqSlpZ2+PDhY8eOaTSaSpUqyR1HCCEcHR0dHR3lTmFA6tSp07lzZycnJ7mDlE/m5uZ9+/ZNS0sbPnx4ZmZmt+1OIQAAIABJREFU27ZtVSqV3KEAFI1KkiS5MwAwaA8ePJg2bdqMGTNcXFwmTZrEHVYo3JIlS4YMGeLt7b1s2TIbGxu54wAoAoovgBfS6XQLFy4cN25cTk7OF198MWLECI1GI3coQH6HDh3q2bOnVqtds2ZN3bp15Y4DoLD4cBuA59u6dWvjxo0//fTToKCgCxcuhIaG0noBvbfeeisuLu6NN95o0aLFmjVr5I4DoLAovgCedujQIW9v744dO9apU+fUqVMzZsywtraWOxRgWP7xj3/s2LEjMDCwV69eYWFhubm5cicC8HIUXwD/df369eDg4KZNm2ZmZu7atWvlypWurq5yhwIMlFarnTdv3s8//xweHu7r62vgn08FIFjjC0AvPT191qxZkydPtrOzmzRp0nvvvccn1oFC2rt3r5+fn5WVVUxMTO3ateWOA+CFuOMLKF1ubu7ixYurV68+bdq0sWPHnjlz5v3336f1AoXXsmXLgwcP2traNm/efO3atXLHAfBCFF9A0fSfYBs0aJCvr+/p06dDQ0NNTU3lDgWUPW+++ebOnTv9/Px69OgRFhbGb1MBw0TxBRTq1KlT3bp169ChQ6VKlY4cOfLLL78YyJ+lAMoorVa7YMGCn3/++YcffggKCkpPT5c7EYCnUXwBxbl3715ISEj9+vVv3rz5119/bdmyhY1IgeIyZMiQbdu27dy508vL6+LFi3LHAfA/KL6AgmRkZEybNs3NzW3VqlURERH79+9v06aN3KGA8qZ169YHDx7UaDSenp5btmyROw6A/6L4AoogSVJUVFTdunUnTZo0YsSIc+fODRkyxMiInwBAiXB0dNy9e3e3bt18fHymTZsmdxwA/8Z2ZkD5FxcX9/nnnyckJPTt23f69OkODg5yJwKUYu7cucOGDfPz81uwYIG5ubnccQCl434PUJ6dPXs2ICDAy8vLwsLiyJEjixcvpvUCpWnIkCFbt27dtm1by5YtL1++LHccQOkovkD5dP/+/bCwsAYNGpw4cWL9+vVbtmxp0KCB3KEAJWrTpk1cXFxOTo6Hh8e2bdvkjgMoGsUXKG90Ot2MGTPc3NwWLFgwbdq048ePv/vuu3KHAhTNzc1t375977zzTufOnVnyC8iINb5AubJ+/foRI0bcvHnz008/HT16tJWVldyJAPybJEnTp08fPXp07969582bZ2ZmJnciQHEovkA5ceDAgc8//3zPnj1+fn7Tp0+vWrWq3IkAPEdsbGyfPn1cXV1Xr17t4uIidxxAWVjqAJR5SUlJ77//frNmzbKzs/fu3bty5UpaL2CwfHx8Dhw48OTJE09Pz7/++kvuOICyUHyBMiwtLW3ChAk1a9ZMSEiIjIzct29fixYt5A4F4CWqV68eFxfXqlWrDh06sOQXKE0sdQDKpOzs7F9//XXcuHHZ2dlffvnl8OHDtVqt3KEAFEHekt9BgwbNnDlTo9HInQgo/yi+QNmzdevWESNGnDlzZujQoRMnTrSxsZE7EYBX9Oeff/bt27du3brR0dFVqlSROw5QzrHUAShLDh8+3LZt2w4dOlStWvXUqVMzZsyg9QJlWteuXffv3//gwQMPD4/4+Hi54wDlHMUXKBuuX78eHBzctGnTtLS0Xbt2rV+/3s3NTe5QAIpBzZo14+PjPT09vb29FyxYIHccoDyj+AKGLj09fdq0abVr146Njf31118TEhJat24tdygAxcnKymrNmjUTJ04cPHhwcHCwTqeTOxFQPrHGFzBcubm5S5YsCQsLe/To0eeffx4WFmZqaip3KAAlaN26de+9916DBg2io6MrV64sdxygvOGOL2Cgtm7d2qRJk0GDBnXr1u3ChQsTJkyg9QLlnq+v7549e27cuOHh4bF//3654wDlDcUXMDinT58OCAjo0KGDvb394cOHf/nll0qVKskdCkApqV+//oEDB2rXrt2mTZvffvtN7jhAuULxBQxIcnJySEhI/fr1L1y4sGPHji1bttSrV0/uUABKm52dXWxsbEhIyIABA1jyCxQj1vgCBiHz/9u794Co6vz/459hbjDAgCIKykXxhnnXMkPbLK3WNc0LIKaVd1LLyi74y9av2aaZblSKGWpuaiqg5m03M3PX2tJWzRRRFO+SFxS5D3IZzu+P+TZflsuIXObMcJ6Pv+Scw+e853MOH198+MyZwsJPPvlk4cKFHh4ec+fOnTRpklqtlrsoADLbtGnTpEmT7r///qSkJP7yA9QdwReQmSRJmzdvjomJycjIePHFF+fMmePp6Sl3UQAcxbFjx0aMGFFaWvrVV1/17t1b7nIA58ZSB0BOBw8e7N+/f1RUVP/+/dPS0t5//31SL4DyunfvfujQoQ4dOvTv33/t2rVylwM4N4IvII+0tLTIyMiwsDCDwfDLL7+sXbuWTysFUCUfH5/du3e//PLLzz///Msvv2w2m+WuCHBWBF/A3m7fvj179uyuXbsmJycnJCR8++233bt3l7soAA5No9G8//7769evX7ly5ZAhQ7KysuSuCHBKrPEF7KekpGTNmjVvv/22JElvv/32jBkzNBqN3EUBcCZHjx4dMWKEVqv96quveOoLcK+Y8QXsZOfOnZ06dXrllVcmTpx47ty5l19+mdQL4F717Nnz8OHDgYGBDz300JYtWyrslSTprbfeKiwslKU2wPERfIF6cObMmezs7Or2Hjp06JFHHnn66ad79ep18uTJ999/32g02rM8AI1Js2bN9uzZM2PGjIiIiNmzZ5eVlVl3LVmyZOHChe+++66M5QGOjOAL1NWNGzcef/zxv/zlL5V3XblyJTo6um/fvsXFxf/+978TExNbt25t9wIBNDaWJb9ffPHFxx9/PHToUMsv3nv27Jk9e7YQ4oMPPkhJSZG7RsARscYXqJP8/PyHH374+PHjarU6NTU1JCTEun3JkiWLFi0KDAx87733IiIi5K0TQKP0008/hYeHe3l5LV++fMSIEXl5eWVlZRqNpnfv3gcOHFCpVHIXCDgWZnyB2jObzWPGjDlx4oTlT41vvPGGEKK0tDQ+Pr5du3ZLly6dN29ecnIyqRdAAwkLCzt06JC/v//kyZNNJpNlLCotLT106NDq1avlrg5wOMz4ArU3ffr0+Pj48s/UjI2N/fzzz1NTUydMmPCXv/zF19dXxvIAKIEkSREREdu3by8tLS2/3dPT8+zZs3zQMVAeM75ALc2bN2/FihXlU69Go4mJienYsWNqaupnn31G6gVgB++9995XX31VIfUKIe7cuTNr1ixZSgIcFjO+QG2sXr168uTJlberVKqNGzeOHj3a/iUBUKC///3vQ4cOtfFf+d69ewcOHGjPkgBHRvAF7tnXX389dOjQKj811MXFxd/f/+zZs66urvYvDICi/Pbbb/fdd19eXl51/5Wr1erAwMBTp04xIgEW6nnz5sldA+BMjhw58sc//rG0tLTK/2kkSTKZTB4eHv369bN/bQAUxWg0jhkzxs/P79y5c9nZ2VqttvwzfYUQkiTl5+e7uLgMGDBAphoBx8KML3APLly48MADD+Tk5FReTmfl4uLi7u5+/vz5Zs2a2bM2AEqWkpKybt261atX37p1S6PRlB+jNBrN8ePHO3XqJGN5gIMg+AI1lZmZ2adPnytXrpSUlFg3uri4uLi4WP6P8fDw6NKlS8+ePbt16/bUU08FBATIVywAJTKbzXv37l2/fv3WrVvv3LljGZ3UavVDDz30/fff81hf4L+Cb3p6+k8//SRjNYDDKi4ufuedd86ePevi4iJJkiRJarXaz88vJCQkKCgoKCgoMDDQx8dH7jLhEAIDAx966KE6NnLgwIErV67USz1QoOLi4sOHD//www/Hjh2zvCFh+vTpjzzyiNx1AfYWFhb2X/NQUjkJCQnyFQYAjUR4eLhUZ+Hh4XK/DgBwegkJCeWHVk3lI1j8gMZEpVIlJCRERkbWpZGioiKz2WwwGOqrKjRi9fhBfeHh4UlJSfXVGhTu0qVLeXl5Xbp0kauAxMTE0aNHkzFgT5WX91QRfAFUoNfr5S4BAOokODhY7hIA+fHJbQAAAFAEgi8AAAAUgeALAAAARSD4AgAAQBEIvgAAAFAEgi8AAAAUgeALAAAARSD4AgAAQBEIvgAAAFAEgi8AAAAUgeALAAAARSD4AgAAQBEIvgAAAFCE+gy+S5Ysad68uUqlWrFihe0jH3jgAbVa3aNHj4ZoHJVV7r1//OMfXl5eO3furHvjZWVlsbGxYWFh1R0wefJkT09PlUr166+/1rrBRYsWhYaGurm5ubu7h4aG/vnPf87Nza1r6f/tzJkzL730UufOnY1Go06n8/X1DQ0NHTly5FdffSVkvb03bNigUqls9LAj2LJlS0hIiOp3Wq22VatWY8eOPXXqVK3btPN9W+ElqFQqnU7XvHnzAQMGLF68OCsrq+4ntScGZIfVCAbk+fPn33fffUajUa/Xt2vX7s0338zPz69r6f+NAbkuGJBtkcpJSEiosOVepaWlCSE+/fTTux45cODA7t27N1DjqKxC7+3atctoNO7YsaOOzZ45c6Zfv35CCNtXc+PGjUKIo0eP1rrBIUOGLFmyJCMjIy8vLzExUavVPv744zWpUAiRkJBw18PWrFmj0+n69++/e/furKysO3funDt3bufOnUOGDImOjrYcI9ftPWTIkLZt2woh0tLS7qlN+2vbtq2Xl5ckSfn5+Tt27AgKCvLw8EhNTa11g/a/b60voaysLCsr65///Of48eNVKpW/v/+hQ4dq0nh4eHh4eHgdK6yXdhiQHZazD8iPPPJIXFxcZmZmbm5uQkKCVqv94x//eNfWap4xGJDrBQOyVFUGkHOpg0qlkvHsCjdkyJCcnJyhQ4fWpZFjx47Nnj172rRp9/TLdO0a1Ol0M2bM8PX19fDwiIiIGD58+Lfffnvt2rV6Oe/BgwcnT54cFhb2z3/+88knn/T29tbr9SEhIU899dQnn3xSuzbr6/bOzMw8efLkO++8I4RYu3ZtvbRpB+7u7kOHDv3444/z8/OXLl1aX83a875VqVTe3t4DBgxYs2ZNYmLijRs3LGevy6kdGQOyjJxuQPbw8IiOjm7atKmnp2dkZOSIESN279595cqVejkvA3K9Y0Aurx6CryRJSUlJ8fHx9/qNWq227md3WLXuFgdX/nV17959y5YtY8eO1ev1tr+rhoOOjQa3bt3q6upq/bJVq1ZCiPr649p7771nNpsXLlyo0Wgq7AoJCandH3Pr6/ZOTEwcMmTIsGHDXF1d161bZ/n91f5qdz/36dNHCHHixImGKeoe1O6+tQoPDx8/fnxGRobj/2WfAblKDMjl1X1A3rVrl1qttn7ZrFkzIYTJZLq36qvBgHxXDMh1GZBrE3zNZvOCBQs6duzo5ubWrFmzNm3aLFiwIDIysvKRkiR9+OGHnTp10uv1TZo0GT58eGpqqnXv2bNnQ0ND3d3d3dzcHn744X//+9+W7T/88MN9993n5eXl6uratWvXb7755l4rrLKFTp06qVQqFxeX3r17W34+33zzTcsxf/vb38xm89y5c4OCgtzc3Lp162b5i8wHH3xgMBg8PT0zMjJee+21Vq1anT59urrybHRLlY3b8Mknn7i6ujZv3vyFF17w9/d3dXUNCwv7+eef79qrtjvc6t///ndQUJBKpVq2bJkQYvny5e7u7gaDYfv27YMHDzYajQEBAZa/hdl+XTZIkrR48eKOHTvq9XovL6833nijRleuxtLS0ry9vYODg+veVHFx8d69e5s2bdq3b997+kb73N4bNmwYOXKkp6fnE088cfHixR9++MGy3Snu59LSUiGEZThzivvWhvHjxwshvv7661q30EAYkBmQZR+Qf/vtNzc3tzZt2tS9KQZkBuSaqNOAXH7dQw3X37z33ntqtXr79u0mk+nIkSMtWrQYMGCAZVeF9R9z587V6XTr1q3Lzs4+fvx4r169mjVrdv36dUmSBg4cGBIScuHChZKSkhMnTjz44IOurq5nzpyx/B4wb96827dvZ2Zm9u3b18fHp8rGbaiyhdLS0tatWwcFBZWWllqPfPXVV2NjYyVJev311/V6/ebNm7Oyst566y0XFxfL8pE5c+YIIV5++eWlS5eOHDny1KlT1ZVno1uqa9yG6Ohod3f3kydP3rlzJyUl5YEHHvD09Lx8+bLtXrWxq0LvWf4mtXTpUsuXlpf53Xff5eTkZGRkPPzww+7u7sXFxbZfl9WDDz5YYWnOnDlzVCrVX//616ysLJPJFBcXJ2q2pKy6Bi2Ki4vT09OXLl2q1+stv23flbjbGt8zZ84IIfr27XvXpux/e1+6dMnX19dyx65bt04IMWnSJMsux7yfreuxLCw1v/HGG7a7y3Hu28ovwcryZsrAwMBq7o7/Y+c1vgzIDMgVqrXbgGxRUFDg6ek5c+bMu7ZTk4zBgMyAXF7dB+TKGaA2wfeBBx7o06eP9cupU6e6uLgUFRVJ/90pJpPJw8MjKirKeuR//vMfIcT8+fOlSovNjx8/LoR4/fXXK5xrwYIFQoiMjAyptu+lKN9CbGysECIxMdGyq6CgICgoKCcnp7Cw0GAwWEs1mUx6vX769OnS7xeysLDwro1X1y02GrchOjq6/MU+dOiQEOKdd96x0au2O7wm96v1ZVqGxbNnz9p4XeWrrXC/mkwmg8FQ/s1nNX8vRZUNWrVo0UII4ePj8/HHH1t+nO7qrsH38OHDQohBgwbdtSn7394LFy6cMGGC5d85OTl6vd5oNJpMJssWB7yfy7+XYvPmzS1atGjevHl6erpT3LcVXkJllkVmVe4qz87BlwG5ysYZkK1fNtyAbK22Q4cOubm5d22nJhmDAbnKehiQK6vhgFw5A9RmqcOdO3ekcutazGazVqstv9zHIiUlJT8///7777dueeCBB3Q6nfUvROV17drVy8vLcjuWZ1mXYzaba1Fn5RYmT57s5eX10UcfWXatX79++PDhRqPx9OnTJpOpS5culu1ubm5+fn5Vzu3baLy6bql14+Xdf//9BoMhNTXVRq/eU4fbptPphBAlJSU2XpeNbz979qzJZBo4cOC9nveurly5kpGRsWHDhi+++KJnz54ZGRl1b9PDw0MIUVBQUGF7YmJimzZtLE9R6dSpU4Vz2ef2tvxZzfJvo9H4xBNP5Obmbt++3bLFMe/nnJwclUrl5eX18ssv/+lPf/rPf/7TqlUrp7hvbSsoKJAkyWg01rqFBsKAXGXjDMhWDTcgCyG2bt2amJj4zTffeHp61kuDDMhV1sOAXEFdBuTaBN8//elPR44c2b59e2Fh4eHDh7dt2/bUU09VfgHZ2dni95vYytvbOy8vr8pmtVqtpYP+/ve/DxgwwNfXV6/Xv/nmm7WosLoWPDw8pk6d+tNPP1l+g/n0009nzpwpfv8Ze/vtt62Pi7t06VJ16/Sra7y6brmnxm3Q6/U3b9600av32uE1VMPLXV56eroQwtfXty7nrZJWq/X19X3iiSc2bdqUkpJi+YW4joKDg/V6/dmzZytsj4yMvHDhQnBwcIsWLU6dOtW8efPye+1we584cSI5OXno0KHWO8fyuETrW4kd8362/HZeWlqanp7++eefW9ZhO8V9a5vlL7ChoaF1qaohMCAzIMs1IG/atOn999//17/+1bp16/pqkwGZAbkm6jIg1yb4zps377HHHhs/frzRaBw5cmRkZOTKlSsrH+bt7S2EqNBZ2dnZAQEBlQ8uLS29fft2UFDQ5cuXR4wY4efn9/PPP+fk5CxatOhey7PdwsyZM7VabWxs7Pfffx8YGGh5Gp9lRLCsxbE6cODAPTVeXbfUvHEbSkpKLF1no1fvqcNrroaXuzzL4xeKiorqcl7b2rVrp1arU1JS6t6Uq6vroEGDbt68efDgwZp/lx1u7y+//HLMmDHlb5vbt2+7ubnt2bPn+vXrlmOc5X52ivvWtt27dwshBg8eXJdGGgIDMgOy7W9poAF56dKl69ev37dvX8uWLeuxWQZkBuSaqMuAXPFZITWRkpJy7ty5mzdvVn7USHldunTx8PCwrNex+Pnnn4uLi3v37l354H/+859lZWW9evVKTk4uKSmZPn16SEiIqNWz92y3EBAQEBkZmZCQcPXq1f/5n/+xbAwMDHR1da3Jx9jYaLy6bql54zb861//kiSpb9++Nnr1njq85mp4ucvr0qWLi4vL/v37p02bVpdTW2VmZr700ksbNmywbklLSzObzYGBgfXS/jvvvLNnz5433nhj3759NXzqTUPf3pIkbdq0af369eU3NmnSJCIiYu3atRs2bJg1a5ZwnvvZKe5bG65fvx4bGxsQEDBx4sS6t1a/GJAZkG2r9wFZkqTZs2dnZWVt27atXn6+KmBAZkC2rY4Dcm1mfF988cWgoKC7PkLV1dX1tdde27p16/r163Nzc5OTk6dNm+bv7x8dHW05oLi4OCcnp7S09Jdffpk5c2ZwcPD48eODgoKEEHv37r1z505aWlotlpXctYXXXnuttLQ0Kyvrscces5Y6YcKEjRs3Ll++PDc312w2p6enV/nhCDYar65bat54BZbPKSktLT1+/Pgrr7wSFBQ0fvx4G7161w6vnRpe7vJ8fX3Dw8M3b968evXq3Nzc48eP1/Hxme7u7nv27Nm3b19ubm5JScnRo0eff/55d3d3y1hTd7179163bt2RI0cGDBiwe/fua9eulZaWXrp0ad26dbdv367yWxr69v7pp5+MRqPlw2zKs/zXVf7B6U5xPzvFfWslSVJ+fn5ZWZkkSTdv3kxISOjXr59ard62bZsDrvFlQGZAtq3eB+STJ09+8MEHK1eu1Gq15T9RdsmSJXVp1ooBmQG5vPofkMtPj9fwqQ779u3z8fGxtqDVajt16rRly5a//vWvljfdu7u7jxw5UpKksrKyxYsXt2/fXqvVNmnSZMSIEadPn7Y0smbNmkcffbR58+YajcbHx2fMmDGXLl2y7IqJiWnatKm3t3dERITl+XBt27Z95ZVXKjRuQ5UtWB49Y/Hoo4+uWrWq/LcUFRXFxMQEBQVpNBpfX99Ro0alpKQsWrTIzc1NCBEYGGh9eFZ1jVfXLdU1bvslREdHWz5cW6PRGI3G4cOHnzt3zrLLRq9Wt6vCpVm6dKmfn58QwmAwDBs2LC4uzmAwCCHat29/7ty5+Ph4y80UHBx85swZG6/rwIED/fr18/f3t+zy8/MLCwvbv3+/JEl5eXlTpkzx8fHx8PDo37//3LlzhRABAQHHjh2z8aptNDhs2LA2bdp4eHjo9fq2bdtGRUUlJyfb7kMLUbOPLJYk6cKFC6+88krnzp3d3d1dXV3btGnz8MMPz549+/vvv6/ch7YvRB1v70mTJrm7u2s0mu7du//yyy/WCt99911r57Rq1SouLs6y3RHu5x9//LFDhw6Wg/39/SMiIip0r+Pftzt27OjWrZvBYNDpdC4uLuL3zwrq06fP/PnzMzMza3IXSXZ/qgMDMgOynQfk5ORkUZXFixfb7saaf2SxxIDMgFxPA7Kol8eZxcXFvfLKK+Wv6KuvvqrX660P9VCm+u0Wy6dB1l91tefsl7vyTY+acPbrLmP9dg6+zn6lGggDsgO6p+CL8pz6ukuy1l85A9zzYovr16/PnDmz/IISnU4XFBRUUlJSUlJi+fVFgRqiW+ryzKD6wuVWJme/7s5ef80p55XeEwZkNCbOft0drf57XuPr5uam1WpXr15948aNkpKSq1evrlq1au7cuVFRUXZb/ZaamqqqXlRUlH3KKO9eu8UBX0KV6vdyO8urhiP8mNeFs9dfc47wSh3w55oBuSac5VXDEX7M68Lh6i8//VvDP0N8//33gwYNMhqNarXay8srLCwsLi6upKSknqennU09dsv/+3//z/Lk59atWyclJdV7qffE2S+3YKlDrTj7dZexfjsvdXD2K9VAGJAdEEsdas2pr7ska/2VM4BKKvdZGomJiaNHjy6/BXB2KpUqISEhMjJS7kKgFBEREUKIpKQkB2kHcBBkDNhf5QxQm8eZAQAAAE6H4AsAAABFIPgCAABAEQi+AAAAUASCLwAAABSB4AsAAABFIPgCAABAEQi+AAAAUASCLwAAABSB4AsAAABFIPgCAABAEQi+AAAAUASCLwAAABRBU3lTYmKi/esAGs6BAwdkOW9+fr5Wq9Xr9bKcHXJJT08PCAior6YYkJWmqKiopKTEw8ND7kLqn2Uo5paGzKRyEhIS5C4HAJxeeHi4VGfh4eFyvw4AcHoJCQnlh1aVJElylwQ0TtnZ2YmJiUuXLj1x4kTv3r2nTp36zDPPNMqJHAC1dufOnZ07d8bHx+/du7d9+/aTJk2aOHGir6+v3HUBjRPBF2hwR44ciY+PX79+vVqtHjNmTHR0dK9eveQuCoDMTp48uXbt2pUrVxYUFAwbNmzq1KkDBw5UqVRy1wU0ZgRfwE4sE8DLli1LTk62TACPGTPG09NT7roA2FX5Kd4OHTpMnDiRKV7Abgi+gL1VmAB+9tln+/fvL3dRABpcSkrKunXr4uPjTSYTU7yALAi+gDxycnISEhLi4uKOHz9+3333Pffcc1OnTm3SpIncdQGoZ3l5eRs3boyPjz9y5IhlinfSpEnNmjWTuy5AiQi+gMysE8BlZWVDhw6dOnXqoEGD5C4KQD2w/HRv3LixuLiYKV7AERB8AYdgmQBevnz5sWPHLBPAU6ZMadq0qdx1Abhnlinezz777JdffunYseOECROY4gUcBMEXcCyWKaIvv/zSbDYzAQw4F8vP74YNG0pKSpjiBRwQwRdwROUngDt16vT8888zAQw4rNzc3E2bNq1YseLo0aOhoaHjx49nihdwTARfwKExAQw4MqZ4AedC8AWcgGU+6dNPP/31118tE8CTJ0/28fGRuy5AoSpP8fIjCTgFgi8fYcv9AAAgAElEQVTgTJgABuTFzyDg1Ai+gPNhtgmwM/7qAjQOBF/AiVnXF5aWllomn1hfCNQvpniBxoTgCzg93lEO1DuerAI0SgRfoPHgDeZA3THFCzRiBF+gseFTo4Ba4NMTASUg+AKNFhPAQE1YflLWr19fVlbGFC/QuBF8gUbOMgEcHx9/5MgRJoABK8sUb1xc3PHjxy1TvFOnTm3SpIncdQFoQARfQCmOHDmydu3adevWmUwmJoChZNYpXrVaPXz48Oeee44pXkAhCL6AshQWFu7atSs+Pn7v3r0dOnSYOHHixIkTfX195a4LaHDZ2dmJiYnLli1LTk7u3bv31KlTx4wZ4+npKXddAOyH4Aso1MmTJ9euXbty5cqCggImgNG4lZ/iHTNmTHR0dK9eveQuCoAMCL6Aot25c2fnzp2WCeD27dtPmjSJCWA0GpWneJ955hkPDw+56wIgG4IvACHKTQDn5+c//fTTTADDqVmmeNetW6fVaqOiopjiBWBB8AXwfypPAE+YMKF58+Zy1wXUiGWKd+nSpSdOnGCKF0BlBF8AVTh16tQXX3zBBDCcRYUp3hdeeKFnz55yFwXA4RB8AVSr/ARwu3btJk+ezAQwHEpWVlZSUtInn3ySkpLCFC+AuyL4Arg7ywTwqlWr8vLymACGI6gwxTtt2rQePXrIXRQAR0fwBVBT1gng7777rm3btmPHjp08eXJAQIDcdUFBbty4sWnTppUrV1qneMeOHevu7i53XQCcA8EXwD1LTU3929/+tmrVquzs7EcffXTq1KkjR45Uq9Vy14VGq6ysbN++ffHx8du3b3dzcxs9ejRTvABqgeALoJaKiop27NhhmQBu1arV2LFjZ8yYERgYKHddaFRu3Ljxt7/9beXKlefOnWOKF0AdEXwB1NXp06fXrFmzevXqrKwsJoBRL6xTvNu2bTMYDKNHj54+fXr37t3lrguAcyP4AqgflSeAp0+fHhQUJHddcDLXr1//4osv4uPjz58/zxQvgPpF8AVQz5gARi1UnuKdMWNGt27d5K4LQKNC8AXQIMpPALds2XLcuHFMAKNKlad4x40bZzAY5K4LQCNE8AXQsM6cOfP5559//vnnmZmZjz322NSpU0eMGKHRaOSuCzIrP8Xr7u4eGRnJFC+AhkbwBWAPlSeAp02bFhwcLHddkMG1a9fWrl372WefXbhwgSleAPZE8AVgV0wAK1blKd4XX3yxa9euctcFQEEIvgBkUFxcvH37dssEsL+//7PPPnvXCeC9e/dmZWVFRETYrUjURFJSUpMmTQYNGmTjGKZ4ATgIgi8AOaWlpa1evXrNmjW3bt2yPQE8bNiwXbt2ffDBB6+//rr960SVlixZ8uabbz711FM7duyovNc6xfvVV195eHhERka+9NJLXbp0sX+dAGBB8AUgv8oTwC+88ELr1q2tB1y9ejUoKMhsNqtUqujo6GXLlvF8NHmZzeaXXnppxYoVkiSp1erLly+3bNnSuvfq1avr1q1bsWLFxYsXLVO8zz77rJubm4wFA4AQwkXuAgBA6HS6iIiIb7/99syZM88+++yaNWvatm37+OOPJyUllZaWCiE+//xzlUolhJAkaeXKlU888URubq7cVStXQUHBsGHDPvvsM8vUiYuLy5o1a4QQZWVle/fujYyMDA4OXrRo0RNPPHHixInDhw9PnTqV1AvAETDjC8DhFBcXb9u2LT4+ft++fa1atZowYcKqVauuXbtmPUCr1bZv3/6bb74JCAiQsU5lun79+uDBg0+cOGH5ncSiRYsWU6dOXbNmzW+//WZZsjJ8+HCdTidjnQBQGcEXgOM6e/bsqlWr4uPjs7KyKuzSarVNmzbdvXt3jx49ZKlNmVJSUp588smMjIySkpIKuzw9PceMGTNz5szOnTvLUhsA3BXBF4Cje/rpp//xj3+Un1+00Gg0Op1uy5Ytf/zjH2UpTGn27dv39NNP37lzp8prMWTIkG3btslSGADUEMEXgEO7fv16QECA2Wyucq+Li4tKpfr000+nTJli58KUZu3atZMmTZIkqbprUfktbgDgaHhzGwCHZn1bW5XKysrMZvPUqVPnzZvHr/EN5+OPPx4/frzZbK4u9QohVCrVF198Yc+qAOBeMeMLwHGVlZW1bt06PT29JiPVc889t2rVKq1Wa4fClKOkpGTy5Mlr166965EqlSogIODixYsuLkypAHBQBF8AFUVERGzevFnuKoA6CQ8PT0pKkrsKAI6lio9HAoC+ffu++uqrcldRI4WFhaWlpSaTqaioqLS01N3dvUWLFnIXVRsHDhz46KOPEhIS5C7k/9y4caOgoECj0ej1eoPBoNFonOVxvLGxsXKXAMAREXwBVCEgICAyMlLuKhTno48+otvrBXO9AKrESiwAAAAoAsEXAAAAikDwBQAAgCIQfAEAAKAIBF8AAAAoAsEXAAAAikDwBQAAgCIQfAEAAKAIBF8AAAAoAsEXAAAAikDwBQAAgCIQfAEAAKAIBF8AAAAoAsEXAAAAikDwBYD6V1JSsmDBgnbt2ul0Om9v7y5duly8eLEe2z99+vRLL73UuXNnT09PjUbj5eXVoUOHIUOGHDhwoB7PAgCNDMEXAOrf6NGj165d++WXX5pMplOnTrVt2zY/P7++Gl+9enXXrl2PHz/+4YcfXrlypaCg4OjRo++++252dnZycnJ9nQUAGh+N3AUAQG0UFhYOHDjwp59+csCWN23atG3btmPHjnXt2lUI4e/vv3379voq7+DBg9HR0Y888sg333yj0fzvGB4SEhISEuLt7Z2WllZfJ7orR74EAFAlgi8Ap7R69eqMjAzHbPnTTz/t1auXJfXWu7/85S9ms3nhwoXW1Gv15JNPPvnkkw1x0io58iUAgCqx1AFA7a1bt+7+++93dXV1d3dv3br1u+++K4SQJOnDDz/s1KmTXq9v0qTJ8OHDU1NThRDLly93d3c3GAzbt28fPHiw0WgMCAjYuHGj7dZ++OGH++67z8vLy9XVtWvXrt98840Q4pVXXnnttdfOnTunUqnatWsnhDCbzXPnzg0KCnJzc+vWrVtCQsJdz1iXlm0oLi4+ePBgjx49GqC/RXFx8Xfffefj49OnTx8bhyn8EgBAtSQA+G/h4eHh4eF3PSw2NlYIsXDhwszMzNu3b3/22Wdjx46VJGnu3Lk6nW7dunXZ2dnHjx/v1atXs2bNrl+/LknSnDlzhBDfffddTk5ORkbGww8/7O7uXlxcbKO1pKSkefPm3b59OzMzs2/fvj4+Ppazjxo1qm3bttZiXn/9db1ev3nz5qysrLfeesvFxeXQoUO2z1jHlqtz4cIFIUSPHj0GDBjg5+en1+tDQ0OXLVtWVlZmuz8tec72MWfOnBFC9O3b1/ZhCr8EUo3vYQBKQ/AFUFFNQkNxcbG3t/ejjz5q3VJaWvrRRx+ZTCYPD4+oqCjr9v/85z9CiPnz50u/Z6DCwkLLrri4OCHE2bNnq2utwkkXLFgghMjIyJD+OxsVFhYaDAbrSU0mk16vnz59uo0z1r3l6ljeXvb444//+OOPmZmZ2dnZs2fPFkKsX7/edpfWJPgePnxYCDFo0CAbx3AJJIIvgGqw1AFAbRw/fjw7O7v8ilK1Wv3yyy+npKTk5+fff//91u0PPPCATqf7+eefKzei0+mEECUlJdW1VuF4rVYrhDCbzRW2nz592mQydenSxfKlm5ubn5+f5Y/71Z2x3lu20uv1QojOnTuHhYU1bdrUy8vrnXfe8fLyio+Pt/FdNeTh4SGEMJlMNo7hEgBAdQi+AGojNzdXCOHt7V1he3Z2tvg9n1l5e3vn5eXVojUhxN///vcBAwb4+vrq9fo333yzym8vKCgQQrz99tuq3126dMl2Omy4lv39/YUQt27dsm7R6XTBwcHnzp2zXU9NtG7d2tXV1bLgoTpcAgCoDsEXQG20bNlS/He8s7AkpwoZKzs7OyAgoBatXb58ecSIEX5+fj///HNOTs6iRYuq/HZfX18hRGxsbPm/Z9n+KIeGa9nDw6N9+/YnT54sv7G0tNTLy8vGd9WQXq9/8sknb9269eOPP1bee/v27cmTJ3MJAKA6BF8AtdG6deumTZvu2bOnwvYuXbp4eHhYlqJa/Pzzz8XFxb17965Fa8nJySUlJdOnTw8JCXF1dVWpVFV+e2BgoKur66+//lrz+huuZSHE6NGjjx49ev78ecuXJpPp0qVL9fV0s3nz5un1+lmzZhUWFlbYdeLECY1GwyUAgOoQfAHUhl6vf+utt77//vuZM2f+9ttvZWVleXl5J0+edHV1fe2117Zu3bp+/frc3Nzk5ORp06b5+/tHR0fXorWgoCAhxN69e+/cuZOWllZ+lWrTpk2vXr168eLFvLw8tVo9YcKEjRs3Ll++PDc312w2p6enX7t2zcYZG65lIcSsWbOCg4PHjx9/+fLlzMzMmJiYwsJCy1vc6q5Hjx5ffvnliRMnHn744X/84x85OTklJSUXLlxYuXLlpEmTtFotlwAAqtVw75sD4KRq/o74ZcuWde3a1dXV1dXVtWfPnnFxcZIklZWVLV68uH379lqttkmTJiNGjDh9+rQkSXFxcQaDQQjRvn37c+fOxcfHG41GIURwcPCZM2eqay0mJqZp06be3t4RERHLli0TQrRt2/by5cu//PJLcHCwm5tb//79r1+/XlRUFBMTExQUpNFofH19R40alZKSYvuMdWn5rj1z5cqVMWPGNGnSRK/X9+nT5+uvv77rt9TkqQ5Wly9ffv3117t27erh4aFWq729vXv27Dlp0qQff/yRSyDxVAcA1VBJkiRL4AbgsCIiIoQQSUlJcheiLImJiaNHj2ZMrhfcwwCqxFIHAAAAKALBFwDuTWpqqqp6UVFRchcIAKiaRu4CAMDJhIaGsiABAJwRM74AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFEEjdwEAHNHmzZtVKpXcVSgR3V5fwsPD5S4BgMNRSZIkdw0AHMuBAweuXLkidxWNWWxsrBDi1VdflbuQxiwwMPChhx6SuwoAjoXgCwD2FhkZKYRITEyUuxAAUBbW+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFIHgCwAAAEXQyF0AADR+JpOpqKjI+mVxcbEQIisry7pFr9cbDAYZKgMAJVFJkiR3DQDQyMXFxb344os2Dli2bNmMGTPsVg8AKBPBFwAa3M2bN/39/c1mc5V71Wr1tWvXfH197VwVACgNa3wBoMH5+vo+9thjarW68i61Wj1w4EBSLwDYAcEXAOxh3LhxVf6FTZKkcePG2b8eAFAgljoAgD3k5eX5+vqWf4ubhU6nu3nzptFolKUqAFAUZnwBwB48PT2feuoprVZbfqNGoxk2bBipFwDsg+ALAHYyduzY0tLS8lvMZvPYsWPlqgcAlIalDgBgJ8XFxc2aNcvLy7Nu8fDwuHXrll6vl7EqAFAOZnwBwE50Ol14eLhOp7N8qdVqIyMjSb0AYDcEXwCwn2eeecbysW1CiJKSkmeeeUbeegBAUVjqAAD2U1ZW1qJFi1u3bgkhfHx8bty4UeXDfQEADYEZXwCwHxcXl7Fjx+p0Oq1WO27cOFIvANgTwRcA7GrMmDHFxcWscwAA+9PIXQAACCHEgQMHPvzwQ7mrsBODwSCEWLx4sdyF2MmsWbMeeughuasAAGZ8ATiGK1eubN68We4q7CQ4ODg4ONj2MQcPHjx48KB96mlQmzdvvnLlitxVAIAQzPgCcChJSUlyl2APKSkpQojOnTvbOCYiIkI0ig5RqVRylwAA/4vgCwD2ZjvyAgAaCEsdAAAAoAgEXwAAACgCwRcAAACKQPAFAACAIhB8AQAAoAgEXwAAACgCwRcAAACKQPAFAACAIhB8AQAAoAgEXwAAACgCwRcAAACKQPAFAACAIhB8AQAAoAgEXwDOavLkyZ6eniqV6tdff5W7FiGEKCsri42NDQsLK79x/vz59913n9Fo1Ov17dq1e/PNN/Pz8+vrjFu2bAkJCVGVo9PpmjdvPmDAgMWLF2dlZdXXiQCgcSD4AnBWq1atWrlypdxV/K+0tLQ//OEPs2bNMplM5bfv27fvxRdfvHjx4q1btxYsWPDRRx9FRETU10lHjRp1/vz5tm3benl5SZJUVlaWkZGRmJjYpk2bmJiYzp07Hz58uL7OBQCNAMEXAOrq2LFjs2fPnjZtWo8ePSrs8vDwiI6Obtq0qaenZ2Rk5IgRI3bv3n3lypWGKEOlUnl7ew8YMGDNmjWJiYk3btwYMmRITk5OQ5wLAJwRwReAE1OpVHKXIIQQ3bt337Jly9ixY/V6fYVdu3btUqvV1i+bNWsmhKgwK9wQwsPDx48fn5GRsWLFioY+FwA4C4IvAGciSdLixYs7duyo1+u9vLzeeOMN6y6z2Tx37tygoCA3N7du3bolJCQIIZYvX+7u7m4wGLZv3z548GCj0RgQELBx40bLt+zfv79Pnz4Gg8FoNHbt2jU3N7e6durLb7/95ubm1qZNm3psszrjx48XQnz99dfCSToHABoawReAM/nzn/8cExMTHR1948aN69evz54927pr9uzZH3zwQWxs7LVr14YOHfrMM88cPnx4+vTpr776amFhoaenZ0JCwrlz50JCQqZMmVJSUlJQUDBs2LDw8PDbt2+npaV16NChuLi4unbqpXiTybRv374pU6bodLp6adA2y7qL8+fPC2foHACwBwkAHIBl7tD2MSaTyWAwPP7449YtlunJo0ePFhYWGgyGqKgo65F6vX769OmSJM2ZM0cIUVhYaNkVFxcnhDh79uyJEyeEELt27Sp/Chvt1MSDDz7YvXv36vbOmTOnQ4cOubm5NWkqPDw8PDy8Jkda39xWmWXVr7ydI4RISEioyQsBgIbGjC8Ap3H27FmTyTRw4MDKu06fPm0ymbp06WL50s3Nzc/PLzU1tfKRltnWkpKSkJCQ5s2bjxs3bt68eRcvXrzXdu7V1q1bExMTv/nmG09Pz7q3VhMFBQWSJBmNRsfvHACwD4IvAKeRnp4uhPD19a28q6CgQAjx9ttvW59oe+nSJdvvIXNzc9u3b1///v3fe++9kJCQqKiowsLCWrRTE5s2bXr//ff/9a9/tW7duo5N1dyZM2eEEKGhoQ7eOQBgNwRfAE7D1dVVCFFUVFR5lyUNx8bGlv+T1oEDB2w32Llz5507d169ejUmJiYhIWHJkiW1a8e2pUuXrl+/ft++fS1btqxLO/dq9+7dQojBgwc7cucAgD0RfAE4jS5duri4uOzfv7/yrsDAQFdX13v6CLerV6+ePHlSCOHr67tw4cJevXqdPHmyFu3YIElSTExMcnLytm3bPDw86qXNGrp+/XpsbGxAQMDEiRMds3MAwP4IvgCchq+vb3h4+ObNm1evXp2bm3v8+PH4+HjLLldX1wkTJmzcuHH58uW5ublmszk9Pf3atWs2Wrt69eoLL7yQmppaXFx89OjRS5cu9e3btxbt2HDy5MkPPvhg5cqVWq22/AcLL1mypHYNVkeSpPz8/LKyMkmSbt68mZCQ0K9fP7VavW3bNqPR6JidAwAyaJj3zAHAvanJUx0kScrLy5syZYqPj4+Hh0f//v3nzp0rhAgICDh27FhRUVFMTExQUJBGo/H19R01alRKSkpcXJzBYBBCtG/f/ty5c/Hx8UajUQgRHBz87bffhoWFNWnSRK1Wt2zZcs6cOaWlpZIkVdmO7aoOHDjQr18/f39/y7jq5+cXFha2f//+5OTkKgfexYsX3/WV1uSpDjt27OjWrZvBYNDpdC4uLuL3D2/r06fP/PnzMzMzrUfK2DmCpzoAcBgqSZLsEK8BwLbExMTRo0czIllFREQIIZKSkuQupK5UKlVCQkJkZKTchQAASx0AAACgDARfALiL1NRUVfWioqLkLhAAUCMauQsAAEcXGhrKGgwAaASY8QUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCBq5CwCA/xMRESF3CY7i4MGDgg4BgHpF8AXgEAIDA8PDw+Wuwk5OnTolhOjUqZONY/r27WuvchpWeHh4YGCg3FUAgBBCqCRJkrsGAFCWyMhIIURiYqLchQCAsrDGFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIpA8AUAAIAiEHwBAACgCARfAAAAKALBFwAAAIqgkiRJ7hoAoJH78ssvV69eXVZWZvny9OnTQoiOHTtavnRxcZk0adLYsWNlqw8AlIHgCwAN7tixYz169LBxwK+//tq9e3e71QMAykTwBQB7CA0NtUz0VtauXbu0tDQ71wMACsQaXwCwh2effVar1VbertVqJ0yYYP96AECBmPEFAHs4f/58u3btqhxy09LS2rVrZ/+SAEBpmPEFAHsICQnp2bOnSqUqv1GlUvXu3ZvUCwD2QfAFADt57rnn1Gp1+S1qtfq5556Tqx4AUBqWOgCAnWRkZPj7+1sfaiaEcHFx+e233/z8/GSsCgCUgxlfALCT5s2b/+EPf7BO+qrV6kceeYTUCwB2Q/AFAPt59tlnbXwJAGhQLHUAAPvJzc1t1qxZSUmJEEKr1WZkZHh7e8tdFAAoBTO+AGA/RqNx8ODBGo1Go9H86U9/IvUCgD0RfAHArsaNG2c2m81m89ixY+WuBQCURSN3AQAghBDp6ek//fST3FXYQ0lJiU6nkySpqKgoMTFR7nLsISwsLCAgQO4qAIA1vgAcQ2Ji4ujRo+WuAg0iISEhMjJS7ioAgBlfAI5EIb+K7969W6VSPfnkkzaOiYiIEEIkJSXZq6iGUuHD6gBARgRfALC3QYMGyV0CACgRwRcA7E2jYewFABnwVAcAAAAoAsEXAAAAikDwBQAAgCIQfAEAAKAIBF8AAAAoAsEXAAAAikDwBQAAgCIQfAEAAKAIBF8AAAAoAsEXAAAAikDwBQAAgCIQfAEAAKAIBF8Azmry5Mmenp4qlerXX3+VuxYhhCgrK4uNjQ0LCyu/cdGiRaGhoW5ubu7u7qGhoX/+859zc3Pr64xbtmwJCQlRlaPT6Zo3bz5gwIDFixdnZWXV14kAoHEg+AJwVqtWrVq5cqXcVfyvtLS0P/zhD7NmzTKZTOW3//DDD1OmTLl8+fKNGzfefffdRYsWhYeH19dJR40adf78+bZt23p5eUmSVFZWlpGRkZiY2KZNm5iYmM6dOx8+fLi+zgUAjQDBFwDq6tixY7Nnz542bVqPHj0q7NLpdDNmzPD19fXw8IiIiBg+fPi333577dq1hihDpVJ5e3sPGDBgzZo1iYmJN27cGDJkSE5OTkOcCwCcEcEXgBNTqVRylyCEEN27d9+yZcvYsWP1en2FXVu3bnV1dbV+2apVKyFEfn5+Q5cUHh4+fvz4jIyMFStWNPS5AMBZEHwBOBNJkhYvXtyxY0e9Xu/l5fXGG29Yd5nN5rlz5wYFBbm5uXXr1i0hIUEIsXz5cnd3d4PBsH379sGDBxuNxoCAgI0bN1q+Zf/+/X369DEYDEajsWvXrpbVt1W2U1/S0tK8vb2Dg4Prsc3qjB8/Xgjx9ddfCyfpHABocBIAOABLhLrrYXPmzFGpVH/961+zsrJMJlNcXJwQ4ujRo5Ikvf7663q9fvPmzVlZWW+99ZaLi8uhQ4cs3yKE+O6773JycjIyMh5++GF3d/fi4uL8/Hyj0bho0aLCwsLr16+PHDny5s2bNtqpiQcffLB79+6VtxcXF6enpy9dulSv169bt64mTYWHh4eHh9fkSOsa3wosUTUwMFCStXOEEAkJCTV5IQDQ0Ai+ABxCTYKvyWQyGAyPP/64dYtlevLo0aOFhYUGgyEqKsp6pF6vnz59uvR7tissLLTssmTls2fPnjhxQgixa9eu8qew0U5NVBd8W7RoIYTw8fH5+OOPi4uLa9JU3YOvJEmWVb/ydg7BF4DjYKkDAKdx9uxZk8k0cODAyrtOnz5tMpm6dOli+dLNzc3Pzy81NbXykTqdTghRUlISEhLSvHnzcePGzZs37+LFi/fazj25cuVKRkbGhg0bvvjii549e2ZkZNSxwZooKCiQJMloNDp45wCA3RB8ATiN9PR0IYSvr2/lXQUFBUKIt99+2/pE20uXLlV4slgFbm5u+/bt69+//3vvvRcSEhIVFVVYWFiLdmpCq9X6+vo+8cQTmzZtSklJWbBgQR0brIkzZ84IIUJDQx28cwDAbgi+AJyG5fEIRUVFlXdZ0nBsbGz5P2kdOHDAdoOdO3feuXPn1atXY2JiEhISlixZUrt2aq5du3ZqtTolJVBPSWoAAALaSURBVKW+GrRh9+7dQojBgwc7S+cAQEMj+AJwGl26dHFxcdm/f3/lXYGBga6urvf0EW5Xr149efKkEMLX13fhwoW9evU6efJkLdqxITMz85lnnim/JS0tzWw2BwYG1kv7Nly/fj02NjYgIGDixImO2TkAYH8EXwBOw9fXNzw8fPPmzatXr87NzT1+/Hh8fLxll6ur64QJEzZu3Lh8+fLc3Fyz2Zyenm77cyKuXr36wgsvpKamFhcXHz169NKlS3379q1FOza4u7vv2bNn3759ubm5JSUlR48eff75593d3WfNmlW7BqsjSVJ+fn5ZWZkkSTdv3kxISOjXr59ard62bZvRaHTMzgEAGTTQm+YA4J7U8HFmeXl5U6ZM8fHx8fDw6N+//9y5c4UQAQEBx44dKyoqiomJCQoK0mg0vr6+o0aNSklJiYuLMxgMQoj27dufO3cuPj7eaDQKIYKDg7/99tuwsLAmTZqo1eqWLVvOmTOntLRUkqQq27Fd1YEDB/r16+fv728ZV/38/MLCwvbv3y9J0rBhw9q0aePh4aHX69u2bRsVFZWcnFyTDqnJUx127NjRrVs3g8Gg0+lcXFzE7x/e1qdPn/nz52dmZlqPlLFzBE91AOAwVJIkyZO4AaCcxMTE0aNHMyJZRURECCGSkpLkLqSuVCpVQkJCZGSk3IUAAEsdAAAAoAwEXwC4i9TUVFX1oqKi5C4QAFAjGrkLAABHFxoayhoMAGgEmPEFAACAIhB8AQAAoAgEXwAAACgCwRcAAACKQPAFAACAIhB8AQAAoAgEXwAAACgCwRcAAACKQPAFAACAIhB8AQAAoAgEXwAAACgCwRcAAACKQPAFAACAIhB8AQAAoAgauQsAgP+TmJgodwmOIj09XdAhAFCvCL4AHMjo0aPlLsGx0CEAUI9UkiTJXQMAAADQ4FjjCwAAAEUg+AIAAEARCL4AAABQBIIvAAAAFOH/A66rBvORPfVxAAAAAElFTkSuQmCC\n",
+            "text/plain": [
+              "<IPython.core.display.Image object>"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 61
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Model Compilation Explanation\n",
+        "\n",
+        "The code snippet is a crucial step in preparing the neural network model for training. The `compile` method is used to configure the model with the necessary settings for the training process.\n",
+        "\n",
+        "The `optimizer` parameter is set to 'adam', which refers to the Adam optimization algorithm. Adam is an adaptive learning rate optimization algorithm designed to handle sparse gradients on noisy problems. It's a popular choice due to its efficiency and minimal memory requirement.\n",
+        "\n",
+        "The `loss` parameter is set to 'binary_crossentropy', which is the loss function used for binary classification problems. Crossentropy loss measures the performance of a classification model whose output is a probability value between 0 and 1. Binary crossentropy loss is used when there are only two label classes (labels are supposed to be 0 and 1).\n",
+        "\n",
+        "Lastly, the `metrics` parameter includes 'accuracy' as a metric to be evaluated by the model during training and testing. Accuracy calculates how often predictions equal labels, providing a measure of how many instances were correctly classified.\n",
+        "\n",
+        "By compiling the model with these parameters, we are essentially setting up the model for the backpropagation process: the optimizer will minimize the loss function, improving the model's accuracy over iterations of training."
+      ],
+      "metadata": {
+        "id": "jG9T9W98koQb"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Compile the model\n",
+        "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])"
+      ],
+      "metadata": {
+        "id": "u6H_Fuo7jY4L"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Model Training Explanation\n",
+        "\n",
+        "The code snippet provided is used to train the neural network model with synthetic data. The `fit` method of the `model` object is used to feed the data into the model and to perform the training process.\n",
+        "\n",
+        "- The first argument to `model.fit` is a list containing `fake_query_data` and `fake_value_data`, which are the input features for the model. These should be numpy arrays or tensors containing the input data. In this context, it appears that these arrays represent encoded sequences of data that the model will learn from.\n",
+        "\n",
+        "- The second argument, `fake_labels`, represents the target outputs that the model should learn to predict. These labels are associated with the input data and in binary classification tasks, such as the one implied by the preceding code, they would typically consist of 0s and 1s indicating the class of each sample.\n",
+        "\n",
+        "- `batch_size=batch_size` tells the `fit` method how many samples to process before updating the model's weights. `batch_size` is a hyperparameter that defines the number of samples to work through before updating the internal model parameters. Here, it is set to the variable `batch_size` which was defined earlier in the code (not shown in this snippet but typically would be an integer value).\n",
+        "\n",
+        "- `epochs=10` specifies the number of epochs to run the training for, which is the number of complete passes through the entire training dataset. An epoch is one complete presentation of the data set to be learned to a learning machine. Running for 10 epochs means that the entire dataset will be passed through the neural network a total of 10 times.\n",
+        "\n",
+        "By executing this `fit` method, the model will start the training process, adjusting its weights on the synthetic data for 10 iterations over the dataset, optimizing the loss function specified during the compilation step, and trying to improve the accuracy metric that was also set previously."
+      ],
+      "metadata": {
+        "id": "_ocYyJQclAkl"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "# Fit the model with fake data\n",
+        "model.fit(\n",
+        "    [fake_query_data, fake_value_data],\n",
+        "    fake_labels,\n",
+        "    batch_size=batch_size,\n",
+        "    epochs=10)"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "G7W5L67GjcJG",
+        "outputId": "bd05899d-0dd3-47c9-f38b-30333d7bbd36"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Epoch 1/10\n",
+            "32/32 [==============================] - 1s 17ms/step - loss: 0.6938 - accuracy: 0.4830\n",
+            "Epoch 2/10\n",
+            "32/32 [==============================] - 1s 17ms/step - loss: 0.6853 - accuracy: 0.6190\n",
+            "Epoch 3/10\n",
+            "32/32 [==============================] - 1s 28ms/step - loss: 0.5792 - accuracy: 0.7670\n",
+            "Epoch 4/10\n",
+            "32/32 [==============================] - 1s 24ms/step - loss: 0.3071 - accuracy: 0.9020\n",
+            "Epoch 5/10\n",
+            "32/32 [==============================] - 1s 21ms/step - loss: 0.1207 - accuracy: 0.9710\n",
+            "Epoch 6/10\n",
+            "32/32 [==============================] - 1s 32ms/step - loss: 0.0593 - accuracy: 0.9900\n",
+            "Epoch 7/10\n",
+            "32/32 [==============================] - 1s 24ms/step - loss: 0.0207 - accuracy: 0.9980\n",
+            "Epoch 8/10\n",
+            "32/32 [==============================] - 1s 25ms/step - loss: 0.0079 - accuracy: 1.0000\n",
+            "Epoch 9/10\n",
+            "32/32 [==============================] - 1s 31ms/step - loss: 0.0034 - accuracy: 1.0000\n",
+            "Epoch 10/10\n",
+            "32/32 [==============================] - 1s 17ms/step - loss: 0.0021 - accuracy: 1.0000\n"
+          ]
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": [
+              "<keras.src.callbacks.History at 0x78bed31100d0>"
+            ]
+          },
+          "metadata": {},
+          "execution_count": 60
+        }
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [],
+      "metadata": {
+        "id": "WDsYAb1_jff8"
+      },
+      "execution_count": null,
+      "outputs": []
+    }
+  ]
+}

docs/notebooks/ex14 - music generation.ipynb

@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","metadata":{"id":"uoJsVjtCMunI"},"source":["# Music Generation\n","\n","<p align='center'><img src='https://i.gifer.com/embedded/download/9B9p.gif' width=600></img></src>\n","\n","This notebook is credited to MIT Introduction to Deep Learning course. It's licensed under the MIT License and all credits are given to MIT 6.S191 - Introduction to Deep Learning, [here](http://introtodeeplearning.com)."]},{"cell_type":"markdown","metadata":{"id":"O-97SDET3JG-"},"source":["In this portion of the lab, we will explore building a Recurrent Neural Network (RNN) for music generation. We will train a model to learn the patterns in raw sheet music in [ABC notation](https://en.wikipedia.org/wiki/ABC_notation) and then use this model to generate new music. "]},{"cell_type":"markdown","metadata":{"id":"rsvlBQYCrE4I"},"source":["## Dependencies \n","First, let's download the course repository, install dependencies, and import the relevant packages we'll need for this lab."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":56443,"status":"ok","timestamp":1659045746583,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"riVZCVK65QTH","outputId":"01d6cd85-2c59-4c92-b976-297545eb7f88"},"outputs":[{"name":"stdout","output_type":"stream","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting mitdeeplearning\n","  Downloading mitdeeplearning-0.2.0.tar.gz (2.1 MB)\n","\u001b[K     |████████████████████████████████| 2.1 MB 5.2 MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (1.21.6)\n","Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (2022.6.2)\n","Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (4.64.0)\n","Requirement already satisfied: gym in /usr/local/lib/python3.7/dist-packages (from mitdeeplearning) (0.17.3)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from gym->mitdeeplearning) (1.7.3)\n","Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym->mitdeeplearning) (1.3.0)\n","Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym->mitdeeplearning) (1.5.0)\n","Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym->mitdeeplearning) (0.16.0)\n","Building wheels for collected packages: mitdeeplearning\n","  Building wheel for mitdeeplearning (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for mitdeeplearning: filename=mitdeeplearning-0.2.0-py3-none-any.whl size=2115442 sha256=68593f5b8a6175a9567a655d79ff573a21b6a96056f3114e4d8e9e42ed18bd67\n","  Stored in directory: /root/.cache/pip/wheels/9a/b9/4f/99b7c8c5c75355550b83e1fcfc02956fb40c35eb01e2262877\n","Successfully built mitdeeplearning\n","Installing collected packages: mitdeeplearning\n","Successfully installed mitdeeplearning-0.2.0\n"]}],"source":["# Import Tensorflow 2.0\n","%tensorflow_version 2.x\n","import tensorflow as tf \n","\n","# Download and import the MIT 6.S191 package\n","!pip install mitdeeplearning\n","import mitdeeplearning as mdl\n","\n","# Import all remaining packages\n","import numpy as np\n","import os\n","import time\n","import functools\n","from IPython import display as ipythondisplay\n","from tqdm import tqdm\n","!apt-get install abcmidi timidity > /dev/null 2>&1\n","\n","# Check that we are using a GPU, if not switch runtimes\n","#   using Runtime > Change Runtime Type > GPU\n","assert len(tf.config.list_physical_devices('GPU')) > 0"]},{"cell_type":"markdown","metadata":{"id":"_ajvp0No4qDm"},"source":["## Dataset\n","\n","We've gathered a dataset of thousands of Irish folk songs, represented in the ABC notation. Let's download the dataset and inspect it: \n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":19,"status":"ok","timestamp":1659045746584,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"P7dFnP5q3Jve","outputId":"cc225469-764a-4932-a18c-fb4eed4f1eb1"},"outputs":[{"name":"stdout","output_type":"stream","text":["Found 817 songs in text\n","\n","Example song: \n","X:1\n","T:Alexander's\n","Z: id:dc-hornpipe-1\n","M:C|\n","L:1/8\n","K:D Major\n","(3ABc|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\n","dAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\n","AG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\n","FAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\n"]}],"source":["# Download the dataset\n","songs = mdl.lab1.load_training_data()\n","\n","# Print one of the songs to inspect it in greater detail!\n","example_song = songs[0]\n","print(\"\\nExample song: \")\n","print(example_song)"]},{"cell_type":"markdown","metadata":{"id":"hKF3EHJlCAj2"},"source":["We can easily convert a song in ABC notation to an audio waveform and play it back. Be patient for this conversion to run, it can take some time."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":53},"executionInfo":{"elapsed":15,"status":"ok","timestamp":1659045746584,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"Q6z2aNKmIE1B","outputId":"dddf513a-1ae1-42be-be67-2ae5946c6bd0"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["\"X:1\\nT:Alexander's\\nZ: id:dc-hornpipe-1\\nM:C|\\nL:1/8\\nK:D Major\\n(3ABc|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\\ndAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\\nAG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\\nFAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\""]},"execution_count":3,"metadata":{},"output_type":"execute_result"}],"source":["# sample song\n","example_song"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":14,"status":"ok","timestamp":1659045746585,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"HPNNXE1xIGqj","outputId":"a7673c49-bf52-4e22-98ae-9fe5e4c1e107"},"outputs":[{"data":{"text/plain":["str"]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["# display type\n","type(example_song)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":53},"executionInfo":{"elapsed":12,"status":"ok","timestamp":1659045746585,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"wpB4q9AzL2nV","outputId":"9d94222c-bd10-456c-90ed-265007703520"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["\"X:1\\nT:Alexander's\\nZ: id:dc-hornpipe-1\\nM:C|\\nL:1/8\\nK:D Major\\n(3ABc|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\\ndAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\\nAG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\\nFAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\""]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["# display content\n","example_song"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":76,"output_embedded_package_id":"1Khn-o6ZSDl67VIGZxc3cGl1d0xtsLtWr"},"executionInfo":{"elapsed":22674,"status":"ok","timestamp":1659045769248,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"11toYzhEEKDz","outputId":"3e9492a1-3568-454a-d2aa-43bb301ef6e6"},"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}],"source":["# Convert the ABC notation to audio file and listen to it\n","mdl.lab1.play_song(example_song)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"KLpKO3BGLneU"},"outputs":[],"source":["# some random text (copied from above)\n","# please feel free to change a few random letters in the song\n","# then play it to hear the difference\n","test = str(\"X:1\\nT:Alexander's\\nZ: id:dc-hornpipe-1\\nM:C|\\nL:1/8\\nK:D Major\\n(3AAAa|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\\ndAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\\nAG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\\nFAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\")"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":53},"executionInfo":{"elapsed":82,"status":"ok","timestamp":1659045769249,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"_CQGELyDL4hF","outputId":"3cf629f6-fd19-4d32-c3ea-ac376ebaec6a"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["\"X:1\\nT:Alexander's\\nZ: id:dc-hornpipe-1\\nM:C|\\nL:1/8\\nK:D Major\\n(3AAAa|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\\ndAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\\nAG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\\nFAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\""]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["# display\n","test"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":76,"output_embedded_package_id":"1y7rrErep5QwPQGqB6PIyKO_xDbscBdZC"},"executionInfo":{"elapsed":81,"status":"ok","timestamp":1659045769249,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"_yjnZJ85LokD","outputId":"21adc9a5-ca3c-4ac8-b2f3-d4094a6f3e97"},"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}],"source":["# play this song\n","mdl.lab1.play_song(test)"]},{"cell_type":"markdown","metadata":{"id":"7vH24yyquwKQ"},"source":["One important thing to think about is that this notation of music does not simply contain information on the notes being played, but additionally there is meta information such as the song title, key, and tempo. How does the number of different characters that are present in the text file impact the complexity of the learning problem? This will become important soon, when we generate a numerical representation for the text data."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":36,"status":"ok","timestamp":1659045769249,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"IlCgQBRVymwR","outputId":"91aca103-fa46-470e-c958-c6b86b9e12ce"},"outputs":[{"name":"stdout","output_type":"stream","text":["There are 83 unique characters in the dataset\n"]}],"source":["# Join our list of song strings into a single string containing all songs\n","songs_joined = \"\\n\\n\".join(songs) \n","\n","# Find all unique characters in the joined string\n","vocab = sorted(set(songs_joined))\n","print(\"There are\", len(vocab), \"unique characters in the dataset\")"]},{"cell_type":"markdown","metadata":{"id":"rNnrKn_lL-IJ"},"source":["## Process the dataset for the learning task\n","\n","Let's take a step back and consider our prediction task. We're trying to train a RNN model to learn patterns in ABC music, and then use this model to generate (i.e., predict) a new piece of music based on this learned information. \n","\n","Breaking this down, what we're really asking the model is: given a character, or a sequence of characters, what is the most probable next character? We'll train the model to perform this task. \n","\n","To achieve this, we will input a sequence of characters to the model, and train the model to predict the output, that is, the following character at each time step. RNNs maintain an internal state that depends on previously seen elements, so information about all characters seen up until a given moment will be taken into account in generating the prediction."]},{"cell_type":"markdown","metadata":{"id":"LFjSVAlWzf-N"},"source":["### Vectorize the text\n","\n","Before we begin training our RNN model, we'll need to create a numerical representation of our text-based dataset. To do this, we'll generate two lookup tables: one that maps characters to numbers, and a second that maps numbers back to characters. Recall that we just identified the unique characters present in the text."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"IalZLbvOzf-F"},"outputs":[],"source":["### Define numerical representation of text ###\n","\n","# Create a mapping from character to unique index.\n","# For example, to get the index of the character \"d\", \n","#   we can evaluate `char2idx[\"d\"]`.  \n","char2idx = {u:i for i, u in enumerate(vocab)}\n","\n","# Create a mapping from indices to characters. This is\n","#   the inverse of char2idx and allows us to convert back\n","#   from unique index to the character in our vocabulary.\n","idx2char = np.array(vocab)"]},{"cell_type":"markdown","metadata":{"id":"tZfqhkYCymwX"},"source":["This gives us an integer representation for each character. Observe that the unique characters (i.e., our vocabulary) in the text are mapped as indices from 0 to `len(unique)`. Let's take a peek at this numerical representation of our dataset:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":34,"status":"ok","timestamp":1659045769250,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"r8rqnn6iZDMY","outputId":"92223342-defa-44f0-e9ee-c1f519943e5b"},"outputs":[{"data":{"text/plain":["83"]},"execution_count":12,"metadata":{},"output_type":"execute_result"}],"source":["# total length\n","len(char2idx)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":32,"status":"ok","timestamp":1659045769250,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"FYyNlCNXymwY","outputId":"3c1e3f43-4843-4615-8475-f0bf6c793b1e"},"outputs":[{"name":"stdout","output_type":"stream","text":["{\n","  '\\n':   0,\n","  ' ' :   1,\n","  '!' :   2,\n","  '\"' :   3,\n","  '#' :   4,\n","  \"'\" :   5,\n","  '(' :   6,\n","  ')' :   7,\n","  ',' :   8,\n","  '-' :   9,\n","  '.' :  10,\n","  '/' :  11,\n","  '0' :  12,\n","  '1' :  13,\n","  '2' :  14,\n","  '3' :  15,\n","  '4' :  16,\n","  '5' :  17,\n","  '6' :  18,\n","  '7' :  19,\n","  '8' :  20,\n","  '9' :  21,\n","  ':' :  22,\n","  '<' :  23,\n","  '=' :  24,\n","  '>' :  25,\n","  'A' :  26,\n","  'B' :  27,\n","  'C' :  28,\n","  'D' :  29,\n","  'E' :  30,\n","  'F' :  31,\n","  'G' :  32,\n","  'H' :  33,\n","  'I' :  34,\n","  'J' :  35,\n","  'K' :  36,\n","  'L' :  37,\n","  'M' :  38,\n","  'N' :  39,\n","  'O' :  40,\n","  'P' :  41,\n","  'Q' :  42,\n","  'R' :  43,\n","  'S' :  44,\n","  'T' :  45,\n","  'U' :  46,\n","  'V' :  47,\n","  'W' :  48,\n","  'X' :  49,\n","  'Y' :  50,\n","  'Z' :  51,\n","  '[' :  52,\n","  ']' :  53,\n","  '^' :  54,\n","  '_' :  55,\n","  'a' :  56,\n","  'b' :  57,\n","  'c' :  58,\n","  'd' :  59,\n","  'e' :  60,\n","  'f' :  61,\n","  'g' :  62,\n","  'h' :  63,\n","  'i' :  64,\n","  'j' :  65,\n","  'k' :  66,\n","  'l' :  67,\n","  'm' :  68,\n","  'n' :  69,\n","  'o' :  70,\n","  'p' :  71,\n","  'q' :  72,\n","  'r' :  73,\n","  's' :  74,\n","  't' :  75,\n","  'u' :  76,\n","  'v' :  77,\n","  'w' :  78,\n","  'x' :  79,\n","  'y' :  80,\n","  'z' :  81,\n","  '|' :  82,\n","  ...\n","}\n"]}],"source":["# designated characters\n","# these are the representation of each character\n","# or, in other words, our vocabulary\n","print('{')\n","for char,_ in zip(char2idx, range(100)):\n","    print('  {:4s}: {:3d},'.format(repr(char), char2idx[char]))\n","print('  ...\\n}')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"g-LnKyu4dczc"},"outputs":[],"source":["### Vectorize the songs string ###\n","\n","'''TODO: Write a function to convert the all songs string to a vectorized\n","    (i.e., numeric) representation. Use the appropriate mapping\n","    above to convert from vocab characters to the corresponding indices.\n","\n","  NOTE: the output of the `vectorize_string` function \n","  should be a np.array with `N` elements, where `N` is\n","  the number of characters in the input string\n","'''\n","def vectorize_string(string):\n","    vectorized_output = np.array([char2idx[char] for char in string])\n","    return vectorized_output\n","\n","# def vectorize_string(string):\n","# TODO\n","vectorized_songs = vectorize_string(songs_joined)"]},{"cell_type":"markdown","metadata":{"id":"IqxpSuZ1w-ub"},"source":["We can also look at how the first part of the text is mapped to an integer representation:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":32,"status":"ok","timestamp":1659045769251,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"l1VKcQHcymwb","outputId":"5c43f524-5a22-4e06-ed5f-985966077380"},"outputs":[{"name":"stdout","output_type":"stream","text":["'X:1\\nT:Alex' ---- characters mapped to int ----> [49 22 13  0 45 22 26 67 60 79]\n"]}],"source":["# display\n","print ('{} ---- characters mapped to int ----> {}'.format(repr(songs_joined[:10]), vectorized_songs[:10]))\n","# check that vectorized_songs is a numpy array\n","assert isinstance(vectorized_songs, np.ndarray), \"returned result should be a numpy array\""]},{"cell_type":"markdown","metadata":{"id":"hgsVvVxnymwf"},"source":["### Create training examples and targets\n","\n","Our next step is to actually divide the text into example sequences that we'll use during training. Each input sequence that we feed into our RNN will contain `seq_length` characters from the text. We'll also need to define a target sequence for each input sequence, which will be used in training the RNN to predict the next character. For each input, the corresponding target will contain the same length of text, except shifted one character to the right.\n","\n","To do this, we'll break the text into chunks of `seq_length+1`. Suppose `seq_length` is 4 and our text is \"Hello\". Then, our input sequence is \"Hell\" and the target sequence is \"ello\".\n","\n","The batch method will then let us convert this stream of character indices to sequences of the desired size."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":30,"status":"ok","timestamp":1659045769251,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"LF-N8F7BoDRi","outputId":"3e1c78cd-9897-419a-9b3e-9d844e48cf76"},"outputs":[{"name":"stdout","output_type":"stream","text":["[PASS] test_batch_func_types\n","[PASS] test_batch_func_shapes\n","[PASS] test_batch_func_next_step\n","======\n","[PASS] passed all tests!\n"]}],"source":["### Batch definition to create training examples ###\n","\n","def get_batch(vectorized_songs, seq_length, batch_size):\n","    # the length of the vectorized songs string\n","    n = vectorized_songs.shape[0] - 1\n","    # randomly choose the starting indices for the examples in the training batch\n","    idx = np.random.choice(n-seq_length, batch_size)\n","\n","    '''TODO: construct a list of input sequences for the training batch'''\n","    input_batch = [vectorized_songs[i : i+seq_length] for i in idx]\n","    # input_batch = # TODO\n","    '''TODO: construct a list of output sequences for the training batch'''\n","    output_batch = [vectorized_songs[i+1 : i+seq_length+1] for i in idx]\n","    # output_batch = # TODO\n","\n","    # x_batch, y_batch provide the true inputs and targets for network training\n","    x_batch = np.reshape(input_batch, [batch_size, seq_length])\n","    y_batch = np.reshape(output_batch, [batch_size, seq_length])\n","\n","    # output\n","    return x_batch, y_batch\n","\n","# Perform some simple tests to make sure your batch function is working properly! \n","test_args = (vectorized_songs, 10, 2)\n","if not mdl.lab1.test_batch_func_types(get_batch, test_args) or \\\n","    not mdl.lab1.test_batch_func_shapes(get_batch, test_args) or \\\n","    not mdl.lab1.test_batch_func_next_step(get_batch, test_args): \n","    print(\"======\\n[FAIL] could not pass tests\")\n","else: \n","    print(\"======\\n[PASS] passed all tests!\")"]},{"cell_type":"markdown","metadata":{"id":"_33OHL3b84i0"},"source":["For each of these vectors, each index is processed at a single time step. So, for the input at time step 0, the model receives the index for the first character in the sequence, and tries to predict the index of the next character. At the next timestep, it does the same thing, but the RNN considers the information from the previous step, i.e., its updated state, in addition to the current input.\n","\n","We can make this concrete by taking a look at how this works over the first several characters in our text:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":29,"status":"ok","timestamp":1659045769251,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"0eBu9WZG84i0","outputId":"fdfe3c74-376c-4332-f2bf-aecad4a97b67"},"outputs":[{"name":"stdout","output_type":"stream","text":["Step   0\n","  input: 26 ('A')\n","  expected output: 59 ('d')\n","Step   1\n","  input: 59 ('d')\n","  expected output: 59 ('d')\n","Step   2\n","  input: 59 ('d')\n","  expected output: 60 ('e')\n","Step   3\n","  input: 60 ('e')\n","  expected output: 1 (' ')\n","Step   4\n","  input: 1 (' ')\n","  expected output: 61 ('f')\n"]}],"source":["# get some sample data\n","x_batch, y_batch = get_batch(vectorized_songs, seq_length=5, batch_size=1)\n","\n","for i, (input_idx, target_idx) in enumerate(zip(np.squeeze(x_batch), np.squeeze(y_batch))):\n","    print(\"Step {:3d}\".format(i))\n","    print(\"  input: {} ({:s})\".format(input_idx, repr(idx2char[input_idx])))\n","    print(\"  expected output: {} ({:s})\".format(target_idx, repr(idx2char[target_idx])))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":29,"status":"ok","timestamp":1659045769252,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"2Txv6LtmHwKp","outputId":"2bc4c798-8713-44b1-cb27-f22b5f460fa9"},"outputs":[{"data":{"text/plain":["((1, 5), (1, 5))"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["# display shape\n","x_batch.shape, y_batch.shape"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":28,"status":"ok","timestamp":1659045769252,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"HVIkj6z5MjU_","outputId":"0f2dceba-97f9-4ba3-fea2-a781cb4daded"},"outputs":[{"data":{"text/plain":["(array([[26, 59, 59, 60,  1]]), array([[59, 59, 60,  1, 61]]))"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["# output\n","x_batch, y_batch"]},{"cell_type":"markdown","metadata":{"id":"r6oUuElIMgVx"},"source":["## The Recurrent Neural Network (RNN) model"]},{"cell_type":"markdown","metadata":{"id":"m8gPwEjRzf-Z"},"source":["Now we're ready to define and train a RNN model on our ABC music dataset, and then use that trained model to generate a new song. We'll train our RNN using batches of song snippets from our dataset, which we generated in the previous section.\n","\n","The model is based off the LSTM architecture, where we use a state vector to maintain information about the temporal relationships between consecutive characters. The final output of the LSTM is then fed into a fully connected [`Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) layer where we'll output a softmax over each character in the vocabulary, and then sample from this distribution to predict the next character. \n","\n","As we introduced in the first portion of this lab, we'll be using the Keras API, specifically, [`tf.keras.Sequential`](https://www.tensorflow.org/api_docs/python/tf/keras/models/Sequential), to define the model. Three layers are used to define the model:\n","\n","* [`tf.keras.layers.Embedding`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding): This is the input layer, consisting of a trainable lookup table that maps the numbers of each character to a vector with `embedding_dim` dimensions.\n","* [`tf.keras.layers.LSTM`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM): Our LSTM network, with size `units=rnn_units`. \n","* [`tf.keras.layers.Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense): The output layer, with `vocab_size` outputs.\n","\n","\n","<img src=\"https://raw.githubusercontent.com/aamini/introtodeeplearning/2019/lab1/img/lstm_unrolled-01-01.png\" alt=\"Drawing\"/>"]},{"cell_type":"markdown","metadata":{"id":"rlaOqndqBmJo"},"source":["### Define the RNN model\n","\n","Now, we will define a function that we will use to actually build the model."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"8DsWzojvkbc7"},"outputs":[],"source":["# define a LSTM building block\n","# as a function\n","def LSTM(rnn_units): \n","    return tf.keras.layers.LSTM(\n","        rnn_units, \n","        return_sequences=True, \n","        recurrent_initializer='glorot_uniform',\n","        recurrent_activation='sigmoid',\n","        stateful=True,\n","    )"]},{"cell_type":"markdown","metadata":{"id":"IbWU4dMJmMvq"},"source":["The time has come! Fill in the `TODOs` to define the RNN model within the `build_model` function, and then call the function you just defined to instantiate the model!"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MtCrdfzEI2N0"},"outputs":[],"source":["### Defining the RNN Model ###\n","\n","'''TODO: Add LSTM and Dense layers to define the RNN model using the Sequential API.'''\n","def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n","    model = tf.keras.Sequential([\n","    # Layer 1: Embedding layer to transform indices into dense vectors \n","    #   of a fixed embedding size\n","    tf.keras.layers.Embedding(vocab_size, embedding_dim, batch_input_shape=[batch_size, None]),\n","\n","    # Layer 2: LSTM with `rnn_units` number of units. \n","    # TODO: Call the LSTM function defined above to add this layer.\n","    # LSTM('''TODO'''),\n","    LSTM(rnn_units), \n","\n","    # Layer 3: Dense (fully-connected) layer that transforms the LSTM output\n","    #   into the vocabulary size. \n","    # TODO: Add the Dense layer.\n","    # '''TODO: DENSE LAYER HERE'''\n","    tf.keras.layers.Dense(vocab_size)\n","    ])\n","\n","    return model\n","\n","# Build a simple model with default hyperparameters. You will get the \n","#   chance to change these later.\n","model = build_model(len(vocab), embedding_dim=256, rnn_units=1024, batch_size=32)"]},{"cell_type":"markdown","metadata":{"id":"-ubPo0_9Prjb"},"source":["### Test out the RNN model\n","\n","It's always a good idea to run a few simple checks on our model to see that it behaves as expected.  \n","\n","First, we can use the `Model.summary` function to print out a summary of our model's internal workings. Here we can check the layers in the model, the shape of the output of each of the layers, the batch size, etc."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":27,"status":"ok","timestamp":1659045769253,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"RwG1DD6rDrRM","outputId":"ed3bf619-d193-4059-b845-1b5a66ec2ee4"},"outputs":[{"name":"stdout","output_type":"stream","text":["Model: \"sequential\"\n","_________________________________________________________________\n"," Layer (type)                Output Shape              Param #   \n","=================================================================\n"," embedding (Embedding)       (32, None, 256)           21248     \n","                                                                 \n"," lstm (LSTM)                 (32, None, 1024)          5246976   \n","                                                                 \n"," dense (Dense)               (32, None, 83)            85075     \n","                                                                 \n","=================================================================\n","Total params: 5,353,299\n","Trainable params: 5,353,299\n","Non-trainable params: 0\n","_________________________________________________________________\n"]}],"source":["# summary\n","model.summary()"]},{"cell_type":"markdown","metadata":{"id":"8xeDn5nZD0LX"},"source":["We can also quickly check the dimensionality of our output, using a sequence length of 100. Note that the model can be run on inputs of any length."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":23,"status":"ok","timestamp":1659045769253,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"C-_70kKAPrPU","outputId":"3597ccf8-20c5-4aa3-934c-c772df5ea15f"},"outputs":[{"name":"stdout","output_type":"stream","text":["Input shape:       (32, 100)  # (batch_size, sequence_length)\n","Prediction shape:  (32, 100, 83) # (batch_size, sequence_length, vocab_size)\n"]}],"source":["# display dimensions\n","x, y = get_batch(vectorized_songs, seq_length=100, batch_size=32)\n","pred = model(x)\n","print(\"Input shape:      \", x.shape, \" # (batch_size, sequence_length)\")\n","print(\"Prediction shape: \", pred.shape, \"# (batch_size, sequence_length, vocab_size)\")"]},{"cell_type":"markdown","metadata":{"id":"mT1HvFVUGpoE"},"source":["### Predictions from the untrained model\n","\n","Let's take a look at what our untrained model is predicting.\n","\n","To get actual predictions from the model, we sample from the output distribution, which is defined by a `softmax` over our character vocabulary. This will give us actual character indices. This means we are using a [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) to sample over the example prediction. This gives a prediction of the next character (specifically its index) at each timestep.\n","\n","Note here that we sample from this probability distribution, as opposed to simply taking the `argmax`, which can cause the model to get stuck in a loop.\n","\n","Let's try this sampling out for the first example in the batch."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":22,"status":"ok","timestamp":1659045769254,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"4V4MfFg0RQJg","outputId":"aef9bfa3-883d-47fc-8a81-98b8e0614bc1"},"outputs":[{"data":{"text/plain":["array([68,  7, 12, 27, 58, 37, 61, 16, 41, 52, 12, 18, 14, 56, 35, 32, 55,\n","        6, 57, 13, 30, 50, 43, 80, 44, 59, 56, 48, 41,  3, 62,  0, 76, 38,\n","       30, 27, 75, 68, 56, 24, 48, 47,  0,  0, 30, 57, 61, 71, 54, 41, 41,\n","       15, 47, 53, 25, 29, 12, 23,  0, 25, 60, 41, 51, 61, 43, 39, 29,  4,\n","       26, 28, 40, 34, 42, 47, 52, 20, 52, 61, 18, 63, 36, 76, 62, 45, 71,\n","       80, 58, 65, 51, 18, 56, 63, 70, 52, 20, 70, 61, 15, 37, 10])"]},"execution_count":24,"metadata":{},"output_type":"execute_result"}],"source":["# output of the sample indices\n","sampled_indices = tf.random.categorical(pred[0], num_samples=1)\n","sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()\n","sampled_indices"]},{"cell_type":"markdown","metadata":{"id":"LfLtsP3mUhCG"},"source":["We can now decode these to see the text predicted by the untrained model:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":21,"status":"ok","timestamp":1659045769254,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"xWcFwPwLSo05","outputId":"73e17210-6628-4a69-e33d-2ca2180ae67f"},"outputs":[{"name":"stdout","output_type":"stream","text":["Input: \n"," \"\\nded fed|ced cAG|AGF EDE|FDD D2:|!\\n\\nX:104\\nT:O'Gallagher's Frolics\\nZ: id:dc-jig-85\\nM:6/8\\nL:1/8\\nK:E Mi\"\n","\n","Next Char Predictions: \n"," 'm)0BcLf4P[062aJG_(b1EYRySdaWP\"g\\nuMEBtma=WV\\n\\nEbfp^PP3V]>D0<\\n>ePZfRND#ACOIQV[8[f6hKugTpycjZ6aho[8of3L.'\n"]}],"source":["# print\n","print(\"Input: \\n\", repr(\"\".join(idx2char[x[0]])))\n","print()\n","print(\"Next Char Predictions: \\n\", repr(\"\".join(idx2char[sampled_indices])))"]},{"cell_type":"markdown","metadata":{"id":"HEHHcRasIDm9"},"source":["As you can see, the text predicted by the untrained model is pretty nonsensical! How can we do better? We can train the network!"]},{"cell_type":"markdown","metadata":{"id":"LJL0Q0YPY6Ee"},"source":["## Training the model: loss and training operations\n","\n","Now it's time to train the model!\n","\n","At this point, we can think of our next character prediction problem as a standard classification problem. Given the previous state of the RNN, as well as the input at a given time step, we want to predict the class of the next character -- that is, to actually predict the next character. \n","\n","To train our model on this classification task, we can use a form of the `crossentropy` loss (negative log likelihood loss). Specifically, we will use the [`sparse_categorical_crossentropy`](https://www.tensorflow.org/api_docs/python/tf/keras/losses/sparse_categorical_crossentropy) loss, as it utilizes integer targets for categorical classification tasks. We will want to compute the loss using the true targets -- the `labels` -- and the predicted targets -- the `logits`.\n","\n","Let's first compute the loss using our example predictions from the untrained model: "]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":20,"status":"ok","timestamp":1659045769254,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"4HrXTACTdzY-","outputId":"f650aea8-b885-41e6-f09c-370a951dd913"},"outputs":[{"name":"stdout","output_type":"stream","text":["Prediction shape:  (32, 100, 83)  # (batch_size, sequence_length, vocab_size)\n","scalar_loss:       4.4188385\n"]}],"source":["### Defining the loss function ###\n","\n","'''TODO: define the loss function to compute and return the loss between\n","    the true labels and predictions (logits). Set the argument from_logits=True.'''\n","def compute_loss(labels, logits):\n","    loss = tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)\n","    # loss = tf.keras.losses.sparse_categorical_crossentropy('''TODO''', '''TODO''', from_logits=True) # TODO\n","    return loss\n","\n","'''TODO: compute the loss using the true next characters from the example batch \n","    and the predictions from the untrained model several cells above'''\n","example_batch_loss = compute_loss(y, pred)\n","# example_batch_loss = compute_loss('''TODO''', '''TODO''') # TODO\n","\n","print(\"Prediction shape: \", pred.shape, \" # (batch_size, sequence_length, vocab_size)\") \n","print(\"scalar_loss:      \", example_batch_loss.numpy().mean())"]},{"cell_type":"markdown","metadata":{"id":"0Seh7e6eRqd7"},"source":["Let's start by defining some hyperparameters for training the model. To start, we have provided some reasonable values for some of the parameters. It is up to you to use what we've learned in class to help optimize the parameter selection here!"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JQWUUhKotkAY"},"outputs":[],"source":["### Hyperparameter setting and optimization ###\n","\n","# Optimization parameters:\n","num_training_iterations = 2000  # Increase this to train longer\n","batch_size = 4  # Experiment between 1 and 64\n","seq_length = 100  # Experiment between 50 and 500\n","learning_rate = 5e-3  # Experiment between 1e-5 and 1e-1\n","\n","# Model parameters: \n","vocab_size = len(vocab)\n","embedding_dim = 256 \n","rnn_units = 1024  # Experiment between 1 and 2048\n","\n","# Checkpoint location: \n","checkpoint_dir = './training_checkpoints'\n","checkpoint_prefix = os.path.join(checkpoint_dir, \"my_ckpt\")"]},{"cell_type":"markdown","metadata":{"id":"5cu11p1MKYZd"},"source":["Now, we are ready to define our training operation -- the optimizer and duration of training -- and use this function to train the model. You will experiment with the choice of optimizer and the duration for which you train your models, and see how these changes affect the network's output. Some optimizers you may like to try are [`Adam`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam?version=stable) and [`Adagrad`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adagrad?version=stable).\n","\n","First, we will instantiate a new model and an optimizer. Then, we will use the [`tf.GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape) method to perform the backpropagation operations. \n","\n","We will also generate a print-out of the model's progress through training, which will help us easily visualize whether or not we are minimizing the loss."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":558},"executionInfo":{"elapsed":6970,"status":"ok","timestamp":1659045776205,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"F31vzJ_u66cb","outputId":"20db5c4b-9d92-4bce-88b9-5af1389c9e58"},"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUZf4H8M83mwahJEDoJfQmPQpIFaQINrChnu1U9H541tMDKzYO652eZ0FUbOh5Kh4eShURQcDQq9IiVQg9kLqb5/fHzOzOtmQ32dkNk8/79eKV3cnszDeb8N1nnueZ7yNKKRARkf3ExToAIiKyBhM8EZFNMcETEdkUEzwRkU0xwRMR2VR8rAMwq1evnsrIyIh1GEREZ43Vq1cfUUqlB/pepUrwGRkZyMrKinUYRERnDRH5Ldj32EVDRGRTTPBERDbFBE9EZFNM8ERENsUET0RkU0zwREQ2ZWmCF5FUEflcRLaJyFYR6Wvl+YiIyMPqFvwrAOYqpToA6AZgqxUnyS9y4cs1+8DSx0REHpbd6CQitQEMBHAzACiligAUWXGup+dswcyVe9CwdjLOb13PilMQEZ11rGzBtwSQA+A9EVkrItNFJMV3JxEZLyJZIpKVk5NTrhMdPlUAADhT6KpIvEREtmJlgo8H0BPAG0qpHgDOAJjou5NSappSKlMplZmeHrCcQsjYRUNE5GFlgt8HYJ9SaqX+/HNoCT/iRAQAwPRORORhWYJXSv0OYK+ItNc3DQWwxYpzieecVhyeiOisZHU1yT8D+FhEEgHsAnCLFSfRG/Bgfici8rA0wSul1gHItPIcACBgFw0RkS9b3MnKFjwRkT9bJXgiIvKwR4J3d9GwCU9EZLBFgjem0ZQwvxMRudkiwXOaJBGRP3skeHbCExH5sUeC17+yAU9E5GGPBG9Mk+QgKxGRmz0SfKwDICKqhOyR4I1iY2zAExG52SPB6185TZKIyMMWCR7uUgXM8EREBlskeGEvPBGRH1skeCIi8scET0RkU7ZI8Jz/TkTkzxYJ3sCSBUREHrZK8JxFQ0TkYYsEz1k0RET+bJHgiYjIHxM8EZFNMcETEdmULRI8p0kSEfmzRYInIiJ/TPBERDZliwTPaZJERP5skeCJiMifLRK8McjKoVYiIg9bJHh3ZmeGJyJyi7fy4CKSDSAXgAuAUymVacV5SpTRgmeGJyIyWJrgdRcopY5E4TxcdJuIyMQWXTRGXuei20REHlYneAVgvoisFpHxgXYQkfEikiUiWTk5OeU6iZHY2UVDRORhdYLvr5TqCeAiABNEZKDvDkqpaUqpTKVUZnp6erlOYtSBZxcNEZGHpQleKbVf/3oYwCwA51lyHp+vRERkYYIXkRQRqWk8BjAcwCYrzuVeyYlNeCIiNytn0TQAMEtfJzUewEyl1FwrTuTO71YcnIjoLGVZgldK7QLQzarje5/L+ysREdlmmqSW2UuY4YmI3OyR4NmCJyLyY4sEX8I+eCIiP7ZI8EZqV2zCExG52SLBs0QBEZE/WyR43slKROTPHgne/ZUZnojIYI8Er+d1dtUQEXnYIsHH6WtuO10lsQ2EiKgSsUWCd+gZvsjFJjwRkcEWCd7ooilysgVPRGSwRYJ36Rm+mF00RERutkjwJWzBExH5sUeC1zM8EzwRkYc9Ejy7aIiI/NgiwS/feRQAUMgET0TkZosEbyhmFw0RkZutEnwRW/BERG62SvDsgyci8rBVgucsGiIiD3sleJYqICJys1WCZ7ExIiIPWyT4N67viZrJ8eyDJyIysUWCv6hLIwxuXx9OdtEQEbnZIsEDQEKccJokEZGJfRK8I45dNEREJrZJ8PEOYRcNEZGJbRJ8giOOXTRERCY2SvBswRMRmVme4EXEISJrReR/Vp6HffBERN6i0YK/B8BWq08S74iDs0RBKbbiiYgAixO8iDQFMBrAdCvPAwCJDgEAFLObhogIgPUt+H8AeAhA0L4TERkvIlkikpWTk1PuE8U7tB+F3TRERBrLEryIXAzgsFJqdWn7KaWmKaUylVKZ6enp5T5fop7g84td5T4GEZGdWNmC7wfgUhHJBvApgCEi8pFVJ2tepzoAYM+xPKtOQUR0VrEswSulJimlmiqlMgCMA/CdUuoPVp0vKUH7UUpK2AdPRATYaB58nGiDrMzvRESa+GicRCn1PYDvrTyHnt9RwmmSREQAbNmCZ4InIgJsmOCZ34mINDZK8NpXtuCJiDS2SfCit+BdHGUlIgJgowRvtODZgCci0tgmwTv0DL9w66EYR0JEVDnYJsEbg6wfr9zDipJERLBRgjfmwQOsKElEBISY4EUkRUTi9MftRORSEUmwNrTwxJkyfKGTBceIiEJtwf8AIFlEmgCYD+AGADOsCqo8vBM8SwYTEYWa4EUplQdgLIDXlVJXAehsXVjhizN10fy082jsAiEiqiRCTvAi0hfA9QDm6Nsc1oRUPmJqwR/OLYxhJERElUOoCf5eAJMAzFJKbRaRVgAWWxdW+Mwt+Ho1EmMXCBFRJRFSNUml1BIASwBAH2w9opS628rAwsU+eCIib6HOopkpIrVEJAXAJgBbRORBa0MLjyOOCZ6IyCzULppOSqlTAC4H8C2AltBm0lQajWonux/vOXomhpEQEVUOoSb4BH3e++UAZiuligFUqruJ4h1x6Nk8FQDw9tLdMY6GiCj2Qk3wbwHIBpAC4AcRaQHglFVBldcn4/vEOgQiokoj1EHWVwG8atr0m4hcYE1I5ZcQZ5vKC0REFRbqIGttEXlZRLL0fy9Ba81XKnGmgdZDpwpiGAkRUeyF2uR9F0AugKv1f6cAvGdVUJHw3LfbYh0CEVFMhdRFA6C1UuoK0/MnRWSdFQFFSvWkSnWjLRFR1IXags8Xkf7GExHpByDfmpAiI9HBBE9EVVuoCf5OAP8SkWwRyQbwGoA7LIsqAt5dxqmSRFS1hTqLZj2AbiJSS39+SkTuBbDByuCIiKj8wppXqJQ6pd/RCgD3WxBPRE2evTnWIRARxUxFJo5L2btEX59WddyPZyzPjl0gREQxVpEEX6lKFRheHdcj1iEQEVUKpfbBi0guAidyAVDNkogqKCUp1JmfRET2Vmo2VErVLO+BRSQZ2lquSfp5PldKPVHe44XKXDaYiKgqs7K5WwhgiFLqtF6J8kcR+VYptcLCc3ot/EFEVJVZluCVUgrAaf1pgv7P8n57tuCJiDSWll8UEYde0uAwgAVKqZUB9hlvFDHLycmp8DmZ34mINJYmeKWUSynVHUBTAOeJyDkB9pmmlMpUSmWmp6dX+Jzi00VzutBZ4WMSEZ2NolJAXSl1AsBiACOjcb7NT45wPy4odkXjlERElY5lCV5E0kUkVX9cDcAwAFGp4WueKplfxARPRFWTlbNoGgF4X0Qc0D5IPlNK/c/C8wXEFjwRVVWWteCVUhuUUj2UUl2VUucopZ6y6lyBPDtG6+7fmXO6jD2JiOzJtouYZtTVVhS886M1MY6EiCg2bJvgkxM8C378dvRMDCMhIooN2yb4aqYE/1nW3hhGQkQUG7ZN8LWrJ7gfl1TKupdERNaybYJvXDvZ/Xh19vEYRkJEFBu2TfAigi//73wAwKrsYzGOhogo+myb4AGgVnJC2TsREdmUrRM8C48RUVVm6wTPwVUiqspsnuCZ4Ymo6rJ1gm9omknjdJXEMBIiouizdYI3D7IWOJngiahqsXWCNytkVUkiqmKqTIL/53c7Yh0CEVFUVZkEP2N5NlycVkNEVUiVSfAA0Prhb6A4s4aIqogqleABwMlWPBFVEVUuwbObhoiqCtsn+D6t6ng9//6XHGRMnIOc3MIYRUREFB22T/ATLmjj9XzaDzsBAJsOnIxFOEREUWP7BN+/TT2v58UurYsmTliJjIjszfYJXnwSeZF+R2s8S00Skc3ZPsH7KtZr0rAFT0R2V+USfJGe4B1swRORzVW5BL/veD4ALgZCRPZXJRL8xsnD8deRHby2/XfdASzbcQTHzhQFfd19/16H4X9fYnV4RESWiI91ANFQMzkBjUy14QHgwxW/4cMVv6F6ogNbnhoZ8HWz1u6PRnhERJaoEi14ADiRF7ilnlfEMsJEZE9VJsEPaJce6xCIiKLKsgQvIs1EZLGIbBGRzSJyj1XnCkXr9BpBv/fU11uw52heFKMhIrKelS14J4AHlFKdAPQBMEFEOll4vnJ7d9lu/PO77bEOg4gooixL8Eqpg0qpNfrjXABbATSx6nwVlVvgBACcKXTGOBIiosiISh+8iGQA6AFgZTTOVx4Oh+AP01ei8xPz8FUlmT2TX+TC2j3HYx0GEZ2lLE/wIlIDwBcA7lVKnQrw/fEikiUiWTk5OZbG0rJeStDvzdlwED/uOAIAWLTtsKVxhOr+z9ZhzOvLcfQ0SxsTUfgsTfAikgAtuX+slPoy0D5KqWlKqUylVGZ6urUzXRb/ZTCyp47G5d0bl7pfZbnJdcM+raQxp3ISUXlYOYtGALwDYKtS6mWrzlMeDWoll71TGf67bj8WbDnkte3+f69DxsQ5yK+ECTmvyImdOadjHQYRRZGVd7L2A3ADgI0isk7f9rBS6hsLzxmSOwe1BgAcOlWAr9YdCPl1D8/aiAMn8vHKNT1wz6faj5Q9dTQA4PiZInyp990fyyvCmeNO5Be50K1ZaoSjL587PlyNpduPYNeUUYhjIR6iKsGyBK+U+hGVp7fDS1pKIiaN6ojXgkyNnL3+AJqmVfPbPnPlHgDA3uP+c+ZP+8y+Gf73HwAA658YjtrVEioacoUt08cXilwlSI5zxDgaIoqGKnMnayDG4h+BvP79zrCOVaICL+bd7cn5+Cxrb1jHMkSyZH28Q/tVG+WSicj+qnaCdwVOyr5W7T6GG99d5X5e6PT0sR8/U4SMiXOwaKtn5o3ySfZLfvWfHTRnw0H0enoBMibOwaQvN4YUh9NV4nXucCTo3TKlfagRkb1U6QRfHGJr9uq3fsIPpiR9xRs/uR/3eHoBAOCp/21xb1u1+5jX6wM1xCd/vRlH9VLFn6zaU+r5jauD66avRPtH54YUsy93C54JnqjKqNIJ3qkn+Ct6No3oce//bH2Z+/iOcy7+xX/uvdFF4yrRErzxwTFr7T68OO+XsGJKcLAFT1TVVOkE36WpNsPlqszIJvhQiE+7/pb3fsariwIP+pb49CTd9+/1eG3xjrDOZ6xB6yxhgieqKqp0gr+iZxMseXAw+rSqa+l5xDRaeqbQieU7jwTc7+UFvwbcHmwANxxGgi8OMu5Q5CzBzJV73Fc1RHT2qxIrOgUjImhRN3j5gkj5ev0B5BYUI616YlirRBmtfJdvE74c4ny6e3y9vOBXvLlkJ+qkJGLkOQ0rfD4iir0q3YKPpu9/yQkpuV/8z6V+2yLRghd3F41CodOF3IJir+8bc/tDHXiOhbwiJwqKK99dwkSVFRO8jy5NamPaDb1idv5N+/3qsQVNaq9/vwO/HT2DdXtPANCmbL61ZKffNE3AM2DrdJXgurdXosvk+d47BPkMWbztsFeJg/wiV8ySbKfH56H3lEUxOTfR2ahKd9EEckPfFhjWqQG2PDUCX68/gL9+Edoc9UjKyS1Ees0k9/Mr3vgJvVvW8dvv+bm/4Pm52mya7Kmj3VM2e7VIQ2aGtv/DszZiw74T2Hc8HwCw/0Q+Vv/mXYJ41e5jmLPxIAD/m6tumfGz+/gA0PHxuaiVHI8Nk0dU9MfEo19tRNv6NXHT+Rml7vdZ1l4kxWttkZP5xaXuG0xBsQs5uYVoVqd6qfs5XSWY/PVm/GlwGzRJ9b+bmehswha8j6szm0FEUD0xHtec2zwmMeTkauWB9xzzlERY6TO3vjTFLoXcgmJc9MpSzFy5x+uqwKihAwCHcwuwfMcRXP2WZ17/XTPX4l9lzNA5VRB8UZRpP+xE5jMLQ4rzoxV78MTszV7b9hzNw6FTBV7bHvp8g1fc5TH+w9UY8PxiKKXw+ep9aPPwNwGnjK7afQwfrdiDB/9T9lRXosqOCV734lXdMKBtvViHAQAY9epSbD5wstyvP3SqAF0mz8fWg/7dPWZDX1yC66b7r8Hy9tJdKCj27orpMnke/vbNVq/9ftp51O/DYMo323CkAvXrB76wGOdP/Q4Fxa6IzNl3ukow6pWl7hvVilwlmPLNVjhLlN84hFkEhj2qhONnivDuj7sDdgtS7LGLRndlr6a4slf058MHM/rVH8Pa/7ejZ9yP7/13aK3d3CDLE57IK0aHx+bCYbobK7fAibd+2OV+/uWafe4bum46PwNKKczbfMjvWOXhKlHo8NhcpNdMwrK/Dgm63/zNv2NA23QcOlWAlxb8ipeu6obEeO82y8n8YmwxfdAVFJW470CIVEpas+c4ejRL9ZoOW1U8+PkGLNx6CD2ap6JH87RYh0M+mODL8Oq1PfDj9hzc2DcDF/8zvKQbTYNe+D7ixyxteqb5bt1znpjn9/2X5/+Cbb/n4pwmtXH30Lalnmfy7M349Oc9+PDW3l7bc3ILcf30FQFfs/XgKYz/cDU6NqrlvlIZd24z7D5yBiM6N/QawzArcLrc4wylNTpdSuHYmSLUSUksNfYFWw7h9g+yMGVMF1zXOzZdekXOEuQXuVC7evSrlh7P08ptOMOYyvvZz3vRt3Vdr/GQrOxjmLvpdzx6caeIx1iVsYumDJd2a4znr+yGc5rUjnUoZ5VXv9uB+VsO4eUFv2LjPu/upoMn87HYtCzijOXZKCguwVVv/uR7GPycHXhN2rwi7erD3A21M+c0Hv1qEybMXBM0rh+3H8GR01pS8u3rN1u1+xh6Pr3AXQZ61tp9OO/ZhX4ferv0GUbZpiuoaJswcw26PTU/4PeUUli09VBE7qUIdnwg9Lrgxa4SPPTFBr/f9ZVv/oTpP+6OcHTEBB+GxrVLXwnqlXHd8fCoDlGK5uxhDBZv2n8SJ/OKMfb15e7ZOeXx9foDXgXfDGcKtTGDE3qrEgCWbve+a/gB0+BpKFdk2w/l4uDJfDw6axMO5xa6P1gMxn0D8WEuonIyrxh/+3Zr0DuHv1i9D1cH+MALxHdlMbN5mw/h1vezMM3UvRZJxsdGqL1TxlVTTpBxGqVfOVll+tJdpb5fdsMEH4Z3bznX63n3Zqm4Y1Ar9/PLujfB+IGtsfaxYbh7SJtoh1epKaVw8T9/xA3vrsTBk8FbzqH48ydrA25/bu42AJ6yDADw3rLQW4VOV4lfvfwxry9H3799514F67tth93TTH/aeRTH87SB2gS9Wuf0pbvQZfK8Mgcdp3yzFW8t2YVXF23HybxiTF+6C5+aqoo+8J/1WJUdeObU56v3YWGAJBXow8K4SjlwIr/UeMrLuDAIdfzBuGkv2PuzUr9ymrvp94jE5+uZOVtx+wdZlhw7mKvf+gk99SnM0cYEH4aURO8hi68m9MOkizr67ZeWkoj7h7dHooNvLwBM/3GXO6lv2Ff+2UGhcsQJlFJ4Yd42bDpQ+kyiM4VOlJQofLpqD9o88i1ufi/wlUWuPjX0nk/X4Yo3luNkfjGufXsF3tG7FYxqnc/M2YrcAieOnSnCtB92ImPinIDJLF+fofTqdzsw9o1leGbOVkzU1wX4cXvgWkWGv/xnPW77IAtzNhxElulDwFxnSCnl/gfANO6gyj3j5VRBMU7m+cw80o+VV+g96+rAiXy88f1OFDpdXmsYuBN8kHMYq6at2HW0XDFGyu8nC5AxcQ6WbvdfyyEQY12IQFOMV+0+ZulVSWmYgcIQ7/C0Uox1XUsTiRIDdrB2zwmcP/W7qJ0vTgT5xS78a/HOMvueOz8xD09+vdmdXEPV7UnvPm+ltJlFhvlbDmHKN9oVRc7pQr8pmeYpoDtzPP33R04X4g/v+E9dVUrhdKHT/YECaH3vV5q6cczH7Pj4XFz0ylJ3Io0TwfEzRWg56RtcM80zcL314Cnc/claFDlLcOBEvjtJHz5V4Ncd1XXyfL++fuP4f3hnJUa94imzMf7DLDw3dxvaPzoX7R+di6e+3oL/rtvvvuEu2H+N2eu1NZLLqnr6+vc7yvwgLMvR04VBb5wz7g7/8KffUFKiUOQswc6c0zgapGvJGIN5QS/jXVDswts/7Ip58T7OogmDedrgbQNalrl/YnwcnEWsnRJtcXGC95Zlh7z/+z/9VuFzvuRTCfSprz0LwJz3rFZe4dVre6B3yzrYfug05m4O3AVx6/ve3QcFxS5M/XYbZizPRu+WdUq94c3oXipylqCguATbfs/Fhys8P5txp/Oq3ccw9dttqJHkwIvztbjHndsM101fiaT4OKx+bBjOm7IInRvXwpy7BwQ8V8bEOQC00h6GXUfOYNoPOzF+YGv3eIjhXb2rrHqiZz1gpRQWbj2MXw/l+h2/tA/mQqfLfQf3fRe2w9ieTfzuUP5240EMap+O6onBU1yvZxYi0RGHX5+9yO97xpiKs0Thzo9WY77eJZZWPQFrHx/ut79vtK99twOvLd6B1BjMbDJjgg9DWnXPlLlQBtVm3t4Hl/9rmZUhUQDr955ASmJsFxZ3BWii3h1k7MBs3zHvBd07POZZwausu5mLXCV458fdeNq0utgu/epgxvJsr33fXOK95rDRki10lmDkP7QF4zfr3Vu9nl6Afm08NwHuPuK54ti437vLbco32zB+YOugg655pgbPZ1l7g5YCcZaynOadH652P/77wl/xWdZeLJs4BMUurZW9eFsOnpu7DZd0a4z7LmyL04VOdNXXfjDsOKx9qJjHXJyuEvfU3gT9fopiVwm+2+a5UjieV4yfs48hs0Uaftp1FAmOOLRvWBOvLPRey8GYPlroc7NeQbELX68/gCt7NY3KfRNM8GFIcMShRlI8Thc63YNupencuJb+OglYh71+zSSseuRCvLJwO/6+MHAteLNGtZNx+4BW7uUB29SvgR2HT5fxqqqpInfTRkJ578I9WoG+2n4V6AbLN/WfG90ogLae8NEzRe6uEwC45q3SZ/cs33nE/cFSmuyjeUG/Z7Tgb35vFb7/ResHv/fCtrj3wnZY/It3v/j+E/lYvuOI313Z2w/lYshLS7Rz6bWUDL4flsWuEjwxezNmrtyDhfcPcq9h7DsLC4DfFM/Luzf2Wnf5te+2u99P81V/odOFh7/ciC/X7sfK3cfw4lXdgv78kcIEHwF/HdkBHRrW9Nue4IjDE5d0wuD29eF0leDDFb/hg59+Q82keGx80lOsK9QP8sV/GYzkBIc7weez+yeoXw/xgy8cwRoKN5kWmzcczi39w/O6t/3HEAIpbYjqy7X7cXO/DHdyB4B/LNyOuy4IPDstUMmNbb97un5+P1ngVXPpkVmb3I935pzGX/6zHmv3nHDva8zICsVX6w54PTe6vQBgkmls57b3s9wfGJ+v3of/G9wardJrhHye8uAga5ju0e/KrJbg6QL40+DWuKBD/YD739KvJVrWS0HbBjXx1GXnYMezF2Ht48MC7ju4fXrQ88YJkJzg3e1g3IQz8aIOfrfo+3r+yq5Y8uDgUvehquv173eWvVOEzV5X+voIl77m373Z5pFvy3WuybM3exXvMxv60hJ3cgeAT3/e49f1FAm+VwN5RS4opbB422HLbkRjgg/T7QNbIXvqaPe853DFO+IQ7/NaY+ApqZQk3ai2p3Rty3raKlRGgm+SWg31g9yav+iBQbh9QEuM6dEELeqmYGA77w+RtvVrYHSXRqXGXFmKsJXlwo4NYh0CheFABe+HCEewQe1A/rfhoIWReBS7SvDCvF9wy4yfMX2pNTeiMcFXAjf0bYF7hrbFrf1b+X2vfQP/rp+b+rYAAHdiblkvxX25e/+wdl77NkmthkdGd3J/IDl8uoNKlPKa/hnI2zdmhvRzWK1aggMNawW+m/ipyzpj+k2lxxmNPk+iUI15fbn7yqmiN/8FwwRfCSTFO3DfsHYBW/AdG2kJ/vo+nkJWN52fgaUPXYB/XNMdC+8fiHOa1EbvVtoCHzf2bYHU6gkY0LYeFj0wyK9b529ju3o9N/eDvnRVN3zxp77uu3Pf/+N5WPnwUCQnOPDCld6vK48xPZqUuU+t5ODDQn8b2wXLJvpXlxzbowlu6NMi6OtGdWmINY8N41qzVGnNWJ7tdUNYpHCQtRIxRtxFPIl3/MDW+Ps13b32ExH3vN829bUPgCljuuBPg1ojtXoi1gWYp2toWDsZm54cgSJnCYa9vAQTL+rgviR1xAl6taiDrk1TMaR9ffRuVdf9uit7NUXN5Hjc+ZFWyOu7Bwah2KUwa+1+vyl3wZQ1mHxB+3Q8dnEnXP6vZQEXFSlRCo44Qc2keK9Sx/de2K7UKWcCQZ2URK9B6U/H98G4aYErVRLFQkJc5NvbbMFXIp0b18Kfh7TB8olDkD11NLKnjkanxrUgImXOmU1OcKBtgO6cQGokxaNOSiJWPzYMwzs39CsYleCI80ru2vcEI8/x9NU3Tq2G9g1rYuJF3sXVZt7uXfJ355RR7ta/BKg5eHVmU0y4oDXWPz4c791yHlql18D8+wYBgF+ph076tNMF9w/y2t68buBl+IyZTUZBMPP/nx7NUwO9JCrOzUjDs2POidn5qXIKZep12MeM+BF1IvKuiBwWkU1l702AlkQfGN7ea0A1GoxB2/QagQdqzcb2bIKayfFeyffiro0w+ZJO+Pi23ujT0vPBMPuufnpdGO25iNad8sKVXXG7fifwYxd3woMjOnjVMm9YOxnZU0djy1MjcGNfT9dLh4Zagi+toXO9qSb7W/ri6Zd117qGHKYPybLqBBnvSTjuGdoWU8d2cT+vHuRmq//ceT7G+SwHWVnqFgWa7hvIc1d0KXsnijkru2hmAHgNwAcWnoMi4O4hbZDZIg3ntyl7tszLV3f32/badT29nj86uiN6NE913z2o9GuEOAFeNHU3PTK69MUd4h1xeOqyczCyc0OvW9HjSrmaeXZMF3ysF6xqUTfF6wYX800nZV0RDWqXjt1HzmBk54Y4croQWaaFyj+7o6/XnGpAKyV9nz7A3bxOdSQlxKFn8zSMm7Yi4B2oDp/WWmJ8nF8ly9KM6Nwg4ApaceKp8GiYMqYLHp4VWq2daTdkYuALi8vcr019a+dvl8djF+zrekoAAA5pSURBVHfyuov3bGJB4107rjWHBZRSPwAIfaVoipl4R5zf9MmKuG1AK/RqUcf93N2CD3lZCG/nt6nnleDL+3/BSOotgnTpTLqoA2bf1Q/ZU0fjln4ZAIBLuzf2W6mpS5Pa6Ny4FoZ30qZlXp3ZFDP+eJ5XvL1a1IGIeBWle3R0R3w1oZ/7+YMj2gPQZgd9Or6Pe3vDWslYMWkoZt7WG7f2b4kVk4bi5vMzvGKok5Lo7sb7y3Dtg6VN/Rp+4zUbJw/Hdb2bu+/fKEtKkv9Vx7QbemFYJ+8pqM3StPcw1L+bW/u3dF+1BXLz+RlYeP/AgN/r2rTsxXZGd2kUcMaZr/d8Sn5b4b+m33GoqiVYU1qDg6xkOaPvvF+E5tOnJHn+bAM1xGff1c9ds93XR7f2RruGWuuzX5u66NEsDa/pJV7vMCXjFnVTsGvKKHe/6NieTd0FtqolOjDn7gFQSqFE+bfGzQa1S8fDozrgip5NUdenC2zCBW0wIcCdmSseHgpA66oyrqomX9rZq55MtQTPe/CHPi3w4vxfMbpLI1zWvQlGdG7ormETr/dn3T20LV5Z5F0vxVA90YG8Ihdm39UPdWsk4bb+Lb1WVxreuSE6NqrlXijjjoGtUF//EKpXIxFHzxRh5e5j6N2yDnpPWeR17GvPa45PVu1BRr0U3NCnBW4b0MpvHwComRzvnjAAaHeHp1VPwMQvN2LypZ0x9vXlAWM3PH5JJ+wNciOTWZcKrMx2U98WIRWma9ugBt6+MTOsuvMt6obfJRiKmCd4ERkPYDwANG8emzUtyVpdm6Zi3ePDkFq99PVNQ5Wc4MCmJ0egeoIj4MBU16apfsWlDP1NHzIf36a1mi/okO5Vf8VQ1qCXiPjdVxDoGOMHll1aGtCqTbpKKZM7qktDtG9QC68s+tXrqiK1eiI2TB6OGnrlRPPUWOPDp7QPoWV/HYL8Yhcap2pjP3Vq+P+ejCm8dVMSMWmUtgZCQ32Fswa1knFpt8YAgE6Nanktcj7u3Gb4ZNUeDNTf9wYB7mOYOrYLxvTUxknm3N0fK3Ydw639W0IphX4+V28A8PwVXfHQFxvcz41uuFDmkldkpsrgDvUxtGMDpFZPwIm8Ytz47iqse3wYalfTxo9aTvoGgDaeEmy67xU9m+ILU1lpw4w/WnNlEfMEr5SaBmAaAGRmZrKAuk1FKrkbaiRF7k+3V4s66BV8Gn3UGEkymNev1waN77nQv7ulVnLgsrTmxL7ogUHYfzwfPZqnYsfh0xijt4rTUhKRZn6NflmUXjMJr13bAwDcd18HqpLpHWNPDH7xe/fzbs1S/Qp9+Rp3nufDqnPj2ujcWGtlm6cDm7WuH7i1W9aSmoD3mg5v35iJPcfyQu63T453oG9rzyQC35+rWZ1q2HssH/GOuKCD9M9f2TVggq9fs+zYyyPmCZ7obJFWPcG9RF9l9/FtvTF/8+9eg3et02ugtV7cqkfzNMy8rTeSAvT9XnNuMyzbeRQvXtkV9fUWt9GCL2uWTUYIs48u694Yh08VYnjnBhgSpIaTr6Zp1dxXWUnxnpg/MI191K+VjKlju3gt3tIktRr2m5YrNH/gGeMKvgn+0dEd8cycrX4xVCujBPXsCf3d5UPq10rGm3/ohTs/0kobP3FJJyQnOEq9krKCZQleRD4BMBhAPRHZB+AJpdQ7Vp2PyGrz7h2IvQG6ciqjfm3qedVwDyTYrKnU6oleiRPQxj0+Hd8HHRvVCjmG3i3rBNz+yrgeIR/D8ONfh7jHQIzCek1Sq/kN8iYlxPm87gJs2n8Kl7ymLbAeqIaUudvEaJU/N3ebX4nvYNNeDWkpiUhL8VypjjynITY/OQJnipxeLfR1jw9D96e0xVfSayaF9Z6Gy7IEr5S61qpjE8VC/VrJ7hZtVdTH5+a3YO67sB1Skhx+M38ipbRCf0YP0lW9mmLqFV0hIuhimoXjiBN8NaGfV1mQl67uhi/W7PNafWnzkyOxcvdR3PDOKvzjmu7YfyIfbcsxNTQlKd5rUgCgfYAad2OvnDTUkhucDOyiIaKICjRGEAkXdmyAhVsPoVlaNfRvUw9/HuI/A2l010ZYt/cE7ruwXdDukO7N/Afg5947AHVTPLOcEuPjMKBtOrY9PdKvnlMkzJrQD0u351ia3AFAyrvCuhUyMzNVVlboU4uIqOooKHbh6JkiNEkN/07vzQdOYtXuY7ilX9lrKZ9tRGS1UipgKVW24InorJCc4ChXcge8Z+dUJZWjAAYREUUcEzwRkU0xwRMR2RQTPBGRTTHBExHZFBM8EZFNMcETEdkUEzwRkU1VqjtZRSQHQNkV9QOrB+BIBMOJFMYVnsoaF1B5Y2Nc4bFbXC2UUgGX1qpUCb4iRCQr2O26scS4wlNZ4wIqb2yMKzxVKS520RAR2RQTPBGRTdkpwU+LdQBBMK7wVNa4gMobG+MKT5WJyzZ98ERE5M1OLXgiIjJhgicisqmzPsGLyEgR+UVEdojIxCifu5mILBaRLSKyWUTu0bdPFpH9IrJO/zfK9JpJeqy/iMgIi+PLFpGNegxZ+rY6IrJARLbrX9P07SIir+qxbRCRnhbF1N70vqwTkVMicm8s3jMReVdEDovIJtO2sN8fEblJ33+7iNxkUVwviMg2/dyzRCRV354hIvmm9+1N02t66b//HXrsFVofLkhcYf/eIv1/Nkhc/zbFlC0i6/Tt0Xy/guWH6P2NKaXO2n8AHAB2AmgFIBHAegCdonj+RgB66o9rAvgVQCcAkwH8JcD+nfQYkwC01GN3WBhfNoB6PtueBzBRfzwRwHP641EAvgUgAPoAWBml39/vAFrE4j0DMBBATwCbyvv+AKgDYJf+NU1/nGZBXMMBxOuPnzPFlWHez+c4q/RYRY/9IgviCuv3ZsX/2UBx+Xz/JQCPx+D9CpYfovY3dra34M8DsEMptUspVQTgUwCXRevkSqmDSqk1+uNcAFsBNCnlJZcB+FQpVaiU2g1gB7SfIZouA/C+/vh9AJebtn+gNCsApIpII4tjGQpgp1KqtLuXLXvPlFI/ADgW4HzhvD8jACxQSh1TSh0HsADAyEjHpZSar5Ry6k9XAGha2jH02GoppVYoLUt8YPpZIhZXKYL93iL+f7a0uPRW+NUAPintGBa9X8HyQ9T+xs72BN8EwF7T830oPcFaRkQyAPQAsFLfdJd+mfWucQmG6MerAMwXkdUiMl7f1kApdVB//DuABjGKDQDGwfs/XmV4z8J9f2Lxvv0RWkvP0FJE1orIEhEZoG9roscSjbjC+b1F+/0aAOCQUmq7aVvU3y+f/BC1v7GzPcFXCiJSA8AXAO5VSp0C8AaA1gC6AzgI7RIxFvorpXoCuAjABBEZaP6m3lKJyTxZEUkEcCmA/+ibKst75hbL9ycYEXkEgBPAx/qmgwCaK6V6ALgfwEwRqRXFkCrd783HtfBuRET9/QqQH9ys/hs72xP8fgDNTM+b6tuiRkQSoP3yPlZKfQkASqlDSimXUqoEwNvwdClENV6l1H7962EAs/Q4DhldL/rXw7GIDdqHzhql1CE9xkrxniH89ydq8YnIzQAuBnC9nhigd4Ec1R+vhta/3U6PwdyNY0lc5fi9RfP9igcwFsC/TfFG9f0KlB8Qxb+xsz3B/wygrYi01FuE4wDMjtbJ9f69dwBsVUq9bNpu7rseA8AY3Z8NYJyIJIlISwBtoQ3sWBFbiojUNB5DG6TbpMdgjMLfBOC/pthu1Efy+wA4abqMtIJXy6oyvGem84Xz/swDMFxE0vTuieH6togSkZEAHgJwqVIqz7Q9XUQc+uNW0N6fXXpsp0Skj/53eqPpZ4lkXOH+3qL5f/ZCANuUUu6ul2i+X8HyA6L5N1aRUeLK8A/ayPOv0D6JH4nyuftDu7zaAGCd/m8UgA8BbNS3zwbQyPSaR/RYf0EFR+nLiK0VtBkK6wFsNt4bAHUBLAKwHcBCAHX07QLgX3psGwFkWhhbCoCjAGqbtkX9PYP2AXMQQDG0fs1by/P+QOsT36H/u8WiuHZA64c1/s7e1Pe9Qv/9rgOwBsAlpuNkQku4OwG8Bv3O9QjHFfbvLdL/ZwPFpW+fAeBOn32j+X4Fyw9R+xtjqQIiIps627toiIgoCCZ4IiKbYoInIrIpJngiIptigicisikmeLINETmtf80QkesifOyHfZ4vj+TxiazABE92lAEgrASv3/VYGq8Er5Q6P8yYiKKOCZ7saCqAAaLV+75PRByi1VP/WS+KdQcAiMhgEVkqIrMBbNG3faUXZ9tsFGgTkakAqunH+1jfZlwtiH7sTaLVEr/GdOzvReRz0eq4f6zf2QgRmSpajfANIvJi1N8dqjLKarUQnY0mQqtRfjEA6In6pFLqXBFJArBMRObr+/YEcI7SStoCwB+VUsdEpBqAn0XkC6XURBG5SynVPcC5xkIrtNUNQD39NT/o3+sBoDOAAwCWAegnIluh3dLfQSmlRF+4g8gKbMFTVTAcWo2PddDKtdaFVoMEAFaZkjsA3C0i66HVXG9m2i+Y/gA+UVrBrUMAlgA413TsfUorxLUOWtfRSQAFAN4RkbEA8gIckygimOCpKhAAf1ZKddf/tVRKGS34M+6dRAZDK1DVVynVDcBaAMkVOG+h6bEL2opMTmgVFz+HVhlybgWOT1QqJniyo1xoS6QZ5gH4k166FSLSTq+w6as2gONKqTwR6QBt2TRDsfF6H0sBXKP386dDWz4uaLVLvTZ4baXUNwDug9a1Q2QJ9sGTHW0A4NK7WmYAeAVa98gafaAzB4GXY5sL4E69n/wXaN00hmkANojIGqXU9abtswD0hVa1UwF4SCn1u/4BEUhNAP8VkWRoVxb3l+9HJCobq0kSEdkUu2iIiGyKCZ6IyKaY4ImIbIoJnojIppjgiYhsigmeiMimmOCJiGzq/wECBB7SH+NuIAAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{},"output_type":"display_data"},{"name":"stderr","output_type":"stream","text":["100%|██████████| 2000/2000 [00:19<00:00, 102.90it/s]\n"]},{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUZf4H8M83mwahJEDoJfQmPQpIFaQINrChnu1U9H541tMDKzYO652eZ0FUbOh5Kh4eShURQcDQq9IiVQg9kLqb5/fHzOzOtmQ32dkNk8/79eKV3cnszDeb8N1nnueZ7yNKKRARkf3ExToAIiKyBhM8EZFNMcETEdkUEzwRkU0xwRMR2VR8rAMwq1evnsrIyIh1GEREZ43Vq1cfUUqlB/pepUrwGRkZyMrKinUYRERnDRH5Ldj32EVDRGRTTPBERDbFBE9EZFNM8ERENsUET0RkU0zwREQ2ZWmCF5FUEflcRLaJyFYR6Wvl+YiIyMPqFvwrAOYqpToA6AZgqxUnyS9y4cs1+8DSx0REHpbd6CQitQEMBHAzACiligAUWXGup+dswcyVe9CwdjLOb13PilMQEZ11rGzBtwSQA+A9EVkrItNFJMV3JxEZLyJZIpKVk5NTrhMdPlUAADhT6KpIvEREtmJlgo8H0BPAG0qpHgDOAJjou5NSappSKlMplZmeHrCcQsjYRUNE5GFlgt8HYJ9SaqX+/HNoCT/iRAQAwPRORORhWYJXSv0OYK+ItNc3DQWwxYpzieecVhyeiOisZHU1yT8D+FhEEgHsAnCLFSfRG/Bgfici8rA0wSul1gHItPIcACBgFw0RkS9b3MnKFjwRkT9bJXgiIvKwR4J3d9GwCU9EZLBFgjem0ZQwvxMRudkiwXOaJBGRP3skeHbCExH5sUeC17+yAU9E5GGPBG9Mk+QgKxGRmz0SfKwDICKqhOyR4I1iY2zAExG52SPB6185TZKIyMMWCR7uUgXM8EREBlskeGEvPBGRH1skeCIi8scET0RkU7ZI8Jz/TkTkzxYJ3sCSBUREHrZK8JxFQ0TkYYsEz1k0RET+bJHgiYjIHxM8EZFNMcETEdmULRI8p0kSEfmzRYInIiJ/TPBERDZliwTPaZJERP5skeCJiMifLRK8McjKoVYiIg9bJHh3ZmeGJyJyi7fy4CKSDSAXgAuAUymVacV5SpTRgmeGJyIyWJrgdRcopY5E4TxcdJuIyMQWXTRGXuei20REHlYneAVgvoisFpHxgXYQkfEikiUiWTk5OeU6iZHY2UVDRORhdYLvr5TqCeAiABNEZKDvDkqpaUqpTKVUZnp6erlOYtSBZxcNEZGHpQleKbVf/3oYwCwA51lyHp+vRERkYYIXkRQRqWk8BjAcwCYrzuVeyYlNeCIiNytn0TQAMEtfJzUewEyl1FwrTuTO71YcnIjoLGVZgldK7QLQzarje5/L+ysREdlmmqSW2UuY4YmI3OyR4NmCJyLyY4sEX8I+eCIiP7ZI8EZqV2zCExG52SLBs0QBEZE/WyR43slKROTPHgne/ZUZnojIYI8Er+d1dtUQEXnYIsHH6WtuO10lsQ2EiKgSsUWCd+gZvsjFJjwRkcEWCd7ooilysgVPRGSwRYJ36Rm+mF00RERutkjwJWzBExH5sUeC1zM8EzwRkYc9Ejy7aIiI/NgiwS/feRQAUMgET0TkZosEbyhmFw0RkZutEnwRW/BERG62SvDsgyci8rBVgucsGiIiD3sleJYqICJys1WCZ7ExIiIPWyT4N67viZrJ8eyDJyIysUWCv6hLIwxuXx9OdtEQEbnZIsEDQEKccJokEZGJfRK8I45dNEREJrZJ8PEOYRcNEZGJbRJ8giOOXTRERCY2SvBswRMRmVme4EXEISJrReR/Vp6HffBERN6i0YK/B8BWq08S74iDs0RBKbbiiYgAixO8iDQFMBrAdCvPAwCJDgEAFLObhogIgPUt+H8AeAhA0L4TERkvIlkikpWTk1PuE8U7tB+F3TRERBrLEryIXAzgsFJqdWn7KaWmKaUylVKZ6enp5T5fop7g84td5T4GEZGdWNmC7wfgUhHJBvApgCEi8pFVJ2tepzoAYM+xPKtOQUR0VrEswSulJimlmiqlMgCMA/CdUuoPVp0vKUH7UUpK2AdPRATYaB58nGiDrMzvRESa+GicRCn1PYDvrTyHnt9RwmmSREQAbNmCZ4InIgJsmOCZ34mINDZK8NpXtuCJiDS2SfCit+BdHGUlIgJgowRvtODZgCci0tgmwTv0DL9w66EYR0JEVDnYJsEbg6wfr9zDipJERLBRgjfmwQOsKElEBISY4EUkRUTi9MftRORSEUmwNrTwxJkyfKGTBceIiEJtwf8AIFlEmgCYD+AGADOsCqo8vBM8SwYTEYWa4EUplQdgLIDXlVJXAehsXVjhizN10fy082jsAiEiqiRCTvAi0hfA9QDm6Nsc1oRUPmJqwR/OLYxhJERElUOoCf5eAJMAzFJKbRaRVgAWWxdW+Mwt+Ho1EmMXCBFRJRFSNUml1BIASwBAH2w9opS628rAwsU+eCIib6HOopkpIrVEJAXAJgBbRORBa0MLjyOOCZ6IyCzULppOSqlTAC4H8C2AltBm0lQajWonux/vOXomhpEQEVUOoSb4BH3e++UAZiuligFUqruJ4h1x6Nk8FQDw9tLdMY6GiCj2Qk3wbwHIBpAC4AcRaQHglFVBldcn4/vEOgQiokoj1EHWVwG8atr0m4hcYE1I5ZcQZ5vKC0REFRbqIGttEXlZRLL0fy9Ba81XKnGmgdZDpwpiGAkRUeyF2uR9F0AugKv1f6cAvGdVUJHw3LfbYh0CEVFMhdRFA6C1UuoK0/MnRWSdFQFFSvWkSnWjLRFR1IXags8Xkf7GExHpByDfmpAiI9HBBE9EVVuoCf5OAP8SkWwRyQbwGoA7LIsqAt5dxqmSRFS1hTqLZj2AbiJSS39+SkTuBbDByuCIiKj8wppXqJQ6pd/RCgD3WxBPRE2evTnWIRARxUxFJo5L2btEX59WddyPZyzPjl0gREQxVpEEX6lKFRheHdcj1iEQEVUKpfbBi0guAidyAVDNkogqKCUp1JmfRET2Vmo2VErVLO+BRSQZ2lquSfp5PldKPVHe44XKXDaYiKgqs7K5WwhgiFLqtF6J8kcR+VYptcLCc3ot/EFEVJVZluCVUgrAaf1pgv7P8n57tuCJiDSWll8UEYde0uAwgAVKqZUB9hlvFDHLycmp8DmZ34mINJYmeKWUSynVHUBTAOeJyDkB9pmmlMpUSmWmp6dX+Jzi00VzutBZ4WMSEZ2NolJAXSl1AsBiACOjcb7NT45wPy4odkXjlERElY5lCV5E0kUkVX9cDcAwAFGp4WueKplfxARPRFWTlbNoGgF4X0Qc0D5IPlNK/c/C8wXEFjwRVVWWteCVUhuUUj2UUl2VUucopZ6y6lyBPDtG6+7fmXO6jD2JiOzJtouYZtTVVhS886M1MY6EiCg2bJvgkxM8C378dvRMDCMhIooN2yb4aqYE/1nW3hhGQkQUG7ZN8LWrJ7gfl1TKupdERNaybYJvXDvZ/Xh19vEYRkJEFBu2TfAigi//73wAwKrsYzGOhogo+myb4AGgVnJC2TsREdmUrRM8C48RUVVm6wTPwVUiqspsnuCZ4Ymo6rJ1gm9omknjdJXEMBIiouizdYI3D7IWOJngiahqsXWCNytkVUkiqmKqTIL/53c7Yh0CEVFUVZkEP2N5NlycVkNEVUiVSfAA0Prhb6A4s4aIqogqleABwMlWPBFVEVUuwbObhoiqCtsn+D6t6ng9//6XHGRMnIOc3MIYRUREFB22T/ATLmjj9XzaDzsBAJsOnIxFOEREUWP7BN+/TT2v58UurYsmTliJjIjszfYJXnwSeZF+R2s8S00Skc3ZPsH7KtZr0rAFT0R2V+USfJGe4B1swRORzVW5BL/veD4ALgZCRPZXJRL8xsnD8deRHby2/XfdASzbcQTHzhQFfd19/16H4X9fYnV4RESWiI91ANFQMzkBjUy14QHgwxW/4cMVv6F6ogNbnhoZ8HWz1u6PRnhERJaoEi14ADiRF7ilnlfEMsJEZE9VJsEPaJce6xCIiKLKsgQvIs1EZLGIbBGRzSJyj1XnCkXr9BpBv/fU11uw52heFKMhIrKelS14J4AHlFKdAPQBMEFEOll4vnJ7d9lu/PO77bEOg4gooixL8Eqpg0qpNfrjXABbATSx6nwVlVvgBACcKXTGOBIiosiISh+8iGQA6AFgZTTOVx4Oh+AP01ei8xPz8FUlmT2TX+TC2j3HYx0GEZ2lLE/wIlIDwBcA7lVKnQrw/fEikiUiWTk5OZbG0rJeStDvzdlwED/uOAIAWLTtsKVxhOr+z9ZhzOvLcfQ0SxsTUfgsTfAikgAtuX+slPoy0D5KqWlKqUylVGZ6urUzXRb/ZTCyp47G5d0bl7pfZbnJdcM+raQxp3ISUXlYOYtGALwDYKtS6mWrzlMeDWoll71TGf67bj8WbDnkte3+f69DxsQ5yK+ECTmvyImdOadjHQYRRZGVd7L2A3ADgI0isk7f9rBS6hsLzxmSOwe1BgAcOlWAr9YdCPl1D8/aiAMn8vHKNT1wz6faj5Q9dTQA4PiZInyp990fyyvCmeNO5Be50K1ZaoSjL587PlyNpduPYNeUUYhjIR6iKsGyBK+U+hGVp7fDS1pKIiaN6ojXgkyNnL3+AJqmVfPbPnPlHgDA3uP+c+ZP+8y+Gf73HwAA658YjtrVEioacoUt08cXilwlSI5zxDgaIoqGKnMnayDG4h+BvP79zrCOVaICL+bd7cn5+Cxrb1jHMkSyZH28Q/tVG+WSicj+qnaCdwVOyr5W7T6GG99d5X5e6PT0sR8/U4SMiXOwaKtn5o3ySfZLfvWfHTRnw0H0enoBMibOwaQvN4YUh9NV4nXucCTo3TKlfagRkb1U6QRfHGJr9uq3fsIPpiR9xRs/uR/3eHoBAOCp/21xb1u1+5jX6wM1xCd/vRlH9VLFn6zaU+r5jauD66avRPtH54YUsy93C54JnqjKqNIJ3qkn+Ct6No3oce//bH2Z+/iOcy7+xX/uvdFF4yrRErzxwTFr7T68OO+XsGJKcLAFT1TVVOkE36WpNsPlqszIJvhQiE+7/pb3fsariwIP+pb49CTd9+/1eG3xjrDOZ6xB6yxhgieqKqp0gr+iZxMseXAw+rSqa+l5xDRaeqbQieU7jwTc7+UFvwbcHmwANxxGgi8OMu5Q5CzBzJV73Fc1RHT2qxIrOgUjImhRN3j5gkj5ev0B5BYUI616YlirRBmtfJdvE74c4ny6e3y9vOBXvLlkJ+qkJGLkOQ0rfD4iir0q3YKPpu9/yQkpuV/8z6V+2yLRghd3F41CodOF3IJir+8bc/tDHXiOhbwiJwqKK99dwkSVFRO8jy5NamPaDb1idv5N+/3qsQVNaq9/vwO/HT2DdXtPANCmbL61ZKffNE3AM2DrdJXgurdXosvk+d47BPkMWbztsFeJg/wiV8ySbKfH56H3lEUxOTfR2ahKd9EEckPfFhjWqQG2PDUCX68/gL9+Edoc9UjKyS1Ees0k9/Mr3vgJvVvW8dvv+bm/4Pm52mya7Kmj3VM2e7VIQ2aGtv/DszZiw74T2Hc8HwCw/0Q+Vv/mXYJ41e5jmLPxIAD/m6tumfGz+/gA0PHxuaiVHI8Nk0dU9MfEo19tRNv6NXHT+Rml7vdZ1l4kxWttkZP5xaXuG0xBsQs5uYVoVqd6qfs5XSWY/PVm/GlwGzRJ9b+bmehswha8j6szm0FEUD0xHtec2zwmMeTkauWB9xzzlERY6TO3vjTFLoXcgmJc9MpSzFy5x+uqwKihAwCHcwuwfMcRXP2WZ17/XTPX4l9lzNA5VRB8UZRpP+xE5jMLQ4rzoxV78MTszV7b9hzNw6FTBV7bHvp8g1fc5TH+w9UY8PxiKKXw+ep9aPPwNwGnjK7afQwfrdiDB/9T9lRXosqOCV734lXdMKBtvViHAQAY9epSbD5wstyvP3SqAF0mz8fWg/7dPWZDX1yC66b7r8Hy9tJdKCj27orpMnke/vbNVq/9ftp51O/DYMo323CkAvXrB76wGOdP/Q4Fxa6IzNl3ukow6pWl7hvVilwlmPLNVjhLlN84hFkEhj2qhONnivDuj7sDdgtS7LGLRndlr6a4slf058MHM/rVH8Pa/7ejZ9yP7/13aK3d3CDLE57IK0aHx+bCYbobK7fAibd+2OV+/uWafe4bum46PwNKKczbfMjvWOXhKlHo8NhcpNdMwrK/Dgm63/zNv2NA23QcOlWAlxb8ipeu6obEeO82y8n8YmwxfdAVFJW470CIVEpas+c4ejRL9ZoOW1U8+PkGLNx6CD2ap6JH87RYh0M+mODL8Oq1PfDj9hzc2DcDF/8zvKQbTYNe+D7ixyxteqb5bt1znpjn9/2X5/+Cbb/n4pwmtXH30Lalnmfy7M349Oc9+PDW3l7bc3ILcf30FQFfs/XgKYz/cDU6NqrlvlIZd24z7D5yBiM6N/QawzArcLrc4wylNTpdSuHYmSLUSUksNfYFWw7h9g+yMGVMF1zXOzZdekXOEuQXuVC7evSrlh7P08ptOMOYyvvZz3vRt3Vdr/GQrOxjmLvpdzx6caeIx1iVsYumDJd2a4znr+yGc5rUjnUoZ5VXv9uB+VsO4eUFv2LjPu/upoMn87HYtCzijOXZKCguwVVv/uR7GPycHXhN2rwi7erD3A21M+c0Hv1qEybMXBM0rh+3H8GR01pS8u3rN1u1+xh6Pr3AXQZ61tp9OO/ZhX4ferv0GUbZpiuoaJswcw26PTU/4PeUUli09VBE7qUIdnwg9Lrgxa4SPPTFBr/f9ZVv/oTpP+6OcHTEBB+GxrVLXwnqlXHd8fCoDlGK5uxhDBZv2n8SJ/OKMfb15e7ZOeXx9foDXgXfDGcKtTGDE3qrEgCWbve+a/gB0+BpKFdk2w/l4uDJfDw6axMO5xa6P1gMxn0D8WEuonIyrxh/+3Zr0DuHv1i9D1cH+MALxHdlMbN5mw/h1vezMM3UvRZJxsdGqL1TxlVTTpBxGqVfOVll+tJdpb5fdsMEH4Z3bznX63n3Zqm4Y1Ar9/PLujfB+IGtsfaxYbh7SJtoh1epKaVw8T9/xA3vrsTBk8FbzqH48ydrA25/bu42AJ6yDADw3rLQW4VOV4lfvfwxry9H3799514F67tth93TTH/aeRTH87SB2gS9Wuf0pbvQZfK8Mgcdp3yzFW8t2YVXF23HybxiTF+6C5+aqoo+8J/1WJUdeObU56v3YWGAJBXow8K4SjlwIr/UeMrLuDAIdfzBuGkv2PuzUr9ymrvp94jE5+uZOVtx+wdZlhw7mKvf+gk99SnM0cYEH4aURO8hi68m9MOkizr67ZeWkoj7h7dHooNvLwBM/3GXO6lv2Ff+2UGhcsQJlFJ4Yd42bDpQ+kyiM4VOlJQofLpqD9o88i1ufi/wlUWuPjX0nk/X4Yo3luNkfjGufXsF3tG7FYxqnc/M2YrcAieOnSnCtB92ImPinIDJLF+fofTqdzsw9o1leGbOVkzU1wX4cXvgWkWGv/xnPW77IAtzNhxElulDwFxnSCnl/gfANO6gyj3j5VRBMU7m+cw80o+VV+g96+rAiXy88f1OFDpdXmsYuBN8kHMYq6at2HW0XDFGyu8nC5AxcQ6WbvdfyyEQY12IQFOMV+0+ZulVSWmYgcIQ7/C0Uox1XUsTiRIDdrB2zwmcP/W7qJ0vTgT5xS78a/HOMvueOz8xD09+vdmdXEPV7UnvPm+ltJlFhvlbDmHKN9oVRc7pQr8pmeYpoDtzPP33R04X4g/v+E9dVUrhdKHT/YECaH3vV5q6cczH7Pj4XFz0ylJ3Io0TwfEzRWg56RtcM80zcL314Cnc/claFDlLcOBEvjtJHz5V4Ncd1XXyfL++fuP4f3hnJUa94imzMf7DLDw3dxvaPzoX7R+di6e+3oL/rtvvvuEu2H+N2eu1NZLLqnr6+vc7yvwgLMvR04VBb5wz7g7/8KffUFKiUOQswc6c0zgapGvJGIN5QS/jXVDswts/7Ip58T7OogmDedrgbQNalrl/YnwcnEWsnRJtcXGC95Zlh7z/+z/9VuFzvuRTCfSprz0LwJz3rFZe4dVre6B3yzrYfug05m4O3AVx6/ve3QcFxS5M/XYbZizPRu+WdUq94c3oXipylqCguATbfs/Fhys8P5txp/Oq3ccw9dttqJHkwIvztbjHndsM101fiaT4OKx+bBjOm7IInRvXwpy7BwQ8V8bEOQC00h6GXUfOYNoPOzF+YGv3eIjhXb2rrHqiZz1gpRQWbj2MXw/l+h2/tA/mQqfLfQf3fRe2w9ieTfzuUP5240EMap+O6onBU1yvZxYi0RGHX5+9yO97xpiKs0Thzo9WY77eJZZWPQFrHx/ut79vtK99twOvLd6B1BjMbDJjgg9DWnXPlLlQBtVm3t4Hl/9rmZUhUQDr955ASmJsFxZ3BWii3h1k7MBs3zHvBd07POZZwausu5mLXCV458fdeNq0utgu/epgxvJsr33fXOK95rDRki10lmDkP7QF4zfr3Vu9nl6Afm08NwHuPuK54ti437vLbco32zB+YOugg655pgbPZ1l7g5YCcZaynOadH652P/77wl/xWdZeLJs4BMUurZW9eFsOnpu7DZd0a4z7LmyL04VOdNXXfjDsOKx9qJjHXJyuEvfU3gT9fopiVwm+2+a5UjieV4yfs48hs0Uaftp1FAmOOLRvWBOvLPRey8GYPlroc7NeQbELX68/gCt7NY3KfRNM8GFIcMShRlI8Thc63YNupencuJb+OglYh71+zSSseuRCvLJwO/6+MHAteLNGtZNx+4BW7uUB29SvgR2HT5fxqqqpInfTRkJ578I9WoG+2n4V6AbLN/WfG90ogLae8NEzRe6uEwC45q3SZ/cs33nE/cFSmuyjeUG/Z7Tgb35vFb7/ResHv/fCtrj3wnZY/It3v/j+E/lYvuOI313Z2w/lYshLS7Rz6bWUDL4flsWuEjwxezNmrtyDhfcPcq9h7DsLC4DfFM/Luzf2Wnf5te+2u99P81V/odOFh7/ciC/X7sfK3cfw4lXdgv78kcIEHwF/HdkBHRrW9Nue4IjDE5d0wuD29eF0leDDFb/hg59+Q82keGx80lOsK9QP8sV/GYzkBIc7weez+yeoXw/xgy8cwRoKN5kWmzcczi39w/O6t/3HEAIpbYjqy7X7cXO/DHdyB4B/LNyOuy4IPDstUMmNbb97un5+P1ngVXPpkVmb3I935pzGX/6zHmv3nHDva8zICsVX6w54PTe6vQBgkmls57b3s9wfGJ+v3of/G9wardJrhHye8uAga5ju0e/KrJbg6QL40+DWuKBD/YD739KvJVrWS0HbBjXx1GXnYMezF2Ht48MC7ju4fXrQ88YJkJzg3e1g3IQz8aIOfrfo+3r+yq5Y8uDgUvehquv173eWvVOEzV5X+voIl77m373Z5pFvy3WuybM3exXvMxv60hJ3cgeAT3/e49f1FAm+VwN5RS4opbB422HLbkRjgg/T7QNbIXvqaPe853DFO+IQ7/NaY+ApqZQk3ai2p3Rty3raKlRGgm+SWg31g9yav+iBQbh9QEuM6dEELeqmYGA77w+RtvVrYHSXRqXGXFmKsJXlwo4NYh0CheFABe+HCEewQe1A/rfhoIWReBS7SvDCvF9wy4yfMX2pNTeiMcFXAjf0bYF7hrbFrf1b+X2vfQP/rp+b+rYAAHdiblkvxX25e/+wdl77NkmthkdGd3J/IDl8uoNKlPKa/hnI2zdmhvRzWK1aggMNawW+m/ipyzpj+k2lxxmNPk+iUI15fbn7yqmiN/8FwwRfCSTFO3DfsHYBW/AdG2kJ/vo+nkJWN52fgaUPXYB/XNMdC+8fiHOa1EbvVtoCHzf2bYHU6gkY0LYeFj0wyK9b529ju3o9N/eDvnRVN3zxp77uu3Pf/+N5WPnwUCQnOPDCld6vK48xPZqUuU+t5ODDQn8b2wXLJvpXlxzbowlu6NMi6OtGdWmINY8N41qzVGnNWJ7tdUNYpHCQtRIxRtxFPIl3/MDW+Ps13b32ExH3vN829bUPgCljuuBPg1ojtXoi1gWYp2toWDsZm54cgSJnCYa9vAQTL+rgviR1xAl6taiDrk1TMaR9ffRuVdf9uit7NUXN5Hjc+ZFWyOu7Bwah2KUwa+1+vyl3wZQ1mHxB+3Q8dnEnXP6vZQEXFSlRCo44Qc2keK9Sx/de2K7UKWcCQZ2URK9B6U/H98G4aYErVRLFQkJc5NvbbMFXIp0b18Kfh7TB8olDkD11NLKnjkanxrUgImXOmU1OcKBtgO6cQGokxaNOSiJWPzYMwzs39CsYleCI80ru2vcEI8/x9NU3Tq2G9g1rYuJF3sXVZt7uXfJ355RR7ta/BKg5eHVmU0y4oDXWPz4c791yHlql18D8+wYBgF+ph076tNMF9w/y2t68buBl+IyZTUZBMPP/nx7NUwO9JCrOzUjDs2POidn5qXIKZep12MeM+BF1IvKuiBwWkU1l702AlkQfGN7ea0A1GoxB2/QagQdqzcb2bIKayfFeyffiro0w+ZJO+Pi23ujT0vPBMPuufnpdGO25iNad8sKVXXG7fifwYxd3woMjOnjVMm9YOxnZU0djy1MjcGNfT9dLh4Zagi+toXO9qSb7W/ri6Zd117qGHKYPybLqBBnvSTjuGdoWU8d2cT+vHuRmq//ceT7G+SwHWVnqFgWa7hvIc1d0KXsnijkru2hmAHgNwAcWnoMi4O4hbZDZIg3ntyl7tszLV3f32/badT29nj86uiN6NE913z2o9GuEOAFeNHU3PTK69MUd4h1xeOqyczCyc0OvW9HjSrmaeXZMF3ysF6xqUTfF6wYX800nZV0RDWqXjt1HzmBk54Y4croQWaaFyj+7o6/XnGpAKyV9nz7A3bxOdSQlxKFn8zSMm7Yi4B2oDp/WWmJ8nF8ly9KM6Nwg4ApaceKp8GiYMqYLHp4VWq2daTdkYuALi8vcr019a+dvl8djF+zrekoAAA5pSURBVHfyuov3bGJB4107rjWHBZRSPwAIfaVoipl4R5zf9MmKuG1AK/RqUcf93N2CD3lZCG/nt6nnleDL+3/BSOotgnTpTLqoA2bf1Q/ZU0fjln4ZAIBLuzf2W6mpS5Pa6Ny4FoZ30qZlXp3ZFDP+eJ5XvL1a1IGIeBWle3R0R3w1oZ/7+YMj2gPQZgd9Or6Pe3vDWslYMWkoZt7WG7f2b4kVk4bi5vMzvGKok5Lo7sb7y3Dtg6VN/Rp+4zUbJw/Hdb2bu+/fKEtKkv9Vx7QbemFYJ+8pqM3StPcw1L+bW/u3dF+1BXLz+RlYeP/AgN/r2rTsxXZGd2kUcMaZr/d8Sn5b4b+m33GoqiVYU1qDg6xkOaPvvF+E5tOnJHn+bAM1xGff1c9ds93XR7f2RruGWuuzX5u66NEsDa/pJV7vMCXjFnVTsGvKKHe/6NieTd0FtqolOjDn7gFQSqFE+bfGzQa1S8fDozrgip5NUdenC2zCBW0wIcCdmSseHgpA66oyrqomX9rZq55MtQTPe/CHPi3w4vxfMbpLI1zWvQlGdG7ormETr/dn3T20LV5Z5F0vxVA90YG8Ihdm39UPdWsk4bb+Lb1WVxreuSE6NqrlXijjjoGtUF//EKpXIxFHzxRh5e5j6N2yDnpPWeR17GvPa45PVu1BRr0U3NCnBW4b0MpvHwComRzvnjAAaHeHp1VPwMQvN2LypZ0x9vXlAWM3PH5JJ+wNciOTWZcKrMx2U98WIRWma9ugBt6+MTOsuvMt6obfJRiKmCd4ERkPYDwANG8emzUtyVpdm6Zi3ePDkFq99PVNQ5Wc4MCmJ0egeoIj4MBU16apfsWlDP1NHzIf36a1mi/okO5Vf8VQ1qCXiPjdVxDoGOMHll1aGtCqTbpKKZM7qktDtG9QC68s+tXrqiK1eiI2TB6OGnrlRPPUWOPDp7QPoWV/HYL8Yhcap2pjP3Vq+P+ejCm8dVMSMWmUtgZCQ32Fswa1knFpt8YAgE6Nanktcj7u3Gb4ZNUeDNTf9wYB7mOYOrYLxvTUxknm3N0fK3Ydw639W0IphX4+V28A8PwVXfHQFxvcz41uuFDmkldkpsrgDvUxtGMDpFZPwIm8Ytz47iqse3wYalfTxo9aTvoGgDaeEmy67xU9m+ILU1lpw4w/WnNlEfMEr5SaBmAaAGRmZrKAuk1FKrkbaiRF7k+3V4s66BV8Gn3UGEkymNev1waN77nQv7ulVnLgsrTmxL7ogUHYfzwfPZqnYsfh0xijt4rTUhKRZn6NflmUXjMJr13bAwDcd18HqpLpHWNPDH7xe/fzbs1S/Qp9+Rp3nufDqnPj2ujcWGtlm6cDm7WuH7i1W9aSmoD3mg5v35iJPcfyQu63T453oG9rzyQC35+rWZ1q2HssH/GOuKCD9M9f2TVggq9fs+zYyyPmCZ7obJFWPcG9RF9l9/FtvTF/8+9eg3et02ugtV7cqkfzNMy8rTeSAvT9XnNuMyzbeRQvXtkV9fUWt9GCL2uWTUYIs48u694Yh08VYnjnBhgSpIaTr6Zp1dxXWUnxnpg/MI191K+VjKlju3gt3tIktRr2m5YrNH/gGeMKvgn+0dEd8cycrX4xVCujBPXsCf3d5UPq10rGm3/ohTs/0kobP3FJJyQnOEq9krKCZQleRD4BMBhAPRHZB+AJpdQ7Vp2PyGrz7h2IvQG6ciqjfm3qedVwDyTYrKnU6oleiRPQxj0+Hd8HHRvVCjmG3i3rBNz+yrgeIR/D8ONfh7jHQIzCek1Sq/kN8iYlxPm87gJs2n8Kl7ymLbAeqIaUudvEaJU/N3ebX4nvYNNeDWkpiUhL8VypjjynITY/OQJnipxeLfR1jw9D96e0xVfSayaF9Z6Gy7IEr5S61qpjE8VC/VrJ7hZtVdTH5+a3YO67sB1Skhx+M38ipbRCf0YP0lW9mmLqFV0hIuhimoXjiBN8NaGfV1mQl67uhi/W7PNafWnzkyOxcvdR3PDOKvzjmu7YfyIfbcsxNTQlKd5rUgCgfYAad2OvnDTUkhucDOyiIaKICjRGEAkXdmyAhVsPoVlaNfRvUw9/HuI/A2l010ZYt/cE7ruwXdDukO7N/Afg5947AHVTPLOcEuPjMKBtOrY9PdKvnlMkzJrQD0u351ia3AFAyrvCuhUyMzNVVlboU4uIqOooKHbh6JkiNEkN/07vzQdOYtXuY7ilX9lrKZ9tRGS1UipgKVW24InorJCc4ChXcge8Z+dUJZWjAAYREUUcEzwRkU0xwRMR2RQTPBGRTTHBExHZFBM8EZFNMcETEdkUEzwRkU1VqjtZRSQHQNkV9QOrB+BIBMOJFMYVnsoaF1B5Y2Nc4bFbXC2UUgGX1qpUCb4iRCQr2O26scS4wlNZ4wIqb2yMKzxVKS520RAR2RQTPBGRTdkpwU+LdQBBMK7wVNa4gMobG+MKT5WJyzZ98ERE5M1OLXgiIjJhgicisqmzPsGLyEgR+UVEdojIxCifu5mILBaRLSKyWUTu0bdPFpH9IrJO/zfK9JpJeqy/iMgIi+PLFpGNegxZ+rY6IrJARLbrX9P07SIir+qxbRCRnhbF1N70vqwTkVMicm8s3jMReVdEDovIJtO2sN8fEblJ33+7iNxkUVwviMg2/dyzRCRV354hIvmm9+1N02t66b//HXrsFVofLkhcYf/eIv1/Nkhc/zbFlC0i6/Tt0Xy/guWH6P2NKaXO2n8AHAB2AmgFIBHAegCdonj+RgB66o9rAvgVQCcAkwH8JcD+nfQYkwC01GN3WBhfNoB6PtueBzBRfzwRwHP641EAvgUgAPoAWBml39/vAFrE4j0DMBBATwCbyvv+AKgDYJf+NU1/nGZBXMMBxOuPnzPFlWHez+c4q/RYRY/9IgviCuv3ZsX/2UBx+Xz/JQCPx+D9CpYfovY3dra34M8DsEMptUspVQTgUwCXRevkSqmDSqk1+uNcAFsBNCnlJZcB+FQpVaiU2g1gB7SfIZouA/C+/vh9AJebtn+gNCsApIpII4tjGQpgp1KqtLuXLXvPlFI/ADgW4HzhvD8jACxQSh1TSh0HsADAyEjHpZSar5Ry6k9XAGha2jH02GoppVYoLUt8YPpZIhZXKYL93iL+f7a0uPRW+NUAPintGBa9X8HyQ9T+xs72BN8EwF7T830oPcFaRkQyAPQAsFLfdJd+mfWucQmG6MerAMwXkdUiMl7f1kApdVB//DuABjGKDQDGwfs/XmV4z8J9f2Lxvv0RWkvP0FJE1orIEhEZoG9roscSjbjC+b1F+/0aAOCQUmq7aVvU3y+f/BC1v7GzPcFXCiJSA8AXAO5VSp0C8AaA1gC6AzgI7RIxFvorpXoCuAjABBEZaP6m3lKJyTxZEUkEcCmA/+ibKst75hbL9ycYEXkEgBPAx/qmgwCaK6V6ALgfwEwRqRXFkCrd783HtfBuRET9/QqQH9ys/hs72xP8fgDNTM+b6tuiRkQSoP3yPlZKfQkASqlDSimXUqoEwNvwdClENV6l1H7962EAs/Q4DhldL/rXw7GIDdqHzhql1CE9xkrxniH89ydq8YnIzQAuBnC9nhigd4Ec1R+vhta/3U6PwdyNY0lc5fi9RfP9igcwFsC/TfFG9f0KlB8Qxb+xsz3B/wygrYi01FuE4wDMjtbJ9f69dwBsVUq9bNpu7rseA8AY3Z8NYJyIJIlISwBtoQ3sWBFbiojUNB5DG6TbpMdgjMLfBOC/pthu1Efy+wA4abqMtIJXy6oyvGem84Xz/swDMFxE0vTuieH6togSkZEAHgJwqVIqz7Q9XUQc+uNW0N6fXXpsp0Skj/53eqPpZ4lkXOH+3qL5f/ZCANuUUu6ul2i+X8HyA6L5N1aRUeLK8A/ayPOv0D6JH4nyuftDu7zaAGCd/m8UgA8BbNS3zwbQyPSaR/RYf0EFR+nLiK0VtBkK6wFsNt4bAHUBLAKwHcBCAHX07QLgX3psGwFkWhhbCoCjAGqbtkX9PYP2AXMQQDG0fs1by/P+QOsT36H/u8WiuHZA64c1/s7e1Pe9Qv/9rgOwBsAlpuNkQku4OwG8Bv3O9QjHFfbvLdL/ZwPFpW+fAeBOn32j+X4Fyw9R+xtjqQIiIps627toiIgoCCZ4IiKbYoInIrIpJngiIptigicisikmeLINETmtf80QkesifOyHfZ4vj+TxiazABE92lAEgrASv3/VYGq8Er5Q6P8yYiKKOCZ7saCqAAaLV+75PRByi1VP/WS+KdQcAiMhgEVkqIrMBbNG3faUXZ9tsFGgTkakAqunH+1jfZlwtiH7sTaLVEr/GdOzvReRz0eq4f6zf2QgRmSpajfANIvJi1N8dqjLKarUQnY0mQqtRfjEA6In6pFLqXBFJArBMRObr+/YEcI7SStoCwB+VUsdEpBqAn0XkC6XURBG5SynVPcC5xkIrtNUNQD39NT/o3+sBoDOAAwCWAegnIluh3dLfQSmlRF+4g8gKbMFTVTAcWo2PddDKtdaFVoMEAFaZkjsA3C0i66HVXG9m2i+Y/gA+UVrBrUMAlgA413TsfUorxLUOWtfRSQAFAN4RkbEA8gIckygimOCpKhAAf1ZKddf/tVRKGS34M+6dRAZDK1DVVynVDcBaAMkVOG+h6bEL2opMTmgVFz+HVhlybgWOT1QqJniyo1xoS6QZ5gH4k166FSLSTq+w6as2gONKqTwR6QBt2TRDsfF6H0sBXKP386dDWz4uaLVLvTZ4baXUNwDug9a1Q2QJ9sGTHW0A4NK7WmYAeAVa98gafaAzB4GXY5sL4E69n/wXaN00hmkANojIGqXU9abtswD0hVa1UwF4SCn1u/4BEUhNAP8VkWRoVxb3l+9HJCobq0kSEdkUu2iIiGyKCZ6IyKaY4ImIbIoJnojIppjgiYhsigmeiMimmOCJiGzq/wECBB7SH+NuIAAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{},"output_type":"display_data"}],"source":["### Define optimizer and training operation ###\n","\n","'''TODO: instantiate a new model for training using the `build_model`\n","  function and the hyperparameters created above.'''\n","model = build_model(vocab_size, embedding_dim, rnn_units, batch_size)\n","# model = build_model('''TODO: arguments''')\n","\n","'''TODO: instantiate an optimizer with its learning rate.\n","  Checkout the tensorflow website for a list of supported optimizers.\n","  https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/\n","  Try using the Adam optimizer to start.'''\n","# optimizer = # TODO\n","optimizer = tf.keras.optimizers.Adam(learning_rate)\n","\n","@tf.function\n","def train_step(x, y): \n","    # Use tf.GradientTape()\n","    with tf.GradientTape() as tape:\n","        '''TODO: feed the current input into the model and generate predictions'''\n","        y_hat = model(x) # TODO\n","        # y_hat = model('''TODO''')\n","\n","        '''TODO: compute the loss!'''\n","        loss = compute_loss(y, y_hat) # TODO\n","        # loss = compute_loss('''TODO''', '''TODO''')\n","\n","        # Now, compute the gradients \n","        '''\n","            TODO: complete the function call for gradient computation. \n","            Remember that we want the gradient of the loss with respect all \n","            of the model parameters. \n","\n","            HINT: use `model.trainable_variables` to get a list of all model\n","            parameters.\n","        '''\n","        # grads = tape.gradient('''TODO''', '''TODO''')\n","        # notice that this grads object is updated\n","        grads = tape.gradient(loss, model.trainable_variables) # TODO\n","\n","        # Apply the gradients to the optimizer so it can update the model accordingly\n","        optimizer.apply_gradients(zip(grads, model.trainable_variables))\n","\n","    # output\n","    return loss\n","\n","##################\n","# Begin training!#\n","##################\n","\n","# setup\n","history = []\n","plotter = mdl.util.PeriodicPlotter(sec=2, xlabel='Iterations', ylabel='Loss')\n","if hasattr(tqdm, '_instances'): tqdm._instances.clear() # clear if it exists\n","\n","# start\n","for iter in tqdm(range(num_training_iterations)):\n","    # Grab a batch and propagate it through the network\n","    x_batch, y_batch = get_batch(vectorized_songs, seq_length, batch_size)\n","    loss = train_step(x_batch, y_batch)\n","\n","    # Update the progress bar\n","    history.append(loss.numpy().mean())\n","    plotter.plot(history)\n","\n","    # Update the model with the changed weights!\n","    if iter % 100 == 0:\n","        model.save_weights(checkpoint_prefix)\n","    \n","# Save the trained model and the weights\n","model.save_weights(checkpoint_prefix)"]},{"cell_type":"markdown","metadata":{"id":"kKkD5M6eoSiN"},"source":["## Generate music using the RNN model\n","\n","Now, we can use our trained RNN model to generate some music! When generating music, we'll have to feed the model some sort of seed to get it started (because it can't predict anything without something to start with!).\n","\n","Once we have a generated seed, we can then iteratively predict each successive character (remember, we are using the ABC representation for our music) using our trained RNN. More specifically, recall that our RNN outputs a `softmax` over possible successive characters. For inference, we iteratively sample from these distributions, and then use our samples to encode a generated song in the ABC format.\n","\n","Then, all we have to do is write it to a file and listen!"]},{"cell_type":"markdown","metadata":{"id":"JIPcXllKjkdr"},"source":["### Restore the latest checkpoint\n","\n","To keep this inference step simple, we will use a batch size of 1. Because of how the RNN state is passed from timestep to timestep, the model will only be able to accept a fixed batch size once it is built. \n","\n","To run the model with a different `batch_size`, we'll need to rebuild the model and restore the weights from the latest checkpoint, i.e., the weights after the last checkpoint during training:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":296,"status":"ok","timestamp":1659045776633,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"LycQ-ot_jjyu","outputId":"c8a1eb3f-f802-42b8-85f5-bde018dd1773"},"outputs":[{"name":"stdout","output_type":"stream","text":["Model: \"sequential_2\"\n","_________________________________________________________________\n"," Layer (type)                Output Shape              Param #   \n","=================================================================\n"," embedding_2 (Embedding)     (1, None, 256)            21248     \n","                                                                 \n"," lstm_2 (LSTM)               (1, None, 1024)           5246976   \n","                                                                 \n"," dense_2 (Dense)             (1, None, 83)             85075     \n","                                                                 \n","=================================================================\n","Total params: 5,353,299\n","Trainable params: 5,353,299\n","Non-trainable params: 0\n","_________________________________________________________________\n"]}],"source":["# build model\n","'''TODO: Rebuild the model using a batch_size=1'''\n","# model = build_model('''TODO''', '''TODO''', '''TODO''', batch_size=1)\n","model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1) # TODO\n","\n","# Restore the model weights for the last checkpoint after training\n","model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n","model.build(tf.TensorShape([1, None]))\n","\n","# summary\n","model.summary()"]},{"cell_type":"markdown","metadata":{"id":"I9b4V2C8N62l"},"source":["Notice that we have fed in a fixed `batch_size` of 1 for inference."]},{"cell_type":"markdown","metadata":{"id":"DjGz1tDkzf-u"},"source":["### The prediction procedure\n","\n","Now, we're ready to write the code to generate text in the ABC music format:\n","\n","* Initialize a \"seed\" start string and the RNN state, and set the number of characters we want to generate.\n","\n","* Use the start string and the RNN state to obtain the probability distribution over the next predicted character.\n","\n","* Sample from multinomial distribution to calculate the index of the predicted character. This predicted character is then used as the next input to the model.\n","\n","* At each time step, the updated RNN state is fed back into the model, so that it now has more context in making the next prediction. After predicting the next character, the updated RNN states are again fed back into the model, which is how it learns sequence dependencies in the data, as it gets more information from the previous predictions.\n","\n","![LSTM inference](https://raw.githubusercontent.com/aamini/introtodeeplearning/2019/lab1/img/lstm_inference.png)\n","\n","Complete and experiment with this code block (as well as some of the aspects of network definition and training!), and see how the model performs. How do songs generated after training with a small number of epochs compare to those generated after a longer duration of training?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WvuwZBX5Ogfd"},"outputs":[],"source":["### Prediction of a generated song ###\n","def generate_text(model, start_string, generation_length=1000):\n","    # Evaluation step (generating ABC text using the learned RNN model)\n","\n","    '''TODO: convert the start string to numbers (vectorize)'''\n","    input_eval = [char2idx[s] for s in start_string] # TODO\n","    # input_eval = ['''TODO''']\n","    input_eval = tf.expand_dims(input_eval, 0)\n","\n","    # Empty string to store our results\n","    text_generated = []\n","\n","    # Here batch size == 1\n","    model.reset_states()\n","    tqdm._instances.clear()\n","\n","    for i in tqdm(range(generation_length)):\n","        '''TODO: evaluate the inputs and generate the next character predictions'''\n","        predictions = model(input_eval)\n","        # predictions = model('''TODO''')\n","\n","        # Remove the batch dimension\n","        predictions = tf.squeeze(predictions, 0)\n","\n","        '''TODO: use a multinomial distribution to sample'''\n","        predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n","        # predicted_id = tf.random.categorical('''TODO''', num_samples=1)[-1,0].numpy()\n","\n","        # Pass the prediction along with the previous hidden state\n","        #   as the next inputs to the model\n","        input_eval = tf.expand_dims([predicted_id], 0)\n","\n","        '''TODO: add the predicted character to the generated text!'''\n","        # Hint: consider what format the prediction is in vs. the output\n","        text_generated.append(idx2char[predicted_id]) # TODO \n","        # text_generated.append('''TODO''')\n","\n","    # output\n","    return (start_string + ''.join(text_generated))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":5897,"status":"ok","timestamp":1659045782527,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"ktovv0RFhrkn","outputId":"b0cef9ef-524d-40ce-c55b-3d3771ad54ec"},"outputs":[{"name":"stderr","output_type":"stream","text":["100%|██████████| 1000/1000 [00:05<00:00, 168.15it/s]\n"]}],"source":["# generated_text = generate_text('''TODO''', start_string=\"X\", generation_length=1000)\n","'''TODO: Use the model and the function defined above to generate ABC format text of length 1000!\n","    As you may notice, ABC files start with \"X\" - this may be a good start string.'''\n","generated_text = generate_text(model, start_string=\"X\", generation_length=1000) # TODO"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":122},"executionInfo":{"elapsed":15,"status":"ok","timestamp":1659045782528,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"hd-g_19FNRv0","outputId":"e0d45810-396b-4861-ec8e-08bf17c6059a"},"outputs":[{"data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["\"X:1/8\\nK:F Major\\nA|FDD AFA|def gfe|cdB AFA|BdB AFD|!\\nF2G AGF|EcE d2B|cfe C2|!\\nF3 D2|G2 c2 B3/2c/|cA BA|!\\nfb af:|!\\necA cAG|c3 efg|!\\nagf edc|dfd cAF|GBd efg|!\\nfga a2f|dfe dff|gee dcB|def g2:|!\\n[2 dggf gbag|fdcA dcd|e|fcAF GFED|]!\\nDAaA gAfA|edcd Acde|fde^cd egfe|defg a^g|]!\\n\\nX:26\\nT:Padrmingan's Ryan\\nZ: id:dc-reel-211\\nM:C\\nL:1/8\\nK:A Major\\nB|ADDAF DFc|cdc A3|AGE GFE:|!\\nBc|E2F GFG|A2G AFD|E3 DG|!\\nFEF GED|EDC BFc|dgf e^dc|!\\nG>A BA|G3 A FA|E2 D:|!\\ned cA|B2 G>B|ce fe|B/c/dcB|AFA GED:|!\\n\\nX:167\\nT:ancy Cnor\\nZ: id:dc-reel-83\\nM:C\\nL:1/8\\nK:A Major\\nGB|BGE E2A|cFE BAF|d3 d2B|!\\nG3 A BG|EA E2|D2 E2 D2|d4 B4|d2 Bc|d4 B2|e4 Bd|!\\nE2 G>A|B2 F2|D3 e ed|c2 d2|e3 e ec|!\\nd2 ed =cA|e2 d2|e4 de|f3/2c/ A2|d2 c2 d2|e2 dc|BA B>c|AF FE|!\\nFA AB|c/B/Ace|fdc d2|ece dcB|]!\\ndB|d2 d>f|dc d2|e2 e>f|ge f2|ed c2|B/c/d Ba|!\\nae ef|ga b/g/f/g/f/|bg e/|dB AB|!\\n[1 Bc|d/c/C/|dc B/c/|dc/d/ cB|AF FD|G2:|!\\n\\nX:35\\nT:Black Jenasty Whel\\nZ: id:dc-jig-5\\nM:6/8\\nL:1/8\\nK:A Dorian\\nedc|Bdc-jig-89\\nM:6/8\\nL:1/8\\nK:G Major\\ne|dBB2 dBAB|GEDE GAcd|fdd^c d=cAA\""]},"execution_count":32,"metadata":{},"output_type":"execute_result"}],"source":["# display content\n","# this should be a bunch of strings and characters\n","# that are defined above\n","generated_text"]},{"cell_type":"markdown","metadata":{"id":"AM2Uma_-yVIq"},"source":["### Play back the generated music!\n","\n","We can now call a function to convert the ABC format text to an audio file, and then play that back to check out our generated music! Try training longer if the resulting song is not long enough, or re-generating the song!"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":14,"status":"ok","timestamp":1659045782528,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"LrOtG64bfLto","outputId":"4fbded93-ce3e-4328-e391-db613e6eaffe"},"outputs":[{"name":"stdout","output_type":"stream","text":["Found 3 songs in text\n"]}],"source":["### Play back generated songs ###\n","generated_songs = mdl.lab1.extract_song_snippet(generated_text)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":12,"status":"ok","timestamp":1659045782528,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"Lp7rBBUqJc_b","outputId":"d77c7e57-fe1e-48e9-f694-8745516eeaa0"},"outputs":[{"data":{"text/plain":["3"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["# length of the generated songs\n","len(generated_songs)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":11,"status":"ok","timestamp":1659045782528,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"_x0JrOJeJVid","outputId":"7f7962b3-8357-48f3-bf81-6a99ac88d9bd"},"outputs":[{"name":"stdout","output_type":"stream","text":["0 ,  --, \n","X:1/8\n","K:F Major\n","A|FDD AFA|def gfe|cdB AFA|BdB AFD|!\n","F2G AGF|EcE d2B|cfe C2|!\n","F3 D2|G2 c2 B3/2c/|cA BA|!\n","fb af:|!\n","ecA cAG|c3 efg|!\n","agf edc|dfd cAF|GBd efg|!\n","fga a2f|dfe dff|gee dcB|def g2:|!\n","[2 dggf gbag|fdcA dcd|e|fcAF GFED|]!\n","DAaA gAfA|edcd Acde|fde^cd egfe|defg a^g|]!\n","1 ,  --, \n","X:26\n","T:Padrmingan's Ryan\n","Z: id:dc-reel-211\n","M:C\n","L:1/8\n","K:A Major\n","B|ADDAF DFc|cdc A3|AGE GFE:|!\n","Bc|E2F GFG|A2G AFD|E3 DG|!\n","FEF GED|EDC BFc|dgf e^dc|!\n","G>A BA|G3 A FA|E2 D:|!\n","ed cA|B2 G>B|ce fe|B/c/dcB|AFA GED:|!\n"]}],"source":["# display enumerated output of the generated songs\n","for i, song in enumerate(generated_songs[0:2]):\n","    print(i, ',  --, ')\n","    print(song)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"background_save":true,"base_uri":"https://localhost:8080/","height":870,"output_embedded_package_id":"1lOSVQRN_2Y04HGhcG9NeXJRsQfIvFrDK"},"executionInfo":{"elapsed":33780,"status":"ok","timestamp":1659045817735,"user":{"displayName":"Yiqiao Yin","userId":"16271867367914268422"},"user_tz":240},"id":"MR-ztxb8JSwZ","outputId":"122c5b98-c78d-45c5-b0a3-a430b69d7dd6"},"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}],"source":["# generate hearable file output \n","# if the file is a wave format\n","for i, song in enumerate(generated_songs): \n","    # Synthesize the waveform from a song\n","    waveform = mdl.lab1.play_song(song)\n","\n","    # If its a valid song (correct syntax), lets play it! \n","    if waveform:\n","        print(\"Generated song\", i)\n","        print(song)\n","        ipythondisplay.display(waveform)\n","    else:\n","        print(\"Generated song\", i)\n","        print('not wave form!')"]},{"cell_type":"markdown","metadata":{"id":"HgVvcrYmSKGG"},"source":["## Experiment and **get awarded for the best songs**!\n","\n","Congrats on making your first sequence model in TensorFlow! It's a pretty big accomplishment, and hopefully you have some sweet tunes to show for it.\n","\n","Consider how you may improve your model and what seems to be most important in terms of performance. Here are some ideas to get you started:\n","\n","*  How does the number of training epochs affect the performance?\n","*  What if you alter or augment the dataset? \n","*  Does the choice of start string significantly affect the result? \n","\n","Try to optimize your model and submit your best song! **MIT students and affiliates will be eligible for prizes during the IAP offering**. To enter the competition, MIT students and affiliates should upload the following to the course Canvas:\n","\n","* a recording of your song;\n","* Jupyter notebook with the code you used to generate the song;\n","* a description and/or diagram of the architecture and hyperparameters you used -- if there are any additional or interesting modifications you made to the template code, please include these in your description.\n","\n","You can also tweet us at [@MITDeepLearning](https://twitter.com/MITDeepLearning) a copy of the song! See this example song generated by a previous 6.S191 student (credit Ana Heart): <a href=\"https://twitter.com/AnaWhatever16/status/1263092914680410112?s=20\">song from May 20, 2020.</a></blockquote> \n","<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n","\n","Have fun and happy listening!\n"]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"ex14 - music generation.ipynb","provenance":[{"file_id":"https://github.com/aamini/introtodeeplearning/blob/master/lab1/solutions/Part2_Music_Generation_Solution.ipynb","timestamp":1658616494410}],"toc_visible":true},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}