Upload 145 files

.gitattributes

@@ -111,3 +111,21 @@ Transformer[[:space:]]Mechanism/Transformer_Implementation/home/jovyan/work/W4A1
 Transformer[[:space:]]Mechanism/Transformer_Implementation/home/jovyan/work/W4A1/transformer.png filter=lfs diff=lfs merge=lfs -text
 Transformer[[:space:]]Mechanism/Transformer[[:space:]]Pre-Processing/home/jovyan/work/W4A4_UGL_POS/glove/glove.6B.100d.txt filter=lfs diff=lfs merge=lfs -text
 Transformer[[:space:]]Mechanism/Transformer[[:space:]]Pre-Processing/home/jovyan/work/W4A4_UGL_POS/preprocessing.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/NMT_with_Attention/NMT[[:space:]]with[[:space:]]MBR/Files/tf/images/att.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/NMT_with_Attention/NMT[[:space:]]with[[:space:]]MBR/Files/tf/images/attention.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/NMT_with_Attention/NMT[[:space:]]with[[:space:]]MBR/Files/tf/images/NMTModel.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/NMT_with_Attention/NMT[[:space:]]with[[:space:]]MBR/Files/tf/por-eng/por.txt filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/data/c4-en-10k.json filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/data/c4-en-10k.jsonl filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/data/train-v2.0.json filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/images/colab_help_2.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/images/fulltransformer.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/images/qa.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/pretrained_models/model_c4.data-00000-of-00001 filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/QA/QA_T5/Files/tf/pretrained_models/model_qa3.data-00000-of-00001 filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/Text_Summarization/Summarization/tf/images/decoder_layer.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/Text_Summarization/Summarization/tf/images/decoder.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/Text_Summarization/Summarization/tf/images/encoder_layer.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/Text_Summarization/Summarization/tf/images/encoder.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/Text_Summarization/Summarization/tf/images/self-attention.png filter=lfs diff=lfs merge=lfs -text
+NLP[[:space:]]with[[:space:]]Attention[[:space:]]Models/Text_Summarization/Summarization/tf/images/transformer.png filter=lfs diff=lfs merge=lfs -text

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/.ipynb_checkpoints/C4W1_Basic_Attention-checkpoint.ipynb

@@ -0,0 +1,302 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9c74bac5",
+   "metadata": {},
+   "source": [
+    "# Basic Attention Operation: Ungraded Lab\n",
+    "\n",
+    "As you've learned, attention allows a seq2seq decoder to use information from each encoder step instead of just the final encoder hidden state. In the attention operation, the encoder outputs are weighted based on the decoder hidden state, then combined into one context vector. This vector is then used as input to the decoder to predict the next output step.\n",
+    "\n",
+    "In this ungraded lab, you'll implement a basic attention operation as described in [Bhadanau, et al (2014)](https://arxiv.org/abs/1409.0473) using Numpy.\n",
+    "\n",
+    "This is a practice notebook, where you can train writing your code. All of the solutions are provided at the end of the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a5288920",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import the libraries and define the functions you will need for this lab\n",
+    "import numpy as np\n",
+    "\n",
+    "def softmax(x, axis=0):\n",
+    "    \"\"\" Calculate softmax function for an array x along specified axis\n",
+    "    \n",
+    "        axis=0 calculates softmax across rows which means each column sums to 1 \n",
+    "        axis=1 calculates softmax across columns which means each row sums to 1\n",
+    "    \"\"\"\n",
+    "    return np.exp(x) / np.expand_dims(np.sum(np.exp(x), axis=axis), axis)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9a6e0293",
+   "metadata": {},
+   "source": [
+    "## 1: Calculating alignment scores\n",
+    "\n",
+    "The first step is to calculate the alignment scores. This is a measure of similarity between the decoder hidden state and each encoder hidden state. From the paper, this operation looks like\n",
+    "\n",
+    "$$\n",
+    "\\large e_{ij} = v_a^\\top \\tanh{\\left(W_a s_{i-1} + U_a h_j\\right)}\n",
+    "$$\n",
+    "\n",
+    "where $W_a \\in \\mathbb{R}^{n\\times m}$, $U_a \\in \\mathbb{R}^{n \\times m}$, and $v_a \\in \\mathbb{R}^m$\n",
+    "are the weight matrices and $n$ is the hidden state size. In practice, this is implemented as a feedforward neural network with two layers, where $m$ is the size of the layers in the alignment network. It looks something like:\n",
+    "\n",
+    "![alignment model](./images/alignment_model_3.jpg)\n",
+    "\n",
+    "Here $h_j$ are the encoder hidden states for each input step $j$ and $s_{i - 1}$ is the decoder hidden state of the previous step. The first layer corresponds to $W_a$ and $U_a$, while the second layer corresponds to $v_a$.\n",
+    "\n",
+    "To implement this, first concatenate the encoder and decoder hidden states to produce an array with size $K \\times 2n$ where $K$ is the number of encoder states/steps. For this, use `np.concatenate` ([docs](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html)). Note that there is only one decoder state so you'll need to reshape it to successfully concatenate the arrays. The easiest way is to use `decoder_state.repeat` ([docs](https://numpy.org/doc/stable/reference/generated/numpy.repeat.html#numpy.repeat)) to match the hidden state array size.\n",
+    "\n",
+    "Then, apply the first layer as a matrix multiplication between the weights and the concatenated input. Use the tanh function to get the activations. Finally, compute the matrix multiplication of the second layer weights and the activations. This returns the alignment scores."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72857076",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "hidden_size = 16\n",
+    "attention_size = 10\n",
+    "input_length = 5\n",
+    "\n",
+    "np.random.seed(42)\n",
+    "\n",
+    "# Synthetic vectors used to test\n",
+    "encoder_states = np.random.randn(input_length, hidden_size)\n",
+    "decoder_state = np.random.randn(1, hidden_size)\n",
+    "\n",
+    "# Weights for the neural network, these are typically learned through training\n",
+    "# Use these in the alignment function below as the layer weights\n",
+    "layer_1 = np.random.randn(2 * hidden_size, attention_size)\n",
+    "layer_2 = np.random.randn(attention_size, 1)\n",
+    "\n",
+    "# Implement this function. Replace None with your code. Solution at the bottom of the notebook\n",
+    "def alignment(encoder_states, decoder_state):\n",
+    "    # First, concatenate the encoder states and the decoder state\n",
+    "    inputs = None\n",
+    "    assert inputs.shape == (input_length, 2 * hidden_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the concatenated inputs and layer_1, with tanh activation\n",
+    "    activations = None\n",
+    "    assert activations.shape == (input_length, attention_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the activations with layer_2. Remember that you don't need tanh here\n",
+    "    scores = None\n",
+    "    assert scores.shape == (input_length, 1)\n",
+    "    \n",
+    "    return scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fb638355",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run this to test your alignment function\n",
+    "scores = alignment(encoder_states, decoder_state)\n",
+    "print(scores)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f26aae76",
+   "metadata": {},
+   "source": [
+    "If you implemented the function correctly, you should get these scores:\n",
+    "\n",
+    "```python\n",
+    "[[4.35790943]\n",
+    " [5.92373433]\n",
+    " [4.18673175]\n",
+    " [2.11437202]\n",
+    " [0.95767155]]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "58b8cfa9",
+   "metadata": {},
+   "source": [
+    "## 2: Turning alignment into weights\n",
+    "\n",
+    "The next step is to calculate the weights from the alignment scores. These weights determine the encoder outputs that are the most important for the decoder output. These weights should be between 0 and 1. You can use the softmax function (which is already implemented above) to get these weights from the attention scores. Pass the attention scores vector to the softmax function to get the weights. Mathematically,\n",
+    "\n",
+    "$$\n",
+    "\\large \\alpha_{ij} = \\frac{\\exp{\\left(e_{ij}\\right)}}{\\sum_{k=1}^K \\exp{\\left(e_{ik}\\right)}}\n",
+    "$$\n",
+    "\n",
+    "\n",
+    "\n",
+    "## 3: Weight the encoder output vectors and sum\n",
+    "\n",
+    "The weights tell you the importance of each input word with respect to the decoder state. In this step, you use the weights to modulate the magnitude of the encoder vectors. Words with little importance will be scaled down relative to important words. Multiply each encoder vector by its respective weight to get the alignment vectors, then sum up the weighted alignment vectors to get the context vector. Mathematically,\n",
+    "\n",
+    "$$\n",
+    "\\large c_i = \\sum_{j=1}^K\\alpha_{ij} h_{j}\n",
+    "$$\n",
+    "\n",
+    "Implement these steps in the `attention` function below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4546cbb5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Implement this function. Replace None with your code.\n",
+    "def attention(encoder_states, decoder_state):\n",
+    "    \"\"\" Example function that calculates attention, returns the context vector \n",
+    "    \n",
+    "        Arguments:\n",
+    "        encoder_vectors: NxM numpy array, where N is the number of vectors and M is the vector length\n",
+    "        decoder_vector: 1xM numpy array, M is the vector length, much be the same M as encoder_vectors\n",
+    "    \"\"\" \n",
+    "    \n",
+    "    # First, calculate the alignment scores\n",
+    "    scores = None\n",
+    "    \n",
+    "    # Then take the softmax of the alignment scores to get a weight distribution\n",
+    "    weights = None\n",
+    "    \n",
+    "    # Multiply each encoder state by its respective weight\n",
+    "    weighted_scores = None\n",
+    "    \n",
+    "    # Sum up weighted alignment vectors to get the context vector and return it\n",
+    "    context = None\n",
+    "    return context\n",
+    "\n",
+    "context_vector = attention(encoder_states, decoder_state)\n",
+    "print(context_vector)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d9f3df4",
+   "metadata": {},
+   "source": [
+    "If you implemented the `attention` function correctly, the context vector should be\n",
+    "\n",
+    "```python\n",
+    "[-0.63514569  0.04917298 -0.43930867 -0.9268003   1.01903919 -0.43181409\n",
+    "  0.13365099 -0.84746874 -0.37572203  0.18279832 -0.90452701  0.17872958\n",
+    " -0.58015282 -0.58294027 -0.75457577  1.32985756]\n",
+    "```\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4210899c",
+   "metadata": {},
+   "source": [
+    "## See below for solutions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3ba0d629",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "# Solution\n",
+    "def alignment(encoder_states, decoder_state):\n",
+    "    # First, concatenate the encoder states and the decoder state.\n",
+    "    inputs = np.concatenate((encoder_states, decoder_state.repeat(input_length, axis=0)), axis=1)\n",
+    "    assert inputs.shape == (input_length, 2*hidden_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the concatenated inputs and the first layer, with tanh activation\n",
+    "    activations = np.tanh(np.matmul(inputs, layer_1))\n",
+    "    assert activations.shape == (input_length, attention_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the activations with the second layer. Remember that you don't need tanh here\n",
+    "    scores = np.matmul(activations, layer_2)\n",
+    "    assert scores.shape == (input_length, 1)\n",
+    "    \n",
+    "    return scores\n",
+    "\n",
+    "# Run this to test your alignment function\n",
+    "scores = alignment(encoder_states, decoder_state)\n",
+    "print(scores)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f80faecb",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "# Solution\n",
+    "def attention(encoder_states, decoder_state):\n",
+    "    \"\"\" Example function that calculates attention, returns the context vector \n",
+    "    \n",
+    "        Arguments:\n",
+    "        encoder_vectors: NxM numpy array, where N is the number of vectors and M is the vector length\n",
+    "        decoder_vector: 1xM numpy array, M is the vector length, much be the same M as encoder_vectors\n",
+    "    \"\"\" \n",
+    "    \n",
+    "    # First, calculate the dot product of each encoder vector with the decoder vector\n",
+    "    scores = alignment(encoder_states, decoder_state)\n",
+    "    \n",
+    "    # Then take the softmax of those scores to get a weight distribution\n",
+    "    weights = softmax(scores)\n",
+    "    \n",
+    "    # Multiply each encoder state by its respective weight\n",
+    "    weighted_scores = encoder_states * weights\n",
+    "    \n",
+    "    # Sum up the weights encoder states\n",
+    "    context = np.sum(weighted_scores, axis=0)\n",
+    "    \n",
+    "    return context\n",
+    "\n",
+    "context_vector = attention(encoder_states, decoder_state)\n",
+    "print(context_vector)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16a6caa8",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/.ipynb_checkpoints/C4W1_Bleu_Score-checkpoint.ipynb

@@ -0,0 +1,585 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Calculating the Bilingual Evaluation Understudy (BLEU) score: Ungraded Lab"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this ungraded lab, you will implement a popular metric for evaluating the quality of machine-translated text: the BLEU score proposed by Kishore Papineni, et al. in their 2002 paper [\"BLEU: a Method for Automatic Evaluation of Machine Translation\"](https://www.aclweb.org/anthology/P02-1040.pdf). The BLEU score works by comparing a \"candidate\" text to one or more \"reference\" texts. The score is higher the better the result. In the following sections you will calculate this value using your own implementation as well as using functions from a library."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 1.  Importing the Libraries\n",
+    "\n",
+    "You will start by importing the Python libraries. First, you will implement your own version of the BLEU Score using NumPy. To verify that your implementation is correct, you will compare the results with those generated by the [SacreBLEU library](https://github.com/mjpost/sacrebleu). This package provides hassle-free computation of shareable, comparable, and reproducible BLEU scores. It also knows all the standard test sets and handles downloading, processing, and tokenization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
+      "[nltk_data]   Package punkt is already up-to-date!\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Requirement already satisfied: sacrebleu in /opt/conda/lib/python3.10/site-packages (2.3.1)\n",
+      "Requirement already satisfied: portalocker in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (2.8.2)\n",
+      "Requirement already satisfied: regex in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (2023.10.3)\n",
+      "Requirement already satisfied: tabulate>=0.8.9 in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (0.9.0)\n",
+      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (1.24.3)\n",
+      "Requirement already satisfied: colorama in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (0.4.6)\n",
+      "Requirement already satisfied: lxml in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (4.9.3)\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np                  # import numpy to make numerical computations.\n",
+    "import nltk                         # import NLTK to handle simple NL tasks like tokenization.\n",
+    "nltk.download(\"punkt\")\n",
+    "from nltk.util import ngrams\n",
+    "from collections import Counter     # import a counter.\n",
+    "!pip3 install 'sacrebleu'           # install the sacrebleu package.\n",
+    "import sacrebleu                    # import sacrebleu in order compute the BLEU score.\n",
+    "import matplotlib.pyplot as plt     # import pyplot in order to make some illustrations."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 2.  BLEU score\n",
+    "\n",
+    "## 2.1 Definitions and formulas\n",
+    "\n",
+    "You have seen how to calculate the BLEU score in this week's lectures. Formally, you can express the BLEU score as:\n",
+    "\n",
+    "$$BLEU = BP\\times\\Bigl(\\prod_{i=1}^{n}precision_i\\Bigr)^{(1/n)}.\\tag{1}$$\n",
+    "\n",
+    "\n",
+    "The BLEU score depends on the $BP$, which stands for Brevity Penalty, and the weighted geometric mean precision for different lengths of n-grams, both of which are described below. The product runs from $i=1$ to $i=n$ to account for 1-grams to n-grams and the exponent of $1/n$ is there to calculate the geometrical average. In this notebook, you will use $n=4$\n",
+    "\n",
+    "The **Brevity Penalty** is defined as an exponential decay:\n",
+    "\n",
+    "$$BP = min\\Bigl(1, e^{(1-({len(ref)}/{len(cand)}))}\\Bigr),\\tag{2}$$\n",
+    "\n",
+    "where ${len(ref)}$ and ${len(cand)}$ refer to the length or count of words in the reference and candidate translations. The brevity penalty helps to handle very short translations. \n",
+    "\n",
+    "The **precision** is defined as :\n",
+    "\n",
+    "$$precision_i = \\frac {\\sum_{s_i \\in{cand}}min\\Bigl(C(s_i, cand), C(s_i, ref)\\Bigr)}{\\sum_{s_i \\in{cand}} C(s_i, cand)}.\\tag{3}$$\n",
+    "\n",
+    "The sum goes over all the i-grams $s_i$ in the candidate sentence $cand$. $C(s_i, cand)$ and $C(s_i, ref)$ are the counts of the i-grams in the candidate and reference sentences respectively. So the sum counts all the n-grams in the candidate sentence that also appear in the reference sentence, but only counts them as many times as they appear in the reference sentence and not more. This is then divided by the total number of i-grams in the candidate sentence."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2.2 Visualizing the BLEU score\n",
+    "\n",
+    "### Brevity Penalty:\n",
+    "The brevity penalty penalizes generated translations that are shorter than the reference sentence. It compensates for the fact that the BLEU score has no recall term."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABcvElEQVR4nO3deVwU9eMG8Gd2WW4BEUEEBLxRPBCVwEytxLRMy1LTvLPwzLM0y6vMX5rmrZlXlleKV2UmWZ5oCoIniiIKKoqgcsq1+/n9Ye7XlcNdBAaW5/167eslw8zOswPMPs5+ZkYSQggQERERGQmF3AGIiIiIShLLDRERERkVlhsiIiIyKiw3REREZFRYboiIiMiosNwQERGRUWG5ISIiIqNiIneAsqbRaHDr1i1UqVIFkiTJHYeIiIj0IIRAWloaatasCYWi6GMzla7c3Lp1C25ubnLHICIiomKIj4+Hq6trkfNUunJTpUoVAI82jo2NjcxpiIiISB+pqalwc3PTvo8XpdKVm8cfRdnY2LDcEBERVTD6DCnhgGIiIiIyKiw3REREZFRYboiIiMiosNwQERGRUWG5ISIiIqPCckNERERGheWGiIiIjArLDRERERkVlhsiIiIyKiw3REREZFRkLTeHDh1C165dUbNmTUiShJ07dz5zmYMHD8LX1xfm5uaoXbs2VqxYUfpBiYiIqMKQtdxkZGSgWbNmWLJkiV7zx8bGokuXLmjbti0iIiLw2WefYfTo0QgODi7lpERERFRRyHrjzM6dO6Nz5856z79ixQrUqlULCxYsAAB4eXkhLCwM3377LXr06FFKKfWj1ggkpDyUNQM9Hycbc6iU/KSWiKiiq1B3BT927BgCAwN1pnXq1AmrV69Gbm4uVCpVvmWys7ORnZ2t/To1NbVUsiVnZOPFb/4pleemslHDxhxL+/rA191e7ihERPQcKlS5uX37NpycnHSmOTk5IS8vD0lJSXB2ds63zOzZszFjxowyyWdmwv/1V1RqjcDt1Cz0XnkcU99ohPdfcIckSXLHIiKiYqhQ5QZAvjccIUSB0x+bPHkyxo0bp/06NTUVbm5uJZ7LsYo5Ln2l/0dsVL5kZOfhk21n8PvZBHyx6zwi4h9g9ttNYGailDsaEREZqEIdaqhRowZu376tMy0xMREmJiaoVq1agcuYmZnBxsZG50H0NCszEyzp44MpXbygkIDtp27imz8uyR2LiIiKoUKVG39/f4SEhOhM27dvH1q2bFngeBsiQ0iShKEv1cby930BAGtDYxERd1/mVEREZChZy016ejoiIyMRGRkJ4NGp3pGRkYiLiwPw6COl/v37a+cPCgrC9evXMW7cOERFRWHNmjVYvXo1JkyYIEd8MlKdGtfAWz4uEAKYFHwWOXkauSMREZEBZC03YWFh8PHxgY+PDwBg3Lhx8PHxwdSpUwEACQkJ2qIDAJ6entizZw8OHDiA5s2b48svv8SiRYtkPw2cjM8XbzSCvZUpLt1Jw/cHY+SOQ0REBpDE4xG5lURqaipsbW2RkpLC8TdUpF2RN/Hx5kiYKhXY8/GLqOtYRe5IRESVliHv3xVqzA1RWXqzWU10aFAdOWoNJgWfhUZTqf4fQERUYbHcEBVCkiR89VYTWJkqEXb9Pn7+97rckYiISA8sN0RFcLGzwKTODQEA3/xxETfuZ8qciIiInoXlhugZ+vq5o5VHVWTkqPHZjnOoZMPUiIgqHJYbomdQKCT8X4+mMDVR4FD0XWw/dVPuSEREVASWGyI91KlujbGv1gcAzPztAhLTsmROREREhWG5IdLT0Lae8HaxQcrDXEzbdV7uOEREVAiWGyI9mSgVmNOjGUwUEv44dxt7zibIHYmIiArAckNkgEY1bTC8fR0AwNRd53AvI0fmRERE9DSWGyIDjXi5Luo7WSMpPQczf+XHU0RE5Q3LDZGBzEyUmPNOMygkYGfkLeyPuiN3JCIiegLLDVExNHezw9C2tQEAn+04i5SHuTInIiKix1huiIppbMf68HSwwp3UbMz6/YLccYiI6D8sN0TFZK5SYs47TSFJwC9hN3DgUqLckYiICCw3RM+llYc9BgV4AgAmbz+L1Cx+PEVEJDeWG6LnNLFTA7hXs0RCSha+/j1K7jhERJUeyw3Rc7IwVWLuO80gScDmk/E4FH1X7khERJUayw1RCWjtaY8B/h4AgEnBZ/jxFBGRjFhuiErIJ689+njqVkoWZv3Gj6eIiOTCckNUQixNTbQfT20Ji8c/PHuKiEgWLDdEJai15//OnpoUfAYpmfx4ioiorLHcEJWwiZ0aaC/uN/M3XtyPiKissdwQlTALUyW+fffRxf2CT93AXxd47ykiorLEckNUCnzd7bX3npq84yzuZ+TInIiIqPJguSEqJeM61kddR2vcTcvG1N3n5Y5DRFRpsNwQlRJzlRLz3m0GpULCr6dv4fczCXJHIiKqFFhuiEpRMzc7DG9fBwDw+c6zuJuWLXMiIiLjx3JDVMpGvVwPXs42uJ+Ziyk7zkIIIXckIiKjxnJDVMpMTRSY924zqJQS9l24g+2nbsodiYjIqLHcEJWBRjVtMObV+gCA6bvP49aDhzInIiIyXiw3RGXko5dqw6eWHdKy8zBx22loNPx4ioioNLDcEJURE6UC83s2h7lKgaNXkrH+2DW5IxERGSWWG6Iy5Olghc+6eAEAZv9xETF302VORERkfFhuiMrY+37uaFvPAdl5Goz75TTy1Bq5IxERGRWWG6IyplBImPNOU1QxN8Hp+AdY+k+M3JGIiIwKyw2RDJxtLfBlN28AwKK/L+N0/AN5AxERGRGWGyKZdGteE683dYZaIzD2l0g8zFHLHYmIyCiw3BDJRJIkzOruDccqZrh6NwPf7L0odyQiIqPAckMkIztLU8x9txkAYF3oNRyKvitzIiKiio/lhkhm7epXR39/dwDAxG2n8SAzR+ZEREQVG8sNUTkwubMXajtY4U5qNqbsOMebaxIRPQeWG6JywMJUiQW9m8NEIeH3swnYEcGbaxIRFZfs5WbZsmXw9PSEubk5fH19cfjw4SLnX7p0Kby8vGBhYYEGDRpg/fr1ZZSUqHQ1dbXDmFfrAQCm7jqP+HuZMiciIqqYZC03W7ZswZgxYzBlyhRERESgbdu26Ny5M+Li4gqcf/ny5Zg8eTKmT5+O8+fPY8aMGRgxYgR+/fXXMk5OVDqGta+Llu5VkZ6dh/G/nIaaN9ckIjKYJGT8cN/Pzw8tWrTA8uXLtdO8vLzQvXt3zJ49O9/8AQEBaNOmDebOnaudNmbMGISFheHIkSN6rTM1NRW2trZISUmBjY3N878IohIWfy8TnRceRnp2Hj55rQGGt68rdyQiItkZ8v4t25GbnJwchIeHIzAwUGd6YGAgQkNDC1wmOzsb5ubmOtMsLCxw4sQJ5ObmFrpMamqqzoOoPHOzt8T0NxsDAObvi8bZGykyJyIiqlhkKzdJSUlQq9VwcnLSme7k5ITbt28XuEynTp2watUqhIeHQwiBsLAwrFmzBrm5uUhKSipwmdmzZ8PW1lb7cHNzK/HXQlTSerRwQZcmNZCnEfh4SwQyc/LkjkREVGHIPqBYkiSdr4UQ+aY99sUXX6Bz58544YUXoFKp0K1bNwwcOBAAoFQqC1xm8uTJSElJ0T7i4+NLND9RaZAkCV+/1QQ1bMxx9W4Gvvo9Su5IREQVhmzlxsHBAUqlMt9RmsTExHxHcx6zsLDAmjVrkJmZiWvXriEuLg4eHh6oUqUKHBwcClzGzMwMNjY2Og+iisDO0hTzezaDJAEb/43DvvMFH9EkIiJdspUbU1NT+Pr6IiQkRGd6SEgIAgICilxWpVLB1dUVSqUSmzdvxhtvvAGFQvaDUEQlLqCuAz5sWxsA8GnwGdxJzZI5ERFR+SdrIxg3bhxWrVqFNWvWICoqCmPHjkVcXByCgoIAPPpIqX///tr5o6Oj8fPPP+Py5cs4ceIEevfujXPnzuHrr7+W6yUQlbrxgQ3QuKYN7mfmYsLW09Dw9HAioiKZyLnyXr16ITk5GTNnzkRCQgK8vb2xZ88euLs/us9OQkKCzjVv1Go15s2bh0uXLkGlUqFDhw4IDQ2Fh4eHTK+AqPSZmiiwsLcP3lh8GIcvJ2HN0Vh88N/RHCIiyk/W69zIgde5oYpqw7/XMWXHOaiUEnYMbwNvF1u5IxERlZkKcZ0bIjJMn9a1ENjICblqgdGbeXo4EVFhWG6IKghJkvBNj6ZwsjHD1bsZmPnrBbkjERGVSyw3RBVIVStTfNerOSQJ2HwyHnvOJsgdiYio3GG5IapgAuo4YFi7OgCAScFncPPBQ5kTERGVLyw3RBXQ2I710czNDqlZeRi7OZJ3DyciegLLDVEFpFIqsKh3c1ibmeDEtXtY/PdluSMREZUbLDdEFZR7NSt82f3R3cMX7b+Mf68my5yIiKh8YLkhqsDe8nHF2y1coBHAmC2RuJ+RI3ckIiLZsdwQVXBfdvOGp4MVElKy8EnwGVSy63ISEeXDckNUwVmZmWDxez5QKSWEXLiDn45flzsSEZGsWG6IjIC3iy0mdfYCAHz1exQu3EqVORERkXxYboiMxOA2HniloSNy8jQYuekUMrJ5ewYiqpxYboiMhCRJmPtuM9SwMcfVuxmYuuu83JGIiGTBckNkROytTLGwd3MoJCD41A0Eh9+QOxIRUZljuSEyMn61q2HMq/UBAF/sOoeYu+kyJyIiKlssN0RGaESHugioUw2ZOWqM2HAKWblquSMREZUZlhsiI6RUSFjQqzmqWZni4u00zPo9Su5IRERlhuWGyEg52phjfq/mAICfjl/H72cS5A1ERFRGWG6IjFi7+tUxrH0dAMCnwWdwLSlD5kRERKWP5YbIyI3vWB8t3asiPTsPIzZy/A0RGT+WGyIjZ6JUYHEfH1S1VOH8rVR8vYfjb4jIuLHcEFUCzrYW2vE3649x/A0RGTeWG6JKokMDRwzn+BsiqgRYbogqkXEd66OVx6PxN8N5/RsiMlIsN0SViIlSgcXvtYC9lSkuJKRi5m8X5I5ERFTiWG6IKpkatuZY0Ks5JAnY+G8cdkbclDsSEVGJYrkhqoReql8do16uBwCYvP0sLt9JkzkREVHJYbkhqqQ+fqUeXqzrgIe5agzfcAqZOXlyRyIiKhEsN0SVlFIhYUHv5nCsYobLiemYsuMchBByxyIiem4sN0SVmIO1GZb0aQGlQsKOiJvYdCJe7khERM+N5YaokmvtaY+JnRoAAKbvPo8zNx7IG4iI6Dmx3BARPnqpNjo2ckKOWoNhP5/Cg8wcuSMRERUbyw0RQZIkfPtuM7hXs8TNBw8xdkskNBqOvyGiionlhogAALYWKizv6wszEwX+uXQXS/+5InckIqJiYbkhIq1GNW3wVXdvAMD8v6Jx+PJdmRMRERmO5YaIdLzb0g29W7lBCODjzZG49eCh3JGIiAzCckNE+Ux/szG8XWxwLyMHwzacQnYeb7BJRBUHyw0R5WOuUmJ5X1/YWqhwOv4BZv7KG2wSUcXBckNEBXKzt8TC3o9usLnh3zhsDeMF/oioYmC5IaJCtW/giLGv1gcATNl5DudupsiciIjo2VhuiKhIIzvUxcsNHZGTp0HQz+G4n8EL/BFR+cZyQ0RFUigkfNezOWrZW+LG/Yf4eEsk1LzAHxGVYyw3RPRMtpYqrHjfF+YqBQ5F38WCv6LljkREVCjZy82yZcvg6ekJc3Nz+Pr64vDhw0XOv2HDBjRr1gyWlpZwdnbGoEGDkJycXEZpiSqvRjVt8H9vNwUALP77Cv48f1vmREREBZO13GzZsgVjxozBlClTEBERgbZt26Jz586Ii4srcP4jR46gf//+GDJkCM6fP4+tW7fi5MmT+OCDD8o4OVHl1N3HBYPaeAAAxv9yGlcS0+UNRERUAFnLzfz58zFkyBB88MEH8PLywoIFC+Dm5obly5cXOP/x48fh4eGB0aNHw9PTEy+++CI++ugjhIWFFbqO7OxspKam6jyIqPg+6+IFP097pGfn4cOfwpCWlSt3JCIiHbKVm5ycHISHhyMwMFBnemBgIEJDQwtcJiAgADdu3MCePXsghMCdO3ewbds2vP7664WuZ/bs2bC1tdU+3NzcSvR1EFU2KqUCS/u2gLOtOa7ezcC4X07zDuJEVK7IVm6SkpKgVqvh5OSkM93JyQm3bxf8WX5AQAA2bNiAXr16wdTUFDVq1ICdnR0WL15c6HomT56MlJQU7SM+nhciI3peDtZmWP6+L0yVCoRcuMM7iBNRuSL7gGJJknS+FkLkm/bYhQsXMHr0aEydOhXh4eHYu3cvYmNjERQUVOjzm5mZwcbGRudBRM+vuZudzh3E/754R+ZERESPyFZuHBwcoFQq8x2lSUxMzHc057HZs2ejTZs2mDhxIpo2bYpOnTph2bJlWLNmDRISEsoiNhE9oWcrN/T1q/XoDuKbIhFzlwOMiUh+spUbU1NT+Pr6IiQkRGd6SEgIAgICClwmMzMTCoVuZKVSCeDRER8iKnvTujZGK4+qSMvOw9D1YUjlAGMikpmsH0uNGzcOq1atwpo1axAVFYWxY8ciLi5O+zHT5MmT0b9/f+38Xbt2xfbt27F8+XJcvXoVR48exejRo9G6dWvUrFlTrpdBVKmZmiiwrK8vatj8N8B4SyQHGBORrEzkXHmvXr2QnJyMmTNnIiEhAd7e3tizZw/c3d0BAAkJCTrXvBk4cCDS0tKwZMkSjB8/HnZ2dnj55ZfxzTffyPUSiAhA9SpmWNnfF++sOIa/ohKx4K9ojAtsIHcsIqqkJFHJPs9JTU2Fra0tUlJSOLiYqIQFh9/A+K2nAQAr3m+B17ydZU5ERMbCkPdv2c+WIiLj0cPXFYPbeAIAxv1yGpdup8mciIgqI5YbIipRn3VpiIA61ZCZo8bQ9WG4n5EjdyQiqmRYboioRJkoFVjapwXc7C0Qdy8TwzecQq5aI3csIqpEDC437du3x/r16/Hw4cPSyENERqCqlSlW9W8FK1Mljl1Nxle/XZA7EhFVIgaXG19fX3zyySeoUaMGhg4diuPHj5dGLiKq4BrUqILvejUHAPx47Do2/htX9AJERCXE4HIzb9483Lx5E+vXr8fdu3fx0ksvoVGjRvj2229x5w4vv05E/xPYuAYmBNYHAEzddQ4nYu/JnIiIKoNijblRKpXo1q0bdu7ciZs3b6JPnz744osv4Obmhu7du+Pvv/8u6ZxEVEGN6FAXbzR1Rp5GYNjP4bhxP1PuSERk5J5rQPGJEycwdepUfPvtt3B0dMTkyZPh6OiIrl27YsKECSWVkYgqMEmSMPedZmhc0wbJGTkYuj4cGdl5csciIiNmcLlJTEzEvHnz4O3tjbZt2+Lu3bvYvHkzrl27hhkzZmDlypXYtWsXVqxYURp5iagCsjBV4of+LeFgbYaohFSM5S0aiKgUGVxuXF1dsWrVKgwYMAA3btzAtm3b8Nprr0GSJO08rVu3RqtWrUo0KBFVbDXtLPB9P1+YKhXYd+EO5oVckjsSERkpg2+/cPjwYbRt27a08pQ63n6BSF47Im5g7JZHt2j4rlczvOXjKnMiIqoISvX2C9OmTcODBw8KXOnLL79s6NMRUSXzlo8rhrWvAwD4NPgsTsXdlzkRERkbg8vNwYMHkZOT/3LqWVlZOHz4cImEIiLjNjGwATo2ckJOngYfrg/HzQe8KCgRlRwTfWc8c+YMAEAIgQsXLuD27dva76nVauzduxcuLi4ln5CIjI5CIWFBr+bosTwUF2+n4YMfw7AtyB9WZnrvkoiICqX3mBuFQqEdNFzQIhYWFli8eDEGDx5csglLGMfcEJUfN+5novvSo0hKz0HHRk74/n1fKBTSsxckokrHkPdvvcvN9evXIYRA7dq1ceLECVSvXl37PVNTUzg6OkKpVD5f8jLAckNUvoRfv4f3fvgXOXkafPRSbUzu4iV3JCIqhwx5/9b7GLC7uzsAQKPh3X2JqOT4uttj7jtN8fHmSHx/6Co8HazQu3UtuWMRUQWmV7nZvXu33k/45ptvFjsMEVVO3Zq7IOZuBhbtv4zPd55DLXtLBNR1kDsWEVVQen0spVDod1KVJElQq9XPHao08WMpovJJCIHRmyPx6+lbsDE3wY4RbVCnurXcsYionCjx69xoNBq9HuW92BBR+fXoHlRN4VPLDqlZeRiy7iTuZ+S/7AQR0bM8140ziYhKkrlKiZX9WsLFzgLXkjPx0c/hyMnjOD8iMozBt18AgIyMDBw8eBBxcXH5Lug3evToEgtXGvixFFH5d+l2GnosD0V6dh7e9nHBvJ7NdO5fR0SVT6mcCv5YREQEunTpgszMTGRkZMDe3h5JSUmwtLSEo6Mjrl69+lzhSxvLDVHFcDD6LgavOwm1RmDsq/Xx8av15I5ERDIq1XtLjR07Fl27dsW9e/dgYWGB48eP4/r16/D19cW3335b7NBERE9qV786vuzmDQD47q9o7Ii4IXMiIqooDC43kZGRGD9+PJRKJZRKJbKzs+Hm5oY5c+bgs88+K42MRFRJ9fGrhY9eqg0A+HTbWfx7NVnmRERUERhcblQqlfazbycnJ8TFxQEAbG1ttf8mIiopn77WEJ29ayBHrcGHP4Uj5m663JGIqJwzuNz4+PggLCwMANChQwdMnToVGzZswJgxY9CkSZMSD0hElZtCIeG7Xs3R3M0OKQ9zMXjdSdzjKeJEVASDy83XX38NZ2dnAMCXX36JatWqYdiwYUhMTMTKlStLPCARkblKiVUDWsK1qgWuJ2di6PowZOXyulpEVLBinQpekfFsKaKK60piGt5eForUrDx0aVIDS95rwbuIE1USpXq2FBGRXOo6VsHK/i1hqlRgz9nb+HpPlNyRiKgcMrjc3LlzB/369UPNmjVhYmKiPWvq8YOIqDS9ULsa5r7bFACw6kgs1h6NlTkREZU3et0V/EkDBw5EXFwcvvjiCzg7O/OqoURU5ro1d8HNBw8xZ+8lzPztApxtLfCadw25YxFROWHwmJsqVarg8OHDaN68eSlFKl0cc0NkHIQQmLLzHDb+GwczEwU2ffgCWtSqKncsIiolpTrmxs3NDZVsDDIRlUOSJGHmm43xckNHZOdp8MGPYbiWlCF3LCIqBwwuNwsWLMCkSZNw7dq1UohDRKQ/E6UCi9/zQRMXW9zLyMGAtSeQlJ4tdywikpnBH0tVrVoVmZmZyMvLg6WlJVQqlc737927V6IBSxo/liIyPolpWXh7WShu3H+IZq622Dj0BViZGTykkIjKMUPevw3+61+wYEFxcxERlQrHKub4cXBr9FgeitM3UjBi4yn80L8lVEpe7YKoMuJF/IjIaIRfv4++q44jK1eDni1d8U2Ppjyjk8hIlPpF/GJiYvD555/jvffeQ2JiIgBg7969OH/+fHGejoioRPi6V8Xi91pAIQG/hN3Ad39dljsSEcnA4HJz8OBBNGnSBP/++y+2b9+O9PRHd+g9c+YMpk2bVuIBiYgM0bGRE77q/ugmvov2X8aGf6/LnIiIyprB5WbSpEn46quvEBISAlNTU+30Dh064NixYyUajoioOPr41cLoV+oBAL7YeQ4hF+7InIiIypLB5ebs2bN466238k2vXr06kpOTSyQUEdHzGvtqPfRq6QaNAEZuPIWwa+X7TE4iKjkGlxs7OzskJCTkmx4REQEXFxeDAyxbtgyenp4wNzeHr68vDh8+XOi8AwcOhCRJ+R6NGzc2eL1EZNwkScKst7y1F/kb8mMYou+kyR2LiMqAweWmT58++PTTT3H79m1IkgSNRoOjR49iwoQJ6N+/v0HPtWXLFowZMwZTpkxBREQE2rZti86dOyMuLq7A+RcuXIiEhATtIz4+Hvb29nj33XcNfRlEVAmYKBVY2qcFWtSyQ8rDXPRffQI3HzyUOxYRlTKDTwXPzc3FwIEDsXnzZgghYGJiArVajT59+mDdunUG3Rncz88PLVq0wPLly7XTvLy80L17d8yePfuZy+/cuRNvv/02YmNj4e7uXuA82dnZyM7+3xVLU1NT4ebmxlPBiSqRB5k5eHfFMVxOTEed6lbYGhQAeyvTZy9IROWGIaeCF/s6N1evXsWpU6eg0Wjg4+ODevXqGbR8Tk4OLC0tsXXrVp0xPB9//DEiIyNx8ODBZz5H165dkZ2djX379hU6z/Tp0zFjxox801luiCqXhJSH6LEsFLdSstDczQ4bh/rB0pRXMSaqKErlOjcajQZz585FmzZt0Lp1a6xatQpvvPEGevbsaXCxAYCkpCSo1Wo4OTnpTHdycsLt27efuXxCQgL++OMPfPDBB0XON3nyZKSkpGgf8fHxBmcloorP2dYC64e0hp2lCpHxDzB8wynkqjVyxyKiUqB3ufnmm28wadIkWFlZwdnZGfPnz8fo0aOfO8DTVw8VQuh1RdF169bBzs4O3bt3L3I+MzMz2NjY6DyIqHKq61gFawa2grlKgQOX7uKTbWeg0VSqi7QTVQp6l5t169Zh8eLF2LdvH3bt2oWdO3di/fr1KO7dGxwcHKBUKvMdpUlMTMx3NOdpQgisWbMG/fr107nWDhHRs7SoVRXL+/pCqZCwI+Imvvo9qtj7MSIqn/QuN9evX8cbb7yh/bpTp04QQuDWrVvFWrGpqSl8fX0REhKiMz0kJAQBAQFFLnvw4EFcuXIFQ4YMKda6iahy69DQEXPfaQoAWHM0Fkv+viJzIiIqSXqXm5ycHFhYWGi/liQJpqamOmciGWrcuHFYtWoV1qxZg6ioKIwdOxZxcXEICgoC8Gi8TEGnl69evRp+fn7w9vYu9rqJqHJ7u4Urpr7RCAAwLyQaPx3nbRqIjIVBpwp88cUXsLS01H6dk5ODWbNmwdbWVjtt/vz5ej9fr169kJycjJkzZyIhIQHe3t7Ys2eP9rTuhISEfNe8SUlJQXBwMBYuXGhIdCKifAa/6IkHmTlY9PcVTN11DjbmJujW3PCLkRJR+aL3qeDt27d/5kBfSZLw999/l0iw0mLIqWREZPyEEJi2+zzWH7sOE4WEHwa0RIcGjnLHIqKnlMl1bioqlhsieppGIzBmSyR2n74Fc5UCPw/xQ0sPe7ljEdETSuU6N0RExkqhkDCvZzO0b1AdWbkaDFp3Ehdupcodi4iKieWGiAiASqnA8r6+aOleFWlZeei/5gRikzLkjkVExcByQ0T0HwtTJVYPbAUvZxskpWfj/VX/8kabRBUQyw0R0RNsLVT4aUhr1K5uhZsPHqLfqn9xN634l7wgorLHckNE9BQHazP8PMQPLnYWuJqUgX6r/0VKZq7csYhITwaXGw8PD8ycOTPf9WeIiIxJTTsLbPjAD9WrmOHi7TQMXHcCGdl5csciIj0YXG7Gjx+PXbt2oXbt2ujYsSM2b978XFcpJiIqrzwcrPDTkNawtVAhIu4Bhq4PQ1auWu5YRPQMBpebUaNGITw8HOHh4WjUqBFGjx4NZ2dnjBw5EqdOnSqNjEREsmlYwwY/Dm4NK1MlQmOSMXLjKeSqNXLHIqIiFHvMTbNmzbBw4ULcvHkT06ZNw6pVq9CqVSs0a9YMa9as4V12ichoNHezw6oBrWBmosBfUYkY98tpqDXcxxGVV8UuN7m5ufjll1/w5ptvYvz48WjZsiVWrVqFnj17YsqUKejbt29J5iQikpV/nWpY8b4vVEoJv56+hU+Dz0DDgkNULhl040wAOHXqFNauXYtNmzZBqVSiX79++O6779CwYUPtPIGBgXjppZdKNCgRkdw6NHTEot4+GLkpAtvCb8BcpcCX3byfed89IipbBpebVq1aoWPHjli+fDm6d+8OlUqVb55GjRqhd+/eJRKQiKg86dzEGfPVGozZEomfj8fBzESJz1/3YsEhKkcMLjdXr16Fu7t7kfNYWVlh7dq1xQ5FRFSedWvuguxcDT4JPoPVR2JhoVJiQqcGcsciov8YPOamQ4cOSE5Ozjf9wYMHqF27domEIiIq73q2csPMbo0BAEv+uYIlf1+WORERPWZwubl27RrU6vzXecjOzsbNmzdLJBQRUUXQ398Dn3V5NN7w233RWHX4qsyJiAgw4GOp3bt3a//9559/wtbWVvu1Wq3G/v374eHhUaLhiIjKuw9fqoOsXA3mh0Tjq9+jYGqiQH9/D7ljEVVqepeb7t27AwAkScKAAQN0vqdSqeDh4YF58+aVaDgioopg1Mt1kZWrxrIDMZi66zyUCgl9/Yoem0hEpUfvcqPRPLoip6enJ06ePAkHB4dSC0VEVJFIkoSJnRogTyOw8tBVTNlxDiYKCb1a1ZI7GlGlZPDZUrGxsaWRg4ioQpMkCZM7N0SeWmDN0VhM2n4WSoUC7/i6yh2NqNLRq9wsWrQIH374IczNzbFo0aIi5x09enSJBCMiqmgkScIXb3hBrdHgx2PXMXHbaSgVwFs+LDhEZUkSetwEytPTE2FhYahWrRo8PT0LfzJJwtWr5ftsgdTUVNja2iIlJQU2NjZyxyEiIySEwOc7z2HDv3FQSMCC3j54s1lNuWMRVWiGvH/rdeTmyY+i+LEUEVHRJEnCl928odYIbD4Zj7FbIqGUJLze1FnuaESVgsHXuTl48GBp5CAiMioKhYSv32qCd3xdodYIfLw5An+cTZA7FlGlYHC56dixI2rVqoVJkybh7NmzpZGJiMgoKBQSvunRFG/7uCBPIzByEwsOUVkwuNzcunULn3zyCQ4fPoxmzZqhadOmmDNnDm7cuFEa+YiIKjSlQsLcd5vhbR8XqP8rOHtYcIhKlV4DigsTGxuLjRs3YtOmTbh48SJeeukl/P333yWZr8RxQDERyUGtEZi49TS2R9yEUiFhUW8fjsEhMoAh79/PVW6AR7de+OOPP/DFF1/gzJkzBd53qjxhuSEiuag1AhO3ncb2U48KzsLezfFGU55FRaQPQ96/Df5Y6rGjR49i+PDhcHZ2Rp8+fdC4cWP89ttvxX06IiKjp1RImPtOM/Ro8XiQcSR+PX1L7lhERsfgKxR/9tln2LRpE27duoVXX30VCxYsQPfu3WFpaVka+YiIjIpSIWHOO00hScC28BsYsyUSANCV18EhKjEGl5sDBw5gwoQJ6NWrF+8vRURUDMr/zqKSAGwNv4GPN0dAIwS6NXeROxqRUTC43ISGhpZGDiKiSuVxwQEeFZyxWyKRqxa8FxVRCSjWmJuffvoJbdq0Qc2aNXH9+nUAwIIFC7Br164SDUdEZMweXwfnvdZu0Ahg4rbT2HwiTu5YRBWeweVm+fLlGDduHLp06YIHDx5oz46ys7PDggULSjofEZFRUygkzOreBP393SEEMGn7Wfx07JrcsYgqNIPLzeLFi/HDDz9gypQpUCqV2uktW7bkFYuJiIpBoZAw483GGPLioxsTf7HrPFYf4X38iIrL4HITGxsLHx+ffNPNzMyQkZFRIqGIiCobSZLw+eteGNa+DgDgy98uYPmBGJlTEVVMBpcbT09PREZG5pv+xx9/oFGjRiWRiYioUpIkCZ90aoCPX6kHAPhm70Us2n9Z5lREFY/BZ0tNnDgRI0aMQFZWFoQQOHHiBDZt2oTZs2dj1apVpZGRiKjSkCQJYzvWh6mJAnP/vIT5IdHIydNgfGB9SJIkdzyiCsHgcjNo0CDk5eXhk08+QWZmJvr06QMXFxcsXLgQvXv3Lo2MRESVzogOdaFSSvh6z0Us+ecKHuaq8fnrXiw4RHowqNzk5eVhw4YN6Nq1K4YOHYqkpCRoNBo4OjqWVj4iokrrw5fqwMxEiWm7Hw0wzsxR46vu3lAqWHCIimLQmBsTExMMGzYM2dnZAAAHBwcWGyKiUjQgwANz3mkKhQRsOhGH8b9EIk+tkTsWUblm8IBiPz8/RERElEYWIiIqQM+WbljY2wcmCgk7I29hxMZTyM5Tyx2LqNwyuNwMHz4c48ePx5IlS3Ds2DGcOXNG52GoZcuWwdPTE+bm5vD19cXhw4eLnD87OxtTpkyBu7s7zMzMUKdOHaxZs8bg9RIRVSRdm9XEivd9YWqiwJ/n7+DD9eF4mMOCQ1QQSQghDFlAocjfhyRJghACkiRpr1isjy1btqBfv35YtmwZ2rRpg++//x6rVq3ChQsXUKtWrQKX6datG+7cuYOvvvoKdevWRWJiIvLy8hAQEKDXOlNTU2Fra4uUlBTY2NjonZWIqDw4cjkJQ9eH4WGuGn6e9lg9sBWszQw+N4SowjHk/dvgcvP4XlKFcXd31/u5/Pz80KJFCyxfvlw7zcvLC927d8fs2bPzzb9371707t0bV69ehb29vf6hn8ByQ0QVXdi1exi09iTSsvPQ3M0OPw5qDVtLldyxiEqVIe/fBn8s5e7uXuRDXzk5OQgPD0dgYKDO9MDAwELvPL579260bNkSc+bMgYuLC+rXr48JEybg4cOHha4nOzsbqampOg8iooqspYc9Ng59AXaWKkTGP0CvlceQmJYldyyicsPgcpOcnKz9d3x8PKZOnYqJEyc+c6zM05KSkqBWq+Hk5KQz3cnJCbdv3y5wmatXr+LIkSM4d+4cduzYgQULFmDbtm0YMWJEoeuZPXs2bG1ttQ83NzeDchIRlUdNXG2x5UN/OFYxw8XbaXh3xTHE38uUOxZRuaB3uTl79iw8PDzg6OiIhg0bIjIyEq1atcJ3332HlStXokOHDti5c6fBAZ6+INXjsTsF0Wg0kCQJGzZsQOvWrdGlSxfMnz8f69atK/TozeTJk5GSkqJ9xMfHG5yRiKg8alCjCrYFBcDN3gLXkzPxzopQXL6TJncsItnpXW4++eQTNGnSBAcPHkT79u3xxhtvoEuXLkhJScH9+/fx0Ucf4f/+7//0XrGDgwOUSmW+ozSJiYn5juY85uzsDBcXF9ja2mqneXl5QQiBGzduFLiMmZkZbGxsdB5ERMaiVjVLbAsKQAOnKriTmo13vz+G0/EP5I5FJCu9y83Jkycxa9YsvPjii/j2229x69YtDB8+HAqFAgqFAqNGjcLFixf1XrGpqSl8fX0REhKiMz0kJKTQM5/atGmDW7duIT09XTstOjoaCoUCrq6ueq+biMiYONmYY8tHL6C5mx0eZOaizw/HERqTJHcsItnoXW7u3buHGjVqAACsra1hZWWlc8ZS1apVkZZm2OHQcePGYdWqVVizZg2ioqIwduxYxMXFISgoCMCjj5T69++vnb9Pnz6oVq0aBg0ahAsXLuDQoUOYOHEiBg8eDAsLC4PWTURkTOwsTbHhAz+0qVsNGTlqDFx7EvvOFzx+kcjYGTSg+OmxMM97A7devXphwYIFmDlzJpo3b45Dhw5hz5492rOuEhISEBcXp53f2toaISEhePDgAVq2bIm+ffuia9euWLRo0XPlICIyBlZmJlgzsBU6NXZCTp4GwzacQnB4wR/ZExkzva9zo1Ao0LlzZ5iZmQEAfv31V7z88suwsrIC8OiU67179xp0ET858Do3RGTs8tQaTNp+Ftv+KzZfvNEIQ170lDkV0fMplYv4DRo0SK+Vr127Vq/55MJyQ0SVgUYjMGtPFFYfiQUADGtfB590avDcR9yJ5FKqVyiu6FhuiKiyEEJg+cEYzNl7CQDQs6Urvn6rCUyUBl/ijEh2pXqFYiIiqhgkScLw9nXxTY8mUEjAL2E3EPTzKWTllu/hA0TPi+WGiMjI9WpVC9/3awkzEwX+irqDfqv/RUpmrtyxiEoNyw0RUSXQsZETfhrihyrmJjh57T56fn8Mt1N4PyoyTiw3RESVRGtPe2wNenQ/qkt30tBjeShi7qY/e0GiCoblhoioEmlYwwbBwwJQ28EKNx88xLsrjiGSt2sgI8NyQ0RUybjZW2JrkD+autriXkYO3lt5HH9fvCN3LKISw3JDRFQJVbM2w8ahL6BtPQc8zFVj6PpwbDkZ9+wFiSoAlhsiokrK+r/bNfRo4Qq1RuDT4LP4LiQalezyZ2SEWG6IiCoxlVKBb99tilEv1wUALNx/GZOCzyJPrZE5GVHxsdwQEVVykiRhfGADzHrLGwoJ2BIWj6Hrw5CZkyd3NKJiYbkhIiIAQF8/d3zfryXMVQr8c+kueq88jqT0bLljERmM5YaIiLQ6NnLCxqEvoKqlCmdupKDH8lBcS8qQOxaRQVhuiIhIR4taVRE8LABu9ha4npyJt5eHIvz6fbljEemN5YaIiPKpXd0a24e1QROXR9fC6fPDcew5myB3LCK9sNwQEVGBqlcxw+YPX8CrXo7IztNg+IZT+P5gDE8Vp3KP5YaIiAplZWaC7/u1xMAADwDA7D8u4vOd53iqOJVrLDdERFQkpULC9DcbY+objSBJwIZ/4zB0fRjSs3mqOJVPLDdERKSXwS96YsX7vtpTxXuuOIbbKVlyxyLKh+WGiIj01qlxDWz+0B8O1qa4kJCK7kuP4sKtVLljEelguSEiIoM0d7PDjuFtUNfRGrdTs/DuilAcuJQodywiLZYbIiIymJu9JYKDAuBfuxoyctQYvO4k1h+7JncsIgAsN0REVEy2lir8OLg1erRwhUYAU3edx7RdPJOK5MdyQ0RExWZq8uiu4p++1hAA8OOx6xj8YxhSs3JlTkaVGcsNERE9F0mSMKx9Hax43xcWKiUORd9Fj2WhiEvOlDsaVVIsN0REVCJe866BrUH+cLIxw+XEdHRfdhQnr92TOxZVQiw3RERUYrxdbLFrxIvwdrHBvYwc9P3hXwSH35A7FlUyLDdERFSiatia45eP/PFa4xrIUWswfutpzP3zIjQa3pOKygbLDRERlThLUxMs69sCw9vXAQAs/ScGwzecQmYOb9lApY/lhoiISoVCIeGT1xri23ebQaWUsPf8bfRYfgw37nOgMZUulhsiIipV7/i6YtPQF+BgbYqohFR0W3IUJ2I50JhKD8sNERGVupYe9tg18kU0rmmD5Iwc9F11HJtOxMkdi4wUyw0REZUJFzsLbAsKwOtNnZGrFpi8/Sym7TqHXF7RmEoYyw0REZUZC1MllrzngwmB9QE8uqLxgDUncD8jR+ZkZExYboiIqExJkoSRL9fD9/18YWmqRGhMMrovO4roO2lyRyMjwXJDRESy6NS4BrYPD4BrVQtcT87EW0uP4q8Ld+SORUaA5YaIiGTTsIYNdo98EX6e9sjIUWPoT2FYvP8yL/hHz4XlhoiIZGVvZYqfP/BDvxfcIQQwLyQaQT+HI413FqdiYrkhIiLZqZQKfNndG9/0aAJTpQL7LtxB96VHEXM3Xe5oVAGx3BARUbnRq1Ut/BLkjxo25oi5m4HuS44ihONwyEAsN0REVK40d7PDr6NeRGsPe6Rl52Ho+jAs+Cua43BIb7KXm2XLlsHT0xPm5ubw9fXF4cOHC533wIEDkCQp3+PixYtlmJiIiEpb9Spm2DDUDwP83QEAC/66jA9/Ckcqx+GQHmQtN1u2bMGYMWMwZcoUREREoG3btujcuTPi4oq+JPelS5eQkJCgfdSrV6+MEhMRUVlRKRWY0c0bc99pClMTBf6KejQO50oix+FQ0SQhhGzH+fz8/NCiRQssX75cO83Lywvdu3fH7Nmz881/4MABdOjQAffv34ednV2x1pmamgpbW1ukpKTAxsamuNGJiKgMnbnxAB/9FI6ElCxYm5lgXs9m6NS4htyxqAwZ8v4t25GbnJwchIeHIzAwUGd6YGAgQkNDi1zWx8cHzs7OeOWVV/DPP/8UOW92djZSU1N1HkREVLE0dX00DsfP0x7p2Xn46Kdw/N8fF5HH+1JRAWQrN0lJSVCr1XByctKZ7uTkhNu3bxe4jLOzM1auXIng4GBs374dDRo0wCuvvIJDhw4Vup7Zs2fD1tZW+3BzcyvR10FERGXDwdoMP3/gh8FtPAEAKw7GoN/qE7ibli1zMipvZPtY6tatW3BxcUFoaCj8/f2102fNmoWffvpJ70HCXbt2hSRJ2L17d4Hfz87ORnb2/37xU1NT4ebmxo+liIgqsN/O3MKn284gI0cNJxszLOvbAr7u9nLHolJUIT6WcnBwgFKpzHeUJjExMd/RnKK88MILuHz5cqHfNzMzg42Njc6DiIgqtjea1sSukW1Q19Ead1Kz0ev741h7NBYyDiOlckS2cmNqagpfX1+EhIToTA8JCUFAQIDezxMREQFnZ+eSjkdEROVcXccq2DWiDd5o6ow8jcCMXy9g9OZIZGTnyR2NZGYi58rHjRuHfv36oWXLlvD398fKlSsRFxeHoKAgAMDkyZNx8+ZNrF+/HgCwYMECeHh4oHHjxsjJycHPP/+M4OBgBAcHy/kyiIhIJlZmJlj8ng983ati1u9R+PX0LUQlpGLF+76o62gtdzySiazlplevXkhOTsbMmTORkJAAb29v7NmzB+7ujy7alJCQoHPNm5ycHEyYMAE3b96EhYUFGjdujN9//x1dunSR6yUQEZHMJEnCoDaeaOJiixEbT+FKYjq6LTmCOe80w+tNeWS/MpL1Ojdy4HVuiIiM1920bIzadArHr94DAAxu44lJnRvC1ET2C/LTc6oQA4qJiIhKWvUqZvh5iB+C2tUBAKw5Got3vz+G+HuZMiejssRyQ0RERsVEqcCkzg2xqn9L2FqocDr+AV5fdJh3F69EWG6IiMgovdrICb+PfhHN3OyQmvXo7uKzfr+AXF7V2Oix3BARkdFyrWqJrR/544MXH13V+IfDsej5/THcfPBQ5mRUmlhuiIjIqJmaKPD5G42wsp8vbMxNEBH3AF0WHsb+KH5MZaxYboiIqFIIbFwDv49ui2autkh5mIshP4Zh9p4ofkxlhFhuiIio0nCzt8TWoAAMauMBAPj+0FX0Xnkct/gxlVFhuSEiokrF1ESBaV0bY8X7vqhiboLw6/fRhWdTGRWWGyIiqpRe866B30e1RVNXWzzIzMXQ9WGYtuscsnLVckej58RyQ0RElVatapbYFhSAoW0fnU3147HreGtZKK4kpsucjJ4Hyw0REVVqpiYKTHm9EdYOaoVqVqaISkhF18VHsOVkHCrZHYqMBssNERERgA4NHPHHmLZ4sa4DHuaq8WnwWYzaFIHUrFy5o5GBWG6IiIj+41jFHOsHt8anrzWEiULCb2cS8Pqiw4iIuy93NDIAyw0REdETFAoJw9rXwdYgf7jZWyD+3kO8u+IYlh24Ao2GH1NVBCw3REREBfCpVRW/j26LN5o6I08jMGfvJfRfcwKJqVlyR6NnYLkhIiIqhI25Covf88GcHk1hoVLiyJUkvLbwMPadvy13NCoCyw0REVERJElCz1Zu+HVUGzRytsG9jBx8+FM4Jm8/g8ycPLnjUQFYboiIiPRQ17EKdowIwEcv1YYkAZtOxOP1RUdwOv6B3NHoKSw3REREejIzUWJyFy9s+MAPzrbmiE3KQI/loVjy92WoOdi43GC5ISIiMlBAHQfs/fglvP7fYONv90Wj1/fHEH8vU+5oBJYbIiKiYrG1VGHJez6Y37MZrM1MEHb9PjovPIztp27wysYyY7khIiIqJkmS8HYLV/zxcVu0dK+K9Ow8jPvlNEZuikBKJq9sLBeWGyIioufkZm+JzR++gAmB9WGikPD7mQS8tvAQQq8kyR2tUmK5ISIiKgEmSgVGvlwPwcMC4OlghYSULPRZ9S+m7z6PhzlqueNVKiw3REREJaiZmx1+G/Ui+vjVAgCsC73G+1OVMZYbIiKiEmZlZoKv32qCdYNawcnGDFf/O2X82z8vISdPI3c8o8dyQ0REVEraN3DEvjHt0L15TWgEsOSfK+i+9Cgu3k6VO5pRY7khIiIqRbaWKizo7YNlfVugqqUKFxJS0XXxESw/EMML/5USlhsiIqIy0KWJM/aNbYdXvZyQqxb4Zu9F9Pz+GK4lZcgdzeiw3BAREZWR6lXM8EN/X3z7bjNUMTNB+H8X/lt/7Bo0PIpTYlhuiIiIypAkSXjH1xV7x76ENnWr4WGuGlN3nUf/NSdw88FDueMZBZYbIiIiGbjYWeCnwX6Y8WZjmKsUOHIlCYHzD+Ln49d5FOc5sdwQERHJRKGQMCDAA3tGt0Urj6rIyFHj853n0HfVv4hL5k04i4vlhoiISGa1q1tjy4f+mNa1ESxUShy7moxOCw5h3dFYHsUpBpYbIiKickChkDCojSf2jmmLF2rb42GuGtN/vYDeK48jlmdUGYTlhoiIqBxxr2aFjR+8gC+7e8PKVIkT1+7htQWHsOrwVV4XR08sN0REROWMQiGh3wvu+HPsS2hbzwHZeRp89XsU3lkRiiuJaXLHK/dYboiIiMop16qWWD+4Nf7v7SaoYmaCiLgH6LLoCJYduII8Ne9RVRiWGyIionJMkiT0bl0L+8a9hA4NqiMnT4M5ey/h7eWhuHCL96gqCMsNERFRBeBsa4E1A1th3rvNYGNugjM3UtB1yRF8s/cisnLVcscrV1huiIiIKghJktDD1xV/jWuHLk1qQK0RWH4gBp0WHMLRK0lyxys3WG6IiIgqGEcbcyzr64sf+rdEDRtzXE/ORN9V/2LC1tO4n5EjdzzZyV5uli1bBk9PT5ibm8PX1xeHDx/Wa7mjR4/CxMQEzZs3L92ARERE5VTHRk4IGfcS+vu7Q5KAbeE38Or8g9gVeRNCVN7TxmUtN1u2bMGYMWMwZcoUREREoG3btujcuTPi4uKKXC4lJQX9+/fHK6+8UkZJiYiIyqcq5irM7OaNbUEBqO9kjeSMHHy8ORKD1p3EjfuV8xYOkpCx2vn5+aFFixZYvny5dpqXlxe6d++O2bNnF7pc7969Ua9ePSiVSuzcuRORkZF6rzM1NRW2trZISUmBjY3N88QnIiIqV3LyNFhxMAZL/r6CHLUGlqZKjA9sgIEBHlAqJLnjPRdD3r9lO3KTk5OD8PBwBAYG6kwPDAxEaGhoocutXbsWMTExmDZtml7ryc7ORmpqqs6DiIjIGJmaKDD6lXrY83FbtPawR2aOGl/+dgFvLztaqU4bl63cJCUlQa1Ww8nJSWe6k5MTbt++XeAyly9fxqRJk7BhwwaYmJjotZ7Zs2fD1tZW+3Bzc3vu7EREROVZXUdrbP7wBcx+uwmqmJvg9H+njc/eE4XMnDy545U62QcUS5LuYTIhRL5pAKBWq9GnTx/MmDED9evX1/v5J0+ejJSUFO0jPj7+uTMTERGVdwqFhPda18L+J04b//7QVbw67yD+PH/bqAcc63f4oxQ4ODhAqVTmO0qTmJiY72gOAKSlpSEsLAwREREYOXIkAECj0UAIARMTE+zbtw8vv/xyvuXMzMxgZmZWOi+CiIionHt82vj+qDuYtvs8btx/iI9+CscrDR0x/c3GcLO3lDtiiZPtyI2pqSl8fX0REhKiMz0kJAQBAQH55rexscHZs2cRGRmpfQQFBaFBgwaIjIyEn59fWUUnIiKqcF7xckLI2HYY0aEOVEoJ+y8mouN3B7H0nyvIyTOu+1TJduQGAMaNG4d+/fqhZcuW8Pf3x8qVKxEXF4egoCAAjz5SunnzJtavXw+FQgFvb2+d5R0dHWFubp5vOhEREeVnYarExE4N8ZaPCz7feQ7Hr97D3D8vYUfETXzZzRv+darJHbFEyFpuevXqheTkZMycORMJCQnw9vbGnj174O7uDgBISEh45jVviIiIyDB1Hatg09AXsCPiJmb9HoUriel474fjeNvHBZ+97gUH64o9nEPW69zIgde5ISIi+p+UzFzM+fMiNp6IgxCAjbkJPnmtIfq0rgVFObo2jiHv3yw3REREhMj4B5iy4yzO/3c9nGZudpjV3RveLrYyJ3uE5aYILDdEREQFy1Nr8NPx65i3Lxrp2XlQSMD7L7hjXMf6sLM0lTVbhbhCMREREZUvJkoFBrXxxN/j26Frs5rQCGD9sevo8O0BbPw3DmpNxTgewiM3REREVKDQK0mY/ut5RN9JBwB4u9hgxpve8HWvWuZZ+LFUEVhuiIiI9Jer1uCnY9fx3V/RSMt6dOuGt1u4YFLnhnCsYl5mOVhuisByQ0REZLik9GzM2XsRv4TdAABYm5ng41fqYUCAB0xNSn+UC8tNEVhuiIiIii8y/gGm7TqH0zdSAAB1qlth+puN0bZe9VJdL8tNEVhuiIiIno9GI7At/Aa+2XsRyRk5AIBOjZ3w+euNSu1eVSw3RWC5ISIiKhkpD3Ox4K9orD92HWqNgJmJAkHt6mBY+zowVylLdF0sN0VguSEiIipZl26nYfru8zh2NRkA4GJngR0jAkp0wDGvc0NERERlpkGNKtg41A9L+7RATVtz1K5uheoy3p9K1htnEhERkXGQJAmvN3XGyw0dkZqVC0mS775ULDdERERUYixMlbAwLdnxNobix1JERERkVFhuiIiIyKiw3BAREZFRYbkhIiIio8JyQ0REREaF5YaIiIiMCssNERERGRWWGyIiIjIqLDdERERkVFhuiIiIyKiw3BAREZFRYbkhIiIio8JyQ0REREal0t0VXAgBAEhNTZU5CREREenr8fv24/fxolS6cpOcnAwAcHNzkzkJERERGSotLQ22trZFzlPpyo29vT0AIC4u7pkbpzJITU2Fm5sb4uPjYWNjI3cc2XF7/A+3hS5uD13cHv/DbaGrtLaHEAJpaWmoWbPmM+etdOVGoXg0zMjW1pa/hE+wsbHh9ngCt8f/cFvo4vbQxe3xP9wWukpje+h7UIIDiomIiMiosNwQERGRUal05cbMzAzTpk2DmZmZ3FHKBW4PXdwe/8NtoYvbQxe3x/9wW+gqD9tDEvqcU0VERERUQVS6IzdERERk3FhuiIiIyKiw3BAREZFRYbkhIiIio2KU5WbZsmXw9PSEubk5fH19cfjw4SLnz87OxpQpU+Du7g4zMzPUqVMHa9asKaO0pc/Q7bFhwwY0a9YMlpaWcHZ2xqBBg7S3rajIDh06hK5du6JmzZqQJAk7d+585jIHDx6Er68vzM3NUbt2baxYsaL0g5YRQ7fH9u3b0bFjR1SvXh02Njbw9/fHn3/+WTZhS1lxfjceO3r0KExMTNC8efNSy1fWirM9jHk/WpztYaz70dmzZ6NVq1aoUqUKHB0d0b17d1y6dOmZy5X1vtToys2WLVswZswYTJkyBREREWjbti06d+6MuLi4Qpfp2bMn9u/fj9WrV+PSpUvYtGkTGjZsWIapS4+h2+PIkSPo378/hgwZgvPnz2Pr1q04efIkPvjggzJOXvIyMjLQrFkzLFmyRK/5Y2Nj0aVLF7Rt2xYRERH47LPPMHr0aAQHB5dy0rJh6PY4dOgQOnbsiD179iA8PBwdOnRA165dERERUcpJS5+h2+KxlJQU9O/fH6+88kopJZNHcbaHMe9HDd0exrwfPXjwIEaMGIHjx48jJCQEeXl5CAwMREZGRqHLyLIvFUamdevWIigoSGdaw4YNxaRJkwqc/48//hC2trYiOTm5LOKVOUO3x9y5c0Xt2rV1pi1atEi4urqWWkY5ABA7duwocp5PPvlENGzYUGfaRx99JF544YVSTCYPfbZHQRo1aiRmzJhR8oFkZMi26NWrl/j888/FtGnTRLNmzUo1l1z02R7Gvh99kj7bo7LsR4UQIjExUQAQBw8eLHQeOfalRnXkJicnB+Hh4QgMDNSZHhgYiNDQ0AKX2b17N1q2bIk5c+bAxcUF9evXx4QJE/Dw4cOyiFyqirM9AgICcOPGDezZswdCCNy5cwfbtm3D66+/XhaRy5Vjx47l23adOnVCWFgYcnNzZUpVfmg0GqSlpWlvRlvZrF27FjExMZg2bZrcUWRnzPvR4qhM+9GUlBQAKHI/IMe+1KhunJmUlAS1Wg0nJyed6U5OTrh9+3aBy1y9ehVHjhyBubk5duzYgaSkJAwfPhz37t2r8J8XF2d7BAQEYMOGDejVqxeysrKQl5eHN998E4sXLy6LyOXK7du3C9x2eXl5SEpKgrOzs0zJyod58+YhIyMDPXv2lDtKmbt8+TImTZqEw4cPw8TEqHajxWLM+9HiqCz7USEExo0bhxdffBHe3t6FzifHvtSojtw8JkmSztdCiHzTHtNoNJAkCRs2bEDr1q3RpUsXzJ8/H+vWrTOa/3UYsj0uXLiA0aNHY+rUqQgPD8fevXsRGxuLoKCgsoha7hS07QqaXtls2rQJ06dPx5YtW+Do6Ch3nDKlVqvRp08fzJgxA/Xr15c7TrlQGfajhqgs+9GRI0fizJkz2LRp0zPnLet9qVH9l8PBwQFKpTLfUYnExMR8rfExZ2dnuLi46NxG3cvLC0II3LhxA/Xq1SvVzKWpONtj9uzZaNOmDSZOnAgAaNq0KaysrNC2bVt89dVXlepoRY0aNQrcdiYmJqhWrZpMqeS3ZcsWDBkyBFu3bsWrr74qd5wyl5aWhrCwMERERGDkyJEAHr25CyFgYmKCffv24eWXX5Y5Zdky5v1ocVSG/eioUaOwe/duHDp0CK6urkXOK8e+1KiO3JiamsLX1xchISE600NCQhAQEFDgMm3atMGtW7eQnp6unRYdHQ2FQvHMH1h5V5ztkZmZCYVC99dCqVQC+F/Triz8/f3zbbt9+/ahZcuWUKlUMqWS16ZNmzBw4EBs3LjRKMcP6MPGxgZnz55FZGSk9hEUFIQGDRogMjISfn5+ckcsc8a8Hy0OY96PCiEwcuRIbN++HX///Tc8PT2fuYws+9JSG6osk82bNwuVSiVWr14tLly4IMaMGSOsrKzEtWvXhBBCTJo0SfTr1087f1pamnB1dRXvvPOOOH/+vDh48KCoV6+e+OCDD+R6CSXK0O2xdu1aYWJiIpYtWyZiYmLEkSNHRMuWLUXr1q3legklJi0tTURERIiIiAgBQMyfP19ERESI69evCyHyb4urV68KS0tLMXbsWHHhwgWxevVqoVKpxLZt2+R6CSXK0O2xceNGYWJiIpYuXSoSEhK0jwcPHsj1EkqModviacZ2tpSh28PY96OGbg9j3o8OGzZM2NraigMHDujsBzIzM7XzlId9qdGVGyGEWLp0qXB3dxempqaiRYsWOqeoDRgwQLRr105n/qioKPHqq68KCwsL4erqKsaNG6fzg6roDN0eixYtEo0aNRIWFhbC2dlZ9O3bV9y4caOMU5e8f/75RwDI9xgwYIAQouBtceDAAeHj4yNMTU2Fh4eHWL58edkHLyWGbo927doVOX9FVpzfjScZW7kpzvYw5v1ocbaHse5HC9oOAMTatWu185SHfan0X1giIiIio2BUY26IiIiIWG6IiIjIqLDcEBERkVFhuSEiIiKjwnJDRERERoXlhoiIiIwKyw0REREZFZYbIiIiMiosN1Soa9euQZIkREZGlup6MjMz0aNHD9jY2ECSJDx48EDvZSVJws6dO0s0z7p162BnZ1eiz/k8PDw8sGDBghJ/3qNHj6JJkyZQqVTo3r273suVt+0jhMCHH34Ie3v7Mvl9NTZl9XdeVqZPn47mzZtrvx44cOAzf7/bt2+PMWPGlGouKlssNxXcwIEDIUkSJEmCiYkJatWqhWHDhuH+/fsGP8/TOwA3NzckJCTA29u7BBPn9+OPP+Lw4cMIDQ1FQkKCzp2FH3t6h2WMyro0jBs3Ds2bN0dsbCzWrVtX4DylVaxK0t69e7Fu3Tr89ttvZfL7KoeSKvH6vNEbm4ULFxb6+11cBw4cMPg/YoUprb+xivC3W5pM5A5Az++1117D2rVrkZeXhwsXLmDw4MF48OABNm3a9FzPq1QqUaNGjRJKWbiYmBh4eXkZ5ZtSeRYTE4OgoKBye9fmnJwcmJqaPnO+mJgYODs7F3qne30IIaBWq2Fiwl2isSnoP0tUCZTqnauo1A0YMEB069ZNZ9q4ceOEvb299uu8vDwxePBg4eHhIczNzUX9+vXFggULtN+fNm1avpug/fPPPyI2NlYAEBEREdp5Dxw4IFq1aiVMTU1FjRo1xKeffipyc3OLzLht2zbRqFEjYWpqKtzd3cW3336r/d7TN2Ms6OaEa9euLfQmbQDEDz/8ILp37y4sLCxE3bp1xa5du3SWP3/+vOjcubOwsrISjo6O4v333xd3794tNO/atWuFra2tzrTdu3eLFi1aCDMzM+Hp6SmmT5+u87r1ybFr1y5Rt25dYW5uLtq3by/WrVsnAIj79+8XeGO+adOmCSGEcHd3F7NmzRKDBg0S1tbWws3NTXz//fdFbvOsrCwxatQoUb16dWFmZibatGkjTpw4IYQQ2p9rQdvzSQXdKPPJ7bN3717RsGFDYWVlJTp16iRu3bqls/yaNWtEw4YNhZmZmWjQoIFYunRpkZnbtWsnRowYIcaOHSuqVasmXnrpJSFE0T+/AQMG6ORzd3cXQgih0WjEN998Izw9PYW5ublo2rSp2Lp1q3Zdj7f33r17ha+vr1CpVOLvv//We7m//vpL+Pr6CgsLC+Hv7y8uXryY72ft6+srzMzMRLVq1cRbb72l/V52draYOHGiqFmzprC0tBStW7cW//zzT6Hbxd3dvcDXKIQQy5YtE7Vr1xYqlUrUr19frF+/vtDnedbfeXBwsGjfvr2wsLAQTZs2FaGhoTrLHz16VLRt21aYm5sLV1dXMWrUKJGenl7o+p61HX766Sfh6+srrK2thZOTk3jvvffEnTt3DN7Ws2fPFo6OjsLa2loMHjxYfPrppzo3MX16H5meni769esnrKysRI0aNcS3334r2rVrJz7++GO9shX09/P4BprP+v15WmF/Y8/a3j/++KOwsrIS0dHR2vlHjhwp6tWrJ9LT04t83sqi8r1iI/P0H25MTIxo1KiRcHJy0k7LyckRU6dOFSdOnBBXr14VP//8s7C0tBRbtmwRQgiRlpYmevbsKV577TXt7euzs7PzlZsbN24IS0tLMXz4cBEVFSV27NghHBwctG/CBQkLCxMKhULMnDlTXLp0Saxdu1ZYWFho30yTk5PF0KFDhb+/v0hISBDJycn5niMzM1OMHz9eNG7cWJvv8d2GAQhXV1exceNGcfnyZTF69GhhbW2tfZ5bt24JBwcHMXnyZBEVFSVOnTolOnbsKDp06FBo5qfLzd69e4WNjY1Yt26diImJEfv27RMeHh5i+vTp2nmelSM2NlaoVCoxYcIEcfHiRbFp0ybh4uKiLTfZ2dliwYIFwsbGRvsa09LShBCP3tzs7e3F0qVLxeXLl8Xs2bOFQqEQUVFRhb6G0aNHi5o1a4o9e/aI8+fPiwEDBoiqVauK5ORkkZeXJxISEoSNjY1YsGCBzvZ8UnJysnB1dRUzZ87UZnq8fVQqlXj11VfFyZMnRXh4uPDy8hJ9+vTRLrty5Urh7OwsgoODxdWrV0VwcLCwt7cX69atKzRzu3bthLW1tZg4caK4ePGiiIqKeubP78GDB2LmzJnC1dVVJCQkiMTERCGEEJ999plo2LCh2Lt3r4iJiRFr164VZmZm4sCBA0KI/71xNm3aVOzbt09cuXJFJCUl6b2cn5+fOHDggDh//rxo27atCAgI0L6O3377TSiVSjF16lRx4cIFERkZKWbNmqX9fp8+fURAQIA4dOiQuHLlipg7d64wMzPTeaN6UmJioraAPvkat2/fLlQqlVi6dKm4dOmSmDdvnlAqleLvv/8u8Hme9XfesGFD8dtvv4lLly6Jd955R7i7u2sL/JkzZ4S1tbX47rvvRHR0tDh69Kjw8fERAwcOLPTn+aztsHr1arFnzx4RExMjjh07Jl544QXRuXNn7ff12dZbtmwRpqam4ocffhAXL14UU6ZMEVWqVCmy3AwbNky4urqKffv2iTNnzog33nhDWFtb65SborLl5eWJ4OBgAUBcunRJJCQkiAcPHgghnv1797TC/sb02d7vvvuuaNWqlcjNzRV//PGHUKlU2v/AFPa8lQnLTQU3YMAAoVQqhZWVlTA3N9e29Pnz5xe53PDhw0WPHj10nufpI0BPl5vPPvtMNGjQQGg0Gu08S5cuFdbW1kKtVhe4nj59+oiOHTvqTJs4caJo1KiR9uuPP/64wCM2T5o2bZrODusxAOLzzz/Xfp2eni4kSRJ//PGHEEKIL774QgQGBuosEx8fr90xFeTpctO2bVvx9ddf68zz008/CWdnZ71zfPrpp8Lb21vnOaZMmaItNwWt9zF3d3fx/vvva7/WaDTC0dFRLF++vMD86enpQqVSiQ0bNmin5eTkiJo1a4o5c+Zop9na2hZ4xObpdX/33Xc60x4fSbty5Yp22tKlS3UKtZubm9i4caPOcl9++aXw9/cvdF3t2rUTzZs315mmz8/vu+++0zmakZ6eLszNzfMdeRgyZIh47733hBD/e+PcuXNnsZb766+/tN///fffBQDx8OFDIYQQ/v7+om/fvgW+xitXrghJksTNmzd1pr/yyiti8uTJBW8Y8ej3a8eOHTrTAgICxNChQ3Wmvfvuu6JLly6FPk9Rf+erVq3STjt//rwAoC3Q/fr1Ex9++KHOcocPHxYKhUL7up9W1HYoyIkTJwQAbanXd1sHBQXpPI+fn1+h5SYtLU2YmpqKzZs3a7+fnJwsLCwsdMqNvtke/+0Kod/vT0EK+hvTZ3vfu3dPuLq6imHDhgknJyfx1VdfPfN5KxN+wGwEOnTogOXLlyMzMxOrVq1CdHQ0Ro0apTPPihUrsGrVKly/fh0PHz5ETk6OwQN0o6Ki4O/vD0mStNPatGmD9PR03LhxA7Vq1SpwmW7duulMa9OmDRYsWAC1Wg2lUmlQhoI0bdpU+28rKytUqVIFiYmJAIDw8HD8888/sLa2zrdcTEwM6tev/8znDw8Px8mTJzFr1iztNLVajaysLGRmZsLS0vKZOS5duoRWrVrpPG/r1q2L9RolSUKNGjW0z13Q68rNzUWbNm2001QqFVq3bo2oqCi911kUS0tL1KlTR/u1s7OzNs/du3cRHx+PIUOGYOjQodp58vLynjn+oWXLljpfF+fnd+HCBWRlZaFjx44603NycuDj41Po+gxZ7smfh7OzMwAgMTERtWrVQmRkpM7rftKpU6cghMiXOzs7G9WqVStwmcJERUXhww8/1JnWpk0bLFy40KDneayw19SwYUOEh4fjypUr2LBhg3YeIQQ0Gg1iY2Ph5eWV7/mK2g4AEBERgenTpyMyMhL37t2DRqMBAMTFxaFRo0bPzFWrVi1ERUUhKChI53n9/f3xzz//FLjOmJgY5OTkwN/fXzvN3t4eDRo0KFa2Jxny+/Ms+mzvqlWrYvXq1ejUqRMCAgIwadIkg9Zh7FhujICVlRXq1q0LAFi0aBE6dOiAGTNm4MsvvwQA/PLLLxg7dizmzZsHf39/VKlSBXPnzsW///5r0HqEEDrF5vE0APmm67NMSVGpVDpfS5Kk3RlpNBp07doV33zzTb7lHu8on0Wj0WDGjBl4++23833P3NxcrxzPux2Keu6nFfYzKShDcRWU5/F6H+f64Ycf4OfnpzPfs8qslZWVztfF+fk9Xv/vv/8OFxcXne+ZmZkVuj5Dlnvy9T/epo+Xt7CwKDDX43mUSiXCw8PzbYuCCtyzlOTPuKjXpNFo8NFHH2H06NH5livoPzVA0dshIyMDgYGBCAwMxM8//4zq1asjLi4OnTp1Qk5Ojt65DKXP35wh2Z5kyO/Ps+i7vQ8dOgSlUolbt24hIyMDNjY2Bq3HmLHcGKFp06ahc+fOGDZsGGrWrInDhw8jICAAw4cP184TExOjs4ypqSnUanWRz9uoUSMEBwfr7EBDQ0NRpUqVfH/MTy5z5MgRnWmhoaGoX7++QUdt9MlXkBYtWiA4OBgeHh7FPhOmRYsWuHTpkrZAFkfDhg2xZ88enWlhYWE6Xxf3NT6tbt26MDU1xZEjR9CnTx8AQG5uLsLCwgy+lkdxMjk5OcHFxQVXr15F3759DVr2acX5+TVq1AhmZmaIi4tDu3bt9F5XcZd7WtOmTbF//34MGjQo3/d8fHygVquRmJiItm3b6v2cKpUq38/By8sLR44cQf/+/bXTQkNDCzyK8tjz/B2dP3/eoL+BorbDxYsXkZSUhP/7v/+Dm5sbgPx/D/rw8vLC8ePHdbbB8ePHC52/bt26UKlUOH78uLYk3L9/H9HR0dqfuT7ZHp/F9+S2LO7vT0E/E322d2hoKObMmYNff/0VkyZNwqhRo/Djjz8W+byVCa9zY4Tat2+Pxo0b4+uvvwbw6A86LCwMf/75J6Kjo/HFF1/g5MmTOst4eHjgzJkzuHTpEpKSkpCbm5vveYcPH474+HiMGjUKFy9exK5duzBt2jSMGzcOCkXBv0rjx4/H/v378eWXXyI6Oho//vgjlixZggkTJhj0mjw8PBAbG4vIyEgkJSUhOztbr+VGjBiBe/fu4b333sOJEydw9epV7Nu3D4MHD9b7D3/q1KlYv349pk+fjvPnzyMqKgpbtmzB559/rnf+jz76CBcvXsSnn36K6Oho/PLLL9prbzwuih4eHkhPT8f+/fuRlJSEzMxMvZ//SVZWVhg2bBgmTpyIvXv34sKFCxg6dCgyMzMxZMgQg57Lw8MDhw4dws2bN5GUlKT3ctOnT8fs2bOxcOFCREdH4+zZs1i7di3mz59v0PqL8/OrUqUKJkyYgLFjx+LHH39ETEwMIiIisHTpUp2df0kt97Rp06Zh06ZNmDZtGqKionD27FnMmTMHAFC/fn307dsX/fv3x/bt2xEbG4uTJ0/im2++yVd+n+Th4YH9+/fj9u3b2mtYTZw4EevWrcOKFStw+fJlzJ8/H9u3by/yb0ufv/OCfPrppzh27BhGjBiByMhIXL58Gbt378738be+26FWrVowNTXF4sWLcfXqVezevVt7pNkQH3/8MdasWYM1a9YgOjoa06ZNw/nz5wud39raGkOGDMHEiROxf/9+nDt3DgMHDtTZf+mTzd3dHZIk4bfffsPdu3eRnp5e7N+fgv7GnrW909LS0K9fP4waNQqdO3fGxo0b8csvv2Dr1q1FPm+lIsM4HypBBQ0QFEKIDRs2CFNTUxEXFyeysrLEwIEDha2trbCzsxPDhg0TkyZN0hl0l5iYKDp27Cisra1L7VRwlUolatWqJebOnavzfX0GFGdlZYkePXoIOzu7fKeCPz3Q8umBstHR0eKtt94SdnZ2wsLCQjRs2FCMGTNGZ2D0kwoa2Lt3714REBAgLCwshI2NjWjdurVYuXKl9vv65Hh8KriZmZlo3769WL58uc7gSCGECAoKEtWqVct3KvjTAwObNWtW5FlqDx8+FKNGjRIODg75TgUvLF9Bjh07Jpo2bSrMzMzynQr+pB07duQ73XTDhg2iefPmwtTUVFStWlW89NJLYvv27YWu6+nTcR971s/v6QHFQjwadL1w4ULRoEEDoVKpRPXq1UWnTp3EwYMHhRAFDwgt7nIRERECgIiNjdVOCw4O1r52BwcH8fbbb2u/9/jsRQ8PD6FSqUSNGjXEW2+9Jc6cOVPottm9e7eoW7euMDExKfap4ELo/3d+//597fcfO3HihHZZKysr0bRpU52znwpS1HbYuHGj8PDwEGZmZsLf31/s3r1bJ4e+23rWrFnCwcFBWFtbiwEDBohPPvmkyLOl0tLSxPvvvy8sLS2Fk5OTmDNnTr7fvWdlE0KImTNniho1aghJknROBS/q96cgBf2NPWt7Dxo0SDRp0kRkZWVp51+4cKGwt7cXN27cKPJ5KwtJiBIeAEFEepk1axZWrFiB+Ph4uaMQERkVjrkhKiPLli1Dq1atUK1aNRw9ehRz587FyJEj5Y5FRGR0WG6Iysjly5fx1Vdf4d69e6hVqxbGjx+PyZMnyx2LiMjo8GMpIiIiMio8W4qIiIiMCssNERERGRWWGyIiIjIqLDdERERkVFhuiIiIyKiw3BAREZFRYbkhIiIio8JyQ0REREbl/wEzBPVvSKSe5gAAAABJRU5ErkJggg==",
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "reference_length = 1\n",
+    "candidate_length = np.linspace(1.5, 0.5, 100)\n",
+    "\n",
+    "length_ratio = reference_length / candidate_length\n",
+    "BP = np.minimum(1, np.exp(1 - length_ratio))\n",
+    "\n",
+    "# Plot the data\n",
+    "fig, ax = plt.subplots(1)\n",
+    "lines = ax.plot(length_ratio, BP)\n",
+    "ax.set(\n",
+    "    xlabel=\"Ratio of the length of the reference to the candidate text\",\n",
+    "    ylabel=\"Brevity Penalty\",\n",
+    ")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### N-Gram Precision:\n",
+    "The n-gram precision counts how many n-grams (in your case unigrams, bigrams, trigrams, and four-grams for i =1 , ... , 4) match their n-gram counterpart in the reference translations. This term acts as a precision metric. Unigrams account for adequacy while longer n-grams account for fluency of the translation. To avoid overcounting, the n-gram counts are clipped to the maximal n-gram count occurring in the reference ($m_{n}^{ref}$). Typically precision shows exponential decay with the degree of the n-gram."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGdCAYAAADuR1K7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAseElEQVR4nO3de1BV9d7H8c8WZGMomJB4aYuUphTZMbBzwDyWF4qcTj3TxW6aCnMiyiTqlGTlrRN2I7pB+qjHPJVxyuwykrVPpWJkTyJOPWlXtU20kcAOoBYErOcPxz3PDlSWbNiwfL9m1kzrt35rre/2N46ffutmMwzDEAAAgEX08HcBAAAAvkS4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlhLo7wI6W3Nzs3788Uf16dNHNpvN3+UAAIA2MAxDdXV1GjRokHr0OPbczEkXbn788Uc5HA5/lwEAAE5AWVmZTj/99GP2OenCTZ8+fSQd/sMJDQ31czUAAKAtamtr5XA4PP+OH8tJF26OXIoKDQ0l3AAA0M205ZYSbigGAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACW4vdwk5eXp+joaAUHBysuLk5FRUXH7P/SSy/pvPPO0ymnnKKBAwdq5syZqq6u7qRqAQBAV+fXcFNQUKCMjAzNmzdPpaWlGjdunJKTk+VyuVrtv2XLFk2fPl0pKSn64osv9Oqrr+rTTz9VampqJ1cOAAC6Kr+Gm5ycHKWkpCg1NVUxMTHKzc2Vw+FQfn5+q/23bt2qoUOH6o477lB0dLQuvPBC3XLLLdq2bVsnVw4AALoqv4WbhoYGlZSUKCkpyas9KSlJxcXFre6TmJioH374QYWFhTIMQ/v27dNrr72mKVOmHPU89fX1qq2t9VoAAIB1BfrrxFVVVWpqalJkZKRXe2RkpCoqKlrdJzExUS+99JKmTp2qX3/9VY2NjfrLX/6iZ5555qjnyc7O1sKFC31a+7EMnbu+084Fb3uXHD3kAgBOHn6/odhms3mtG4bRou2InTt36o477tCDDz6okpISbdiwQXv27FFaWtpRj5+VlaWamhrPUlZW5tP6AQBA1+K3mZuIiAgFBAS0mKWprKxsMZtzRHZ2tsaOHau//e1vkqRRo0YpJCRE48aN00MPPaSBAwe22Mdut8tut/v+BwAAgC7JbzM3QUFBiouLk9Pp9Gp3Op1KTExsdZ9Dhw6pRw/vkgMCAiQdnvEBAADw62WpzMxMLV++XCtXrtSuXbt05513yuVyeS4zZWVlafr06Z7+l19+uV5//XXl5+dr9+7d+uijj3THHXfoggsu0KBBg/z1MwAAQBfit8tSkjR16lRVV1dr0aJFcrvdio2NVWFhoaKioiRJbrfb6503M2bMUF1dnZ599lnddddd6tu3ryZMmKBHHnnEXz8BAAB0MTbjJLueU1tbq7CwMNXU1Cg0NNTnx+dpKf/haSkAsC4z/377/WkpAAAAXyLcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAAS/F7uMnLy1N0dLSCg4MVFxenoqKio/adMWOGbDZbi+Wcc87pxIoBAEBX5tdwU1BQoIyMDM2bN0+lpaUaN26ckpOT5XK5Wu3/1FNPye12e5aysjL169dP11xzTSdXDgAAuiq/hpucnBylpKQoNTVVMTExys3NlcPhUH5+fqv9w8LCNGDAAM+ybds2/fzzz5o5c2YnVw4AALoqv4WbhoYGlZSUKCkpyas9KSlJxcXFbTrGihUrNGnSJEVFRR21T319vWpra70WAABgXYH+OnFVVZWampoUGRnp1R4ZGamKiorj7u92u/XOO+/o5ZdfPma/7OxsLVy4sF21ApI0dO56f5dw0tq7ZIq/SwDQjfj9hmKbzea1bhhGi7bWrFq1Sn379tWVV155zH5ZWVmqqanxLGVlZe0pFwAAdHF+m7mJiIhQQEBAi1maysrKFrM5v2cYhlauXKlp06YpKCjomH3tdrvsdnu76wUAAN2D32ZugoKCFBcXJ6fT6dXudDqVmJh4zH03bdqkb7/9VikpKR1ZIgAA6Ib8NnMjSZmZmZo2bZri4+OVkJCgZcuWyeVyKS0tTdLhS0rl5eVavXq1134rVqzQH//4R8XGxvqjbAAA0IX5NdxMnTpV1dXVWrRokdxut2JjY1VYWOh5+sntdrd4501NTY3Wrl2rp556yh8lAwCALs6v4UaS0tPTlZ6e3uq2VatWtWgLCwvToUOHOrgqAADQXfn9aSkAAABfItwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABL8Xu4ycvLU3R0tIKDgxUXF6eioqJj9q+vr9e8efMUFRUlu92uM888UytXruykagEAQFcX6M+TFxQUKCMjQ3l5eRo7dqyWLl2q5ORk7dy5U0OGDGl1n2uvvVb79u3TihUrNGzYMFVWVqqxsbGTKwcAAF2VX8NNTk6OUlJSlJqaKknKzc3Vu+++q/z8fGVnZ7fov2HDBm3atEm7d+9Wv379JElDhw7tzJIBAEAX57fLUg0NDSopKVFSUpJXe1JSkoqLi1vd56233lJ8fLweffRRDR48WGeddZbuvvtu/fLLL0c9T319vWpra70WAABgXX6buamqqlJTU5MiIyO92iMjI1VRUdHqPrt379aWLVsUHBysdevWqaqqSunp6dq/f/9R77vJzs7WwoULfV4/AGsYOne9v0s4ae1dMsXfJcCi/H5Dsc1m81o3DKNF2xHNzc2y2Wx66aWXdMEFF+iyyy5TTk6OVq1addTZm6ysLNXU1HiWsrIyn/8GAADQdfht5iYiIkIBAQEtZmkqKytbzOYcMXDgQA0ePFhhYWGetpiYGBmGoR9++EHDhw9vsY/dbpfdbvdt8QAAoMvy28xNUFCQ4uLi5HQ6vdqdTqcSExNb3Wfs2LH68ccfdeDAAU/b119/rR49euj000/v0HoBAED34NfLUpmZmVq+fLlWrlypXbt26c4775TL5VJaWpqkw5eUpk+f7ul/ww03KDw8XDNnztTOnTu1efNm/e1vf9OsWbPUq1cvf/0MAADQhfj1UfCpU6equrpaixYtktvtVmxsrAoLCxUVFSVJcrvdcrlcnv69e/eW0+nU7NmzFR8fr/DwcF177bV66KGH/PUTAABAF+PXcCNJ6enpSk9Pb3XbqlWrWrSNHDmyxaUsAACAI/z+tBQAAIAvEW4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClBJrd4eDBg1qyZInef/99VVZWqrm52Wv77t27fVYcAACAWabDTWpqqjZt2qRp06Zp4MCBstlsHVEXAADACTEdbt555x2tX79eY8eO7Yh6AAAA2sX0PTennnqq+vXr1xG1AAAAtJvpcLN48WI9+OCDOnToUEfUAwAA0C6mL0s98cQT+u677xQZGamhQ4eqZ8+eXtu3b9/us+IAAADMMh1urrzyyg4oAwAAwDdMh5v58+d3RB0AAAA+YTrcHFFSUqJdu3bJZrPp7LPP1ujRo31ZFwAAwAkxHW4qKyt13XXXaePGjerbt68Mw1BNTY0uvvhivfLKKzrttNM6ok4AAIA2Mf201OzZs1VbW6svvvhC+/fv188//6z//d//VW1tre64446OqBEAAKDNTM/cbNiwQf/+978VExPjaTv77LP13HPPKSkpyafFAQAAmGV65qa5ubnF49+S1LNnzxbfmQIAAOhspsPNhAkTNGfOHP3444+etvLyct15552aOHGiT4sDAAAwy3S4efbZZ1VXV6ehQ4fqzDPP1LBhwxQdHa26ujo988wzHVEjAABAm5m+58bhcGj79u1yOp368ssvZRiGzj77bE2aNKkj6gMAADDlhN9zM3nyZE2ePNmXtQAAALRbm8LN008/rb/+9a8KDg7W008/fcy+PA4OAAD8qU3h5sknn9SNN96o4OBgPfnkk0ftZ7PZTIebvLw8PfbYY3K73TrnnHOUm5urcePGtdp348aNuvjii1u079q1SyNHjjR1XgAAYE1tCjd79uxp9b/bq6CgQBkZGcrLy9PYsWO1dOlSJScna+fOnRoyZMhR9/vqq68UGhrqWeetyAAA4AjTT0v9XlNTk3bs2KGff/7Z9L45OTlKSUlRamqqYmJilJubK4fDofz8/GPu179/fw0YMMCzBAQEnGj5AADAYkyHm4yMDK1YsULS4WDz5z//Weeff74cDoc2btzY5uM0NDSopKSkxVuNk5KSVFxcfMx9R48erYEDB2rixIn68MMPj9m3vr5etbW1XgsAALAu009Lvfbaa7rpppskSW+//bb27t2rL7/8UqtXr9a8efP00Ucftek4VVVVampqUmRkpFd7ZGSkKioqWt1n4MCBWrZsmeLi4lRfX69//vOfmjhxojZu3Kg///nPre6TnZ2thQsXmviFAAArGDp3vb9LOGntXTLFr+c3HW6qqqo0YMAASVJhYaGuueYanXXWWUpJSTnuk1StsdlsXuuGYbRoO2LEiBEaMWKEZz0hIUFlZWV6/PHHjxpusrKylJmZ6Vmvra2Vw+EwXScAAOgeTF+WioyM1M6dO9XU1KQNGzZ4Xt536NAhU/e+REREKCAgoMUsTWVlZYvZnGP505/+pG+++eao2+12u0JDQ70WAABgXabDzcyZM3XttdcqNjZWNpvN8yK/Tz75xNTj2EFBQYqLi5PT6fRqdzqdSkxMbPNxSktLNXDgwDb3BwAA1mb6stSCBQsUGxursrIyXXPNNbLb7ZKkgIAAzZ0719SxMjMzNW3aNMXHxyshIUHLli2Ty+VSWlqapMOXlMrLy7V69WpJUm5uroYOHapzzjlHDQ0NevHFF7V27VqtXbvW7M8AAAAWdUKfX7j66qtbtN18882mjzN16lRVV1dr0aJFcrvdio2NVWFhoaKioiRJbrdbLpfL07+hoUF33323ysvL1atXL51zzjlav369LrvsshP5GQAAwIL8/vmF9PR0paent7pt1apVXuv33HOP7rnnHlPHBwAAJxe/f34BAADAl/z6+QUAAABfa/fnFwAAALoS0+Hm6quv1pIlS1q0P/bYY7rmmmt8UhQAAMCJMh1uNm3apClTWr5W+dJLL9XmzZt9UhQAAMCJMh1uDhw4oKCgoBbtPXv25KOUAADA70yHm9jYWBUUFLRof+WVV3T22Wf7pCgAAIATZfolfg888ICuuuoqfffdd5owYYIk6f3339eaNWv06quv+rxAAAAAM0yHm7/85S9644039PDDD+u1115Tr169NGrUKP373//W+PHjO6JGAACANjuhzy9MmTKl1ZuKAQAA/O2E3nPzn//8R8uXL9d9992n/fv3S5K2b9+u8vJynxYHAABglumZm88++0yTJk1SWFiY9u7dq9TUVPXr10/r1q3T999/7/mCNwAAgD+YnrnJzMzUjBkz9M033yg4ONjTnpyczHtuAACA35kON59++qluueWWFu2DBw9WRUWFT4oCAAA4UabDTXBwcKsv6/vqq6902mmn+aQoAACAE2U63FxxxRVatGiRfvvtN0mSzWaTy+XS3LlzddVVV/m8QAAAADNMh5vHH39cP/30k/r3769ffvlF48eP17Bhw9SnTx/9/e9/74gaAQAA2sz001KhoaHasmWLPvjgA23fvl3Nzc06//zzNWnSpI6oDwAAwBRT4aaxsVHBwcHasWOHJkyY4Pn8AgAAQFdh6rJUYGCgoqKi1NTU1FH1AAAAtIvpe27uv/9+ZWVled5MDAAA0JWYvufm6aef1rfffqtBgwYpKipKISEhXtu3b9/us+IAAADMMh1urrzyyg4oAwAAwDdMh5v58+d3RB0AAAA+YTrcHLFt2zbt2rVLNptNMTExiouL82VdAAAAJ8R0uPnhhx90/fXX66OPPlLfvn0lSf/5z3+UmJioNWvWyOFw+LpGAACANjP9tNSsWbP022+/adeuXdq/f7/279+vXbt2yTAMpaSkdESNAAAAbWZ65qaoqEjFxcUaMWKEp23EiBF65plnNHbsWJ8WBwAAYJbpmZshQ4Z4Ppr5/zU2Nmrw4ME+KQoAAOBEmQ43jz76qGbPnq1t27bJMAxJh28unjNnjh5//HGfFwgAAGCG6ctSM2bM0KFDh/THP/5RgYGHd29sbFRgYKBmzZqlWbNmefryFmMAANDZTIeb3NzcDigDAADAN0yHm5tvvrkj6gAAAPAJ0/fc+FpeXp6io6MVHBysuLg4FRUVtWm/jz76SIGBgfrDH/7QsQUCAIBuxa/hpqCgQBkZGZo3b55KS0s1btw4JScny+VyHXO/mpoaTZ8+XRMnTuykSgEAQHfh13CTk5OjlJQUpaamKiYmRrm5uXI4HMrPzz/mfrfccotuuOEGJSQkdFKlAACgu/BbuGloaFBJSYmSkpK82pOSklRcXHzU/f7xj3/ou+++a/MHPOvr61VbW+u1AAAA6/JbuKmqqlJTU5MiIyO92iMjI1VRUdHqPt98843mzp2rl156yfMY+vFkZ2crLCzMs/DtKwAArM3001K//vqrnnnmGX344YeqrKxUc3Oz1/bt27ebOp7NZvNaNwyjRZskNTU16YYbbtDChQt11llntfn4WVlZyszM9KzX1tYScAAAsDDT4WbWrFlyOp26+uqrdcEFF7QaRNoiIiJCAQEBLWZpKisrW8zmSFJdXZ22bdum0tJS3X777ZKk5uZmGYahwMBAvffee5owYUKL/ex2u+x2+wnVCAAAuh/T4Wb9+vUqLCxs90cyg4KCFBcXJ6fTqf/6r//ytDudTl1xxRUt+oeGhurzzz/3asvLy9MHH3yg1157TdHR0e2qBwAAWIPpcDN48GD16dPHJyfPzMzUtGnTFB8fr4SEBC1btkwul0tpaWmSDl9SKi8v1+rVq9WjRw/FxsZ67d+/f38FBwe3aAcAACcv0+HmiSee0L333qvnn39eUVFR7Tr51KlTVV1drUWLFsntdis2NlaFhYWe47rd7uO+8wYAAOD/Mx1u4uPj9euvv+qMM87QKaecop49e3ptN/uxzPT0dKWnp7e6bdWqVcfcd8GCBVqwYIGp8wEAAGszHW6uv/56lZeX6+GHH1ZkZOQJ31AMAADQEUyHm+LiYn388cc677zzOqIeAACAdjH9Er+RI0fql19+6YhaAAAA2s10uFmyZInuuusubdy4UdXV1XzaAAAAdCmmL0tdeumlktTii9xH3izc1NTkm8oAAABOgOlw8+GHH3ZEHQAAAD5hOtyMHz++I+oAAADwCdPh5ohDhw7J5XKpoaHBq33UqFHtLgoAAOBEmQ43P/30k2bOnKl33nmn1e3ccwMAAPzJ9NNSGRkZ+vnnn7V161b16tVLGzZs0AsvvKDhw4frrbfe6ogaAQAA2sz0zM0HH3ygN998U2PGjFGPHj0UFRWlyZMnKzQ0VNnZ2ZoyZUpH1AkAANAmpmduDh48qP79+0uS+vXrp59++kmSdO6552r79u2+rQ4AAMAk0+FmxIgR+uqrryRJf/jDH7R06VKVl5fr+eef18CBA31eIAAAgBmmL0tlZGTI7XZLkubPn69LLrlEL730koKCgo77FW8AAICOZjrc3HjjjZ7/Hj16tPbu3asvv/xSQ4YMUUREhE+LAwAAMMvUZanffvtNZ5xxhnbu3OlpO+WUU3T++ecTbAAAQJdgKtz07NlT9fX1stlsHVUPAABAu5i+oXj27Nl65JFH1NjY2BH1AAAAtIvpe24++eQTvf/++3rvvfd07rnnKiQkxGv766+/7rPiAAAAzDIdbvr27aurrrqqI2oBAABoN9Ph5h//+EdH1AEAAOATpu+5AQAA6MpMz9yMHj261aelbDabgoODNWzYMM2YMUMXX3yxTwoEAAAww/TMzaWXXqrdu3crJCREF198sS666CL17t1b3333ncaMGSO3261JkybpzTff7Ih6AQAAjsn0zE1VVZXuuusuPfDAA17tDz30kL7//nu99957mj9/vhYvXqwrrrjCZ4UCAAC0hemZm3/961+6/vrrW7Rfd911+te//iVJuv766z0f1wQAAOhMpsNNcHCwiouLW7QXFxcrODhYktTc3Cy73d7+6gAAAEwyfVlq9uzZSktLU0lJicaMGSObzab/+Z//0fLly3XfffdJkt59912NHj3a58UCAAAcj+lwc//99ys6OlrPPvus/vnPf0qSRowYof/+7//WDTfcIElKS0vTrbfe6ttKAQAA2sB0uJGkG2+8UTfeeONRt/fq1euECwIAAGiPdr3ELz09XVVVVb6qBQAAoN3aFW5efPFF1dbW+qoWAACAdmtXuDEMw1d1AAAA+ITfvy2Vl5en6OhoBQcHKy4uTkVFRUftu2XLFo0dO1bh4eHq1auXRo4cqSeffLITqwUAAF3dCd1QfERdXV27Tl5QUKCMjAzl5eVp7NixWrp0qZKTk7Vz504NGTKkRf+QkBDdfvvtGjVqlEJCQrRlyxbdcsstCgkJ0V//+td21QIAAKzBrzM3OTk5SklJUWpqqmJiYpSbmyuHw6H8/PxW+48ePVrXX3+9zjnnHA0dOlQ33XSTLrnkkmPO9gAAgJNLm8NNjx49FBAQcMwlMLDtE0ENDQ0qKSlRUlKSV3tSUlKrb0BuTWlpqYqLizV+/Pij9qmvr1dtba3XAgAArKvNaWTdunVH3VZcXKxnnnnG1A3GVVVVampqUmRkpFd7ZGSkKioqjrnv6aefrp9++kmNjY1asGCBUlNTj9o3OztbCxcubHNdAACge2tzuGntC99ffvmlsrKy9Pbbb+vGG2/U4sWLTRdgs9m81g3DaNH2e0VFRTpw4IC2bt2quXPnatiwYa1+zFOSsrKylJmZ6Vmvra2Vw+EwXScAAOgeTuiG4h9//FHz58/XCy+8oEsuuUQ7duxQbGysqWNEREQoICCgxSxNZWVli9mc34uOjpYknXvuudq3b58WLFhw1HBjt9v5iCcAACcRUzcU19TU6N5779WwYcP0xRdf6P3339fbb79tOthIUlBQkOLi4uR0Or3anU6nEhMT23wcwzBUX19v+vwAAMCa2jxz8+ijj+qRRx7RgAEDtGbNmlYvU5mVmZmpadOmKT4+XgkJCVq2bJlcLpfS0tIkHb6kVF5ertWrV0uSnnvuOQ0ZMkQjR46UdPi9N48//rhmz57d7loAAIA1tDnczJ07V7169dKwYcP0wgsv6IUXXmi13+uvv97mk0+dOlXV1dVatGiR3G63YmNjVVhYqKioKEmS2+2Wy+Xy9G9ublZWVpb27NmjwMBAnXnmmVqyZIluueWWNp8TAABYW5vDzfTp0497o++JSE9PV3p6eqvbVq1a5bU+e/ZsZmkAAMAxtTnc/D5oAAAAdEV+/7YUAACALxFuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApfg93OTl5Sk6OlrBwcGKi4tTUVHRUfu+/vrrmjx5sk477TSFhoYqISFB7777bidWCwAAujq/hpuCggJlZGRo3rx5Ki0t1bhx45ScnCyXy9Vq/82bN2vy5MkqLCxUSUmJLr74Yl1++eUqLS3t5MoBAEBX5ddwk5OTo5SUFKWmpiomJka5ublyOBzKz89vtX9ubq7uuecejRkzRsOHD9fDDz+s4cOH6+233+7kygEAQFflt3DT0NCgkpISJSUlebUnJSWpuLi4Tcdobm5WXV2d+vXrd9Q+9fX1qq2t9VoAAIB1+S3cVFVVqampSZGRkV7tkZGRqqioaNMxnnjiCR08eFDXXnvtUftkZ2crLCzMszgcjnbVDQAAuja/31Bss9m81g3DaNHWmjVr1mjBggUqKChQ//79j9ovKytLNTU1nqWsrKzdNQMAgK4r0F8njoiIUEBAQItZmsrKyhazOb9XUFCglJQUvfrqq5o0adIx+9rtdtnt9nbXCwAAuge/zdwEBQUpLi5OTqfTq93pdCoxMfGo+61Zs0YzZszQyy+/rClTpnR0mQAAoJvx28yNJGVmZmratGmKj49XQkKCli1bJpfLpbS0NEmHLymVl5dr9erVkg4Hm+nTp+upp57Sn/70J8+sT69evRQWFua33wEAALoOv4abqVOnqrq6WosWLZLb7VZsbKwKCwsVFRUlSXK73V7vvFm6dKkaGxt122236bbbbvO033zzzVq1alVnlw8AALogv4YbSUpPT1d6enqr234fWDZu3NjxBQEAgG7N709LAQAA+BLhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWIrfw01eXp6io6MVHBysuLg4FRUVHbWv2+3WDTfcoBEjRqhHjx7KyMjovEIBAEC34NdwU1BQoIyMDM2bN0+lpaUaN26ckpOT5XK5Wu1fX1+v0047TfPmzdN5553XydUCAIDuwK/hJicnRykpKUpNTVVMTIxyc3PlcDiUn5/fav+hQ4fqqaee0vTp0xUWFtbJ1QIAgO7Ab+GmoaFBJSUlSkpK8mpPSkpScXGxz85TX1+v2tparwUAAFiX38JNVVWVmpqaFBkZ6dUeGRmpiooKn50nOztbYWFhnsXhcPjs2AAAoOvx+w3FNpvNa90wjBZt7ZGVlaWamhrPUlZW5rNjAwCArifQXyeOiIhQQEBAi1maysrKFrM57WG322W32312PAAA0LX5beYmKChIcXFxcjqdXu1Op1OJiYl+qgoAAHR3fpu5kaTMzExNmzZN8fHxSkhI0LJly+RyuZSWlibp8CWl8vJyrV692rPPjh07JEkHDhzQTz/9pB07digoKEhnn322P34CAADoYvwabqZOnarq6motWrRIbrdbsbGxKiwsVFRUlKTDL+37/TtvRo8e7fnvkpISvfzyy4qKitLevXs7s3QAANBF+TXcSFJ6errS09Nb3bZq1aoWbYZhdHBFAACgO/P701IAAAC+RLgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACW4vdwk5eXp+joaAUHBysuLk5FRUXH7L9p0ybFxcUpODhYZ5xxhp5//vlOqhQAAHQHfg03BQUFysjI0Lx581RaWqpx48YpOTlZLper1f579uzRZZddpnHjxqm0tFT33Xef7rjjDq1du7aTKwcAAF2VX8NNTk6OUlJSlJqaqpiYGOXm5srhcCg/P7/V/s8//7yGDBmi3NxcxcTEKDU1VbNmzdLjjz/eyZUDAICuKtBfJ25oaFBJSYnmzp3r1Z6UlKTi4uJW9/n444+VlJTk1XbJJZdoxYoV+u2339SzZ88W+9TX16u+vt6zXlNTI0mqra1t709oVXP9oQ45Lo6vo8b0CMbWfzpybBlX/+HvrHV1xNgeOaZhGMft67dwU1VVpaamJkVGRnq1R0ZGqqKiotV9KioqWu3f2NioqqoqDRw4sMU+2dnZWrhwYYt2h8PRjurRFYXl+rsCdBTG1poYV+vqyLGtq6tTWFjYMfv4LdwcYbPZvNYNw2jRdrz+rbUfkZWVpczMTM96c3Oz9u/fr/Dw8GOe52RTW1srh8OhsrIyhYaG+rsc+BBja12MrTUxrq0zDEN1dXUaNGjQcfv6LdxEREQoICCgxSxNZWVli9mZIwYMGNBq/8DAQIWHh7e6j91ul91u92rr27fviRducaGhofxlsijG1roYW2tiXFs63ozNEX67oTgoKEhxcXFyOp1e7U6nU4mJia3uk5CQ0KL/e++9p/j4+FbvtwEAACcfvz4tlZmZqeXLl2vlypXatWuX7rzzTrlcLqWlpUk6fElp+vTpnv5paWn6/vvvlZmZqV27dmnlypVasWKF7r77bn/9BAAA0MX49Z6bqVOnqrq6WosWLZLb7VZsbKwKCwsVFRUlSXK73V7vvImOjlZhYaHuvPNOPffccxo0aJCefvppXXXVVf76CZZht9s1f/78Fpfw0P0xttbF2FoT49p+NqMtz1QBAAB0E37//AIAAIAvEW4AAIClEG4AAIClEG4AAIClEG66mc2bN+vyyy/XoEGDZLPZ9MYbb/i7JPhAdna2xowZoz59+qh///668sor9dVXX/m7LPhAfn6+Ro0a5XkhW0JCgt555x1/lwUfy87Ols1mU0ZGhr9LgQg33c7Bgwd13nnn6dlnn+3Q8/z2228denx427Rpk2677TZt3bpVTqdTjY2NSkpK0sGDB31+Lsa2c51++ulasmSJtm3bpm3btmnChAm64oor9MUXX/j0PIyr/3z66adatmyZRo0a1SHHZ2xPgIFuS5Kxbt264/bbtWuXMXbsWMNutxsxMTGG0+n02nfPnj2GJKOgoMAYP368YbfbjZUrVxpVVVXGddddZwwePNjo1auXERsba7z88stexx4/frxx++23G3PmzDH69u1r9O/f31i6dKlx4MABY8aMGUbv3r2NM844wygsLOyAPwHrqqysNCQZmzZtOmY/xrZ7OvXUU43ly5cfdTvj2n3U1dUZw4cPN5xOpzF+/Hhjzpw5x+zP2HYOwk031pZw09TUZIwYMcKYPHmysWPHDqOoqMi44IILWv3LNHToUGPt2rXG7t27jfLycuOHH34wHnvsMaO0tNT47rvvjKefftoICAgwtm7d6jn++PHjjT59+hiLFy82vv76a2Px4sVGjx49jOTkZGPZsmXG119/bdx6661GeHi4cfDgwQ7807CWb775xpBkfP7550ftw9h2P42NjcaaNWuMoKAg44svvmi1D+PavUyfPt3IyMgwDMM4brhhbDsP4aYba0u4eeedd4zAwEDD7XZ72o72fwq5ubnHPedll11m3HXXXZ718ePHGxdeeKFnvbGx0QgJCTGmTZvmaXO73YYk4+OPP27jLzu5NTc3G5dffrnXn2trGNvu47PPPjNCQkKMgIAAIywszFi/fv1R+zKu3ceaNWuM2NhY45dffjEM4/jhhrHtPNxzYyEPP/ywevfu7VlcLpe++uorORwODRgwwNPvggsuaHX/+Ph4r/Wmpib9/e9/16hRoxQeHq7evXvrvffe8/okhiSv68wBAQEKDw/Xueee62k78pX3ysrKdv/Gk8Htt9+uzz77TGvWrPG0Mbbd24gRI7Rjxw5t3bpVt956q26++Wbt3LmTce3GysrKNGfOHL344osKDg5usZ2x9S+/flsKvpWWlqZrr73Wsz5o0CAZhiGbzdam/UNCQrzWn3jiCT355JPKzc3Vueeeq5CQEGVkZKihocGr3++/yG6z2bzajpy/ubnZ1O85Gc2ePVtvvfWWNm/erNNPP93Tzth2b0FBQRo2bJikw/9offrpp3rqqaeUnZ3NuHZTJSUlqqysVFxcnKetqalJmzdv1rPPPqt9+/Yxtn5EuLGQfv36qV+/fl5tI0eOlMvl0r59+zyJ/dNPP23T8YqKinTFFVfopptuknT4L8M333yjmJgY3xYOGYah2bNna926ddq4caOio6O9tjO21mIYhurr6xnXbmzixIn6/PPPvdpmzpypkSNH6t5771V4eLjCw8O9tjO2nYdw080cOHBA3377rWd9z5492rFjh/r166chQ4a06D958mSdeeaZuvnmm/Xoo4+qrq5O8+bNk6Tj/h/EsGHDtHbtWhUXF+vUU09VTk6OKioq+MvUAW677Ta9/PLLevPNN9WnTx9VVFRIksLCwtSrV69W92Fsu4f77rtPycnJcjgcqqur0yuvvKKNGzdqw4YNrfZnXLuHPn36KDY21qstJCRE4eHhLdqPYGw7D/fcdDPbtm3T6NGjNXr0aElSZmamRo8erQcffLDV/gEBAXrjjTd04MABjRkzRqmpqbr//vslqdXrxP/fAw88oPPPP1+XXHKJLrroIg0YMEBXXnmlT38PDsvPz1dNTY0uuugiDRw40LMUFBQcdR/GtnvYt2+fpk2bphEjRmjixIn65JNPtGHDBk2ePLnV/oyrdTG2ncdmGIbh7yLQuT766CNdeOGF+vbbb3XmmWf6uxz4EGNrTYyrdTG2HYNwcxJYt26devfureHDh+vbb7/VnDlzdOqpp2rLli3+Lg3txNhaE+NqXYxt5+Cem5NAXV2d7rnnHpWVlSkiIkKTJk3SE0884e+y4AOMrTUxrtbF2HYOZm4AAIClcEMxAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwlP8DyN1FcQKWPaEAAAAASUVORK5CYII=",
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Mocked dataset showing the precision for different n-grams\n",
+    "data = {\"1-gram\": 0.8, \"2-gram\": 0.7, \"3-gram\": 0.6, \"4-gram\": 0.5}\n",
+    "\n",
+    "# Plot the datapoints defined above\n",
+    "fig, ax = plt.subplots(1)\n",
+    "bars = ax.bar(*zip(*data.items()))\n",
+    "ax.set(ylabel=\"N-gram precision\")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### N-gram BLEU score:\n",
+    "When the n-gram precision is normalized by the brevity penalty (BP), then the exponential decay of n-grams is almost fully compensated. The BLEU score corresponds to a geometric average of this modified n-gram precision."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGdCAYAAADuR1K7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAyCklEQVR4nO3de1RU9f7/8deIMpgKKiR5GZHUDCU9Bl3wkt2ksFVZrbJMTYVOSppIZZp1SutEV8TqgFKax2+ldMJON9Kmm5fMSsJuVlpakA0SWOClIGD//nA5vzMBOhsGB3bPx1p7Leczn8/e7/GzXL367JvNMAxDAAAAFtHG3wUAAAD4EuEGAABYCuEGAABYCuEGAABYCuEGAABYCuEGAABYCuEGAABYCuEGAABYSlt/F3C81dbW6qefflKnTp1ks9n8XQ4AAPCCYRjav3+/evTooTZtjr4285cLNz/99JMcDoe/ywAAAI1QVFSkXr16HbXPXy7cdOrUSdLhv5zg4GA/VwMAALxRUVEhh8Ph/u/40fzlws2RU1HBwcGEGwAAWhlvLinhgmIAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGApfg83mZmZioyMVFBQkGJiYrRx48aj9n/uuec0ZMgQnXDCCerevbumTJmisrKy41QtAABo6fwabnJycpSSkqL58+eroKBAI0eOVEJCggoLC+vtv2nTJk2aNEmJiYn68ssv9Z///Ecff/yxkpKSjnPlAACgpfJruElPT1diYqKSkpIUFRWljIwMORwOZWVl1dt/y5Yt6tOnj2655RZFRkZqxIgRuummm7R169bjXDkAAGip/BZuqqqqlJ+fr/j4eI/2+Ph4bd68ud4xw4YN048//qi8vDwZhqG9e/fqxRdf1CWXXNLgcSorK1VRUeGxAQAA62rrrwOXlpaqpqZG4eHhHu3h4eEqLi6ud8ywYcP03HPPady4cfr9999VXV2tyy67TE888USDx0lLS9OCBQt8WvvR9Jn7+nE7Fjx9/2DDIRcA8Nfh9wuKbTabx2fDMOq0HbF9+3bdcsst+sc//qH8/HytXbtWu3fv1rRp0xrc/7x581ReXu7eioqKfFo/AABoWfy2chMWFqaAgIA6qzQlJSV1VnOOSEtL0/Dhw3X77bdLkgYPHqwOHTpo5MiRuv/++9W9e/c6Y+x2u+x2u+9/AAAAaJH8tnITGBiomJgYOZ1Oj3an06lhw4bVO+bQoUNq08az5ICAAEmHV3wAAAD8eloqNTVVTz/9tJYvX66vvvpKs2fPVmFhofs007x58zRp0iR3/0svvVRr1qxRVlaWdu3apffff1+33HKLzjzzTPXo0cNfPwMAALQgfjstJUnjxo1TWVmZFi5cKJfLpejoaOXl5SkiIkKS5HK5PJ55M3nyZO3fv19PPvmkbr31VnXu3Fnnn3++HnroIX/9BAAA0MLYjL/Y+ZyKigqFhISovLxcwcHBPt8/d0v5D3dLAYB1mfnvt9/vlgIAAPAlwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUvz7nBmhNuM3ff7jNH4AZrNwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABL4a3gAP7SeNu7//C2dzQXVm4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAICl+P3FmZmZmXrkkUfkcrk0aNAgZWRkaOTIkfX2nTx5sv7973/XaR84cKC+/PLL5i4VANCK8FJU//H3S1H9unKTk5OjlJQUzZ8/XwUFBRo5cqQSEhJUWFhYb//FixfL5XK5t6KiInXt2lVXX331ca4cAAC0VH4NN+np6UpMTFRSUpKioqKUkZEhh8OhrKysevuHhITopJNOcm9bt27VL7/8oilTphznygEAQEvlt3BTVVWl/Px8xcfHe7THx8dr8+bNXu1j2bJluvDCCxUREdFgn8rKSlVUVHhsAADAuvwWbkpLS1VTU6Pw8HCP9vDwcBUXFx9zvMvl0htvvKGkpKSj9ktLS1NISIh7czgcTaobAAC0bH6/W8pms3l8NgyjTlt9VqxYoc6dO2vs2LFH7Tdv3jyVl5e7t6KioqaUCwAAWji/3S0VFhamgICAOqs0JSUldVZz/swwDC1fvlwTJ05UYGDgUfva7XbZ7fYm1wsAAFoHv63cBAYGKiYmRk6n06Pd6XRq2LBhRx27fv16ffvtt0pMTGzOEgEAQCvk1+fcpKamauLEiYqNjVVcXJyys7NVWFioadOmSTp8SmnPnj1auXKlx7hly5bprLPOUnR0tD/KBgAALZhfw824ceNUVlamhQsXyuVyKTo6Wnl5ee67n1wuV51n3pSXlys3N1eLFy/2R8kAAKCF8/sTipOTk5WcnFzvdytWrKjTFhISokOHDjVzVQAAoLXy+91SAAAAvkS4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAltLW7ICamhqtWLFCb7/9tkpKSlRbW+vx/TvvvOOz4gAAAMwyHW5mzZqlFStW6JJLLlF0dLRsNltz1AUAANAopsPN6tWr9cILL2jMmDHNUQ8AAECTmL7mJjAwUP369WuOWgAAAJrMdLi59dZbtXjxYhmG0Rz1AAAANInp01KbNm3Su+++qzfeeEODBg1Su3btPL5fs2aNz4oDAAAwy/TKTefOnXXFFVdo1KhRCgsLU0hIiMdmVmZmpiIjIxUUFKSYmBht3LjxqP0rKys1f/58RUREyG63q2/fvlq+fLnp4wIAAGsyvXLzzDPP+OzgOTk5SklJUWZmpoYPH66lS5cqISFB27dvV+/evesdc80112jv3r1atmyZ+vXrp5KSElVXV/usJgAA0LqZDjdH/Pzzz/rmm29ks9l0yimn6MQTTzS9j/T0dCUmJiopKUmSlJGRoXXr1ikrK0tpaWl1+q9du1br16/Xrl271LVrV0lSnz59GvsTAACABZk+LXXw4EFNnTpV3bt31znnnKORI0eqR48eSkxM1KFDh7zeT1VVlfLz8xUfH+/RHh8fr82bN9c75pVXXlFsbKwefvhh9ezZU6eccopuu+02/fbbbw0ep7KyUhUVFR4bAACwLtPhJjU1VevXr9err76qX3/9Vb/++qtefvllrV+/XrfeeqvX+yktLVVNTY3Cw8M92sPDw1VcXFzvmF27dmnTpk364osv9NJLLykjI0Mvvviibr755gaPk5aW5nFNkMPh8LpGAADQ+pgON7m5uVq2bJkSEhIUHBys4OBgjRkzRk899ZRefPFF0wX8+QnHhmE0+NTj2tpa2Ww2PffcczrzzDM1ZswYpaena8WKFQ2u3sybN0/l5eXuraioyHSNAACg9TB9zc2hQ4fqrLZIUrdu3UydlgoLC1NAQECdVZqSkpJ69y9J3bt3V8+ePT3uyoqKipJhGPrxxx/Vv3//OmPsdrvsdrvXdQEAgNbN9MpNXFyc7rnnHv3+++/utt9++00LFixQXFyc1/sJDAxUTEyMnE6nR7vT6dSwYcPqHTN8+HD99NNPOnDggLttx44datOmjXr16mXylwAAACsyvXKzePFiXXzxxerVq5eGDBkim82mbdu2KSgoSOvWrTO1r9TUVE2cOFGxsbGKi4tTdna2CgsLNW3aNEmHTynt2bNHK1eulCSNHz9e9913n6ZMmaIFCxaotLRUt99+u6ZOnar27dub/SkAAMCCTIeb6Oho7dy5U88++6y+/vprGYaha6+9Vtdff73pgDFu3DiVlZVp4cKFcrlcio6OVl5eniIiIiRJLpdLhYWF7v4dO3aU0+nUzJkzFRsbq9DQUF1zzTW6//77zf4MAABgUY16zk379u114403+qSA5ORkJScn1/vdihUr6rSdeuqpdU5lAQAAHOFVuHnllVeUkJCgdu3a6ZVXXjlq38suu8wnhQEAADSGV+Fm7NixKi4uVrdu3TR27NgG+9lsNtXU1PiqNgAAANO8Cje1tbX1/hkAAKClMX0reH1+/fVXX+wGAACgyUyHm4ceekg5OTnuz1dffbW6du2qnj176tNPP/VpcQAAAGaZDjdLly51v5/J6XTqrbfe0tq1a5WQkKDbb7/d5wUCAACYYfpWcJfL5Q43r732mq655hrFx8erT58+Ouuss3xeIAAAgBmmV266dOnifvnk2rVrdeGFF0o6/MJL7pQCAAD+Znrl5sorr9T48ePVv39/lZWVKSEhQZK0bds29evXz+cFAgAAmGE63CxatEh9+vRRUVGRHn74YXXs2FHS4dNVDT1pGAAA4HgxHW7atWun2267rU57SkqKL+oBAABoEl6/AAAALIXXLwAAAEvh9QsAAMBSfPL6BQAAgJbCdLi55ZZb9Pjjj9dpf/LJJ7moGAAA+J3pcJObm6vhw4fXaR82bJhefPFFnxQFAADQWKbDTVlZmUJCQuq0BwcHq7S01CdFAQAANJbpcNOvXz+tXbu2Tvsbb7yhk08+2SdFAQAANJbph/ilpqZqxowZ+vnnn3X++edLkt5++2099thjysjI8HV9AAAAppgON1OnTlVlZaX++c9/6r777pMk9enTR1lZWZo0aZLPCwQAADDDdLiRpOnTp2v69On6+eef1b59e/f7pQAAAPytUc+5qa6u1ltvvaU1a9bIMAxJ0k8//aQDBw74tDgAAACzTK/c/PDDD7r44otVWFioyspKjR49Wp06ddLDDz+s33//XUuWLGmOOgEAALxieuVm1qxZio2N1S+//KL27du726+44gq9/fbbPi0OAADALNMrN5s2bdL777+vwMBAj/aIiAjt2bPHZ4UBAAA0humVm9ra2nrf/P3jjz+qU6dOPikKAACgsUyHm9GjR3s8z8Zms+nAgQO65557NGbMGF/WBgAAYJrp01Lp6ek6//zzNXDgQP3+++8aP368du7cqbCwMK1atao5agQAAPCa6XDTs2dPbdu2TatXr1Z+fr5qa2uVmJio66+/3uMCYwAAAH8wFW7++OMPDRgwQK+99pqmTJmiKVOmNFddAAAAjWLqmpt27dqpsrJSNputueoBAABoEtMXFM+cOVMPPfSQqqurm6MeAACAJjEdbj788EOtWbNGvXv31kUXXaQrr7zSYzMrMzNTkZGRCgoKUkxMjDZu3Nhg3/fee082m63O9vXXX5s+LgAAsCbTFxR37txZV111lU8OnpOTo5SUFGVmZmr48OFaunSpEhIStH37dvXu3bvBcd98842Cg4Pdn0888USf1AMAAFo/0+HmmWee8dnB09PTlZiYqKSkJElSRkaG1q1bp6ysLKWlpTU4rlu3burcubPP6gAAANbRqLeCS1JJSYk2btyoTZs2qaSkxPT4qqoq5efnKz4+3qM9Pj5emzdvPurYoUOHqnv37rrgggv07rvvHrVvZWWlKioqPDYAAGBdpsNNRUWFJk6cqJ49e2rUqFE655xz1LNnT02YMEHl5eVe76e0tFQ1NTUKDw/3aA8PD1dxcXG9Y7p3767s7Gzl5uZqzZo1GjBggC644AJt2LChweOkpaUpJCTEvTkcDq9rBAAArY/pcJOUlKQPP/xQr732mn799VeVl5frtdde09atW3XjjTeaLuDPt5UbhtHgreYDBgzQjTfeqNNPP11xcXHKzMzUJZdcokcffbTB/c+bN0/l5eXuraioyHSNAACg9TB9zc3rr7+udevWacSIEe62iy66SE899ZQuvvhir/cTFhamgICAOqs0JSUldVZzjubss8/Ws88+2+D3drtddrvd6/0BAIDWzfTKTWhoqEJCQuq0h4SEqEuXLl7vJzAwUDExMXI6nR7tTqdTw4YN83o/BQUF6t69u9f9AQCAtZleubnrrruUmpqqlStXukNFcXGxbr/9dt19992m9pWamqqJEycqNjZWcXFxys7OVmFhoaZNmybp8CmlPXv2aOXKlZIO303Vp08fDRo0SFVVVXr22WeVm5ur3Nxcsz8DAABYlOlwk5WVpW+//VYRERHuZ9EUFhbKbrfr559/1tKlS919P/nkk6Pua9y4cSorK9PChQvlcrkUHR2tvLw8RURESJJcLpcKCwvd/auqqnTbbbdpz549at++vQYNGqTXX39dY8aMMfszAACARZkON2PHjvVpAcnJyUpOTq73uxUrVnh8njNnjubMmePT4wMAAGsxHW7uueee5qgDAADAJxr9ED8AAICWiHADAAAshXADAAAshXADAAAshXADAAAsxfTdUoZh6MUXX9S7776rkpIS1dbWeny/Zs0anxUHAABglulwM2vWLGVnZ+u8885TeHh4gy+5BAAA8AfT4ebZZ5/VmjVreCowAABokUxfcxMSEqKTTz65OWoBAABoMtPh5t5779WCBQv022+/NUc9AAAATWL6tNTVV1+tVatWqVu3burTp4/atWvn8f2xXpYJAADQnEyHm8mTJys/P18TJkzggmIAANDimA43r7/+utatW6cRI0Y0Rz0AAABNYvqaG4fDoeDg4OaoBQAAoMlMh5vHHntMc+bM0ffff98M5QAAADSN6dNSEyZM0KFDh9S3b1+dcMIJdS4o3rdvn8+KAwAAMMt0uMnIyGiGMgAAAHzDdLi54YYbmqMOAAAAnzAdbv7Xb7/9pj/++MOjjYuNAQCAP5m+oPjgwYOaMWOGunXrpo4dO6pLly4eGwAAgD+ZDjdz5szRO++8o8zMTNntdj399NNasGCBevTooZUrVzZHjQAAAF4zfVrq1Vdf1cqVK3Xuuedq6tSpGjlypPr166eIiAg999xzuv7665ujTgAAAK+YXrnZt2+fIiMjJR2+vubIrd8jRozQhg0bfFsdAACASabDzcknn+x+gN/AgQP1wgsvSDq8otO5c2df1gYAAGCa6XAzZcoUffrpp5KkefPmua+9mT17tm6//XafFwgAAGCG6WtuZs+e7f7zeeedp6+//lpbt25V3759NWTIEJ8WBwAAYJaplZs//vhD5513nnbs2OFu6927t6688kqCDQAAaBFMhZt27drpiy++kM1ma656AAAAmsT0NTeTJk3SsmXLmqMWAACAJjN9zU1VVZWefvppOZ1OxcbGqkOHDh7fp6en+6w4AAAAs0yHmy+++EKnn366JHlceyOJ01UAAMDvTIebd999tznqAAAA8AnT19z4WmZmpiIjIxUUFKSYmBht3LjRq3Hvv/++2rZtq7/97W/NWyAAAGhVTK/cXHHFFfWefrLZbAoKClK/fv00fvx4DRgw4Jj7ysnJUUpKijIzMzV8+HAtXbpUCQkJ2r59u3r37t3guPLyck2aNEkXXHCB9u7da/YnAAAACzO9chMSEqJ33nlHn3zyiTvkFBQU6J133lF1dbVycnI0ZMgQvf/++8fcV3p6uhITE5WUlKSoqChlZGTI4XAoKyvrqONuuukmjR8/XnFxcWbLBwAAFmc63Jx00kkaP368du3apdzcXK1Zs0bfffedJkyYoL59++qrr77SDTfcoDvuuOOo+6mqqlJ+fr7i4+M92uPj47V58+YGxz3zzDP67rvvdM8993hVb2VlpSoqKjw2AABgXabDzbJly5SSkqI2bf7/0DZt2mjmzJnKzs6WzWbTjBkz9MUXXxx1P6WlpaqpqVF4eLhHe3h4uIqLi+sds3PnTs2dO1fPPfec2rb17oxaWlqaQkJC3JvD4fBqHAAAaJ1Mh5vq6mp9/fXXddq//vpr1dTUSJKCgoK8vi38z/0Mw6h3bE1NjcaPH68FCxbolFNO8breefPmqby83L0VFRV5PRYAALQ+pi8onjhxohITE3XnnXfqjDPOkM1m00cffaQHHnhAkyZNkiStX79egwYNOup+wsLCFBAQUGeVpqSkpM5qjiTt379fW7duVUFBgWbMmCFJqq2tlWEYatu2rd58802df/75dcbZ7XbZ7XazPxMAALRSpsPNokWLFB4erocffth9p1J4eLhmz57tvs4mPj5eF1988VH3ExgYqJiYGDmdTl1xxRXudqfTqcsvv7xO/+DgYH3++ecebZmZmXrnnXf04osvKjIy0uxPAQAAFmQ63AQEBGj+/PmaP3++++Lc4OBgjz5Hu437f6WmpmrixImKjY1VXFycsrOzVVhYqGnTpkk6fEppz549Wrlypdq0aaPo6GiP8d26dVNQUFCddgAA8NdlOtz8r8zMTHcQaYxx48aprKxMCxculMvlUnR0tPLy8hQRESFJcrlcKiwsbEqJAADgL6ZJTyh+4IEHtG/fviYVkJycrO+//16VlZXKz8/XOeec4/5uxYoVeu+99xoce++992rbtm1NOj4AALCWJoUbwzB8VQcAAIBP+P3dUgAAAL7UpGtutm/frh49eviqFgAAgCZrUrjhab8AAKCl8TrcREZGHvOpwzabTd99912TiwIAAGgsr8NNSkpKg999//33Wrp0qSorK31REwAAQKN5HW5mzZpVp23fvn267777lJWVpbPOOksPPfSQT4sDAAAwq1HX3Pz2229KT0/XI488oj59+mjNmjUaM2aMr2sDAAAwzVS4qamp0VNPPaUFCxYoKChITzzxhCZMmOD1G8ABAACam9fh5oUXXtBdd92l8vJy3XnnnZo+fboCAwObszYAAADTvA431157rdq3b6/rrrtOP/zwg+bOnVtvv/T0dJ8VBwAAYJbX4eacc8455q3enJ4CAAD+5nW4OdoLLAEAAFoK3i0FAAAshXADAAAshXADAAAshXADAAAshXADAAAsxau7pT777DOvdzh48OBGFwMAANBUXoWbv/3tb7LZbDIM45jPsqmpqfFJYQAAAI3h1Wmp3bt3a9euXdq9e7dyc3MVGRmpzMxMFRQUqKCgQJmZmerbt69yc3Obu14AAICj8mrlJiIiwv3nq6++Wo8//rjHW8AHDx4sh8Ohu+++W2PHjvV5kQAAAN4yfUHx559/rsjIyDrtkZGR2r59u0+KAgAAaCzT4SYqKkr333+/fv/9d3dbZWWl7r//fkVFRfm0OAAAALO8frfUEUuWLNGll14qh8OhIUOGSJI+/fRT2Ww2vfbaaz4vEAAAwAzT4ebMM8/U7t279eyzz+rrr7+WYRgaN26cxo8frw4dOjRHjQAAAF4zHW4k6YQTTtDf//53X9cCAADQZI16QvH//d//acSIEerRo4d++OEHSdKiRYv08ssv+7Q4AAAAs0yHm6ysLKWmpiohIUG//PKL+6F9Xbp0UUZGhq/rAwAAMMV0uHniiSf01FNPaf78+Wrb9v+f1YqNjdXnn3/u0+IAAADMMh1udu/eraFDh9Zpt9vtOnjwoE+KAgAAaCzT4SYyMlLbtm2r0/7GG29o4MCBvqgJAACg0UzfLXX77bfr5ptv1u+//y7DMPTRRx9p1apVSktL09NPP90cNQIAAHjN9MrNlClTdM8992jOnDk6dOiQxo8fryVLlmjx4sW69tprTReQmZmpyMhIBQUFKSYmRhs3bmyw76ZNmzR8+HCFhoaqffv2OvXUU7Vo0SLTxwQAANbVqOfc3HjjjbrxxhtVWlqq2tpadevWrVEHz8nJUUpKijIzMzV8+HAtXbpUCQkJ2r59u3r37l2nf4cOHTRjxgwNHjxYHTp00KZNm3TTTTepQ4cOPHcHAABIauRzbo4ICwtrdLCRpPT0dCUmJiopKUlRUVHKyMiQw+FQVlZWvf2HDh2q6667ToMGDVKfPn00YcIEXXTRRUdd7QEAAH8tXq3cnH766Xr77bfVpUsXDR06VDabrcG+n3zyiVcHrqqqUn5+vubOnevRHh8fr82bN3u1j4KCAm3evFn3339/g30qKytVWVnp/lxRUeHVvgEAQOvkVbi5/PLLZbfbJUljx471yYFLS0tVU1Oj8PBwj/bw8HAVFxcfdWyvXr30888/q7q6Wvfee6+SkpIa7JuWlqYFCxb4pGYAANDyeRVuunTpojZtDp/BmjJlinr16uX+3FR/XgUyDOOoK0OStHHjRh04cEBbtmzR3Llz1a9fP1133XX19p03b55SU1PdnysqKuRwOJpeOAAAaJG8Cjepqam69tprFRQUpMjISLlcriZdayMdvl4nICCgzipNSUlJndWcP4uMjJQknXbaadq7d6/uvffeBsON3W53rzoBAADr82r5pUePHsrNzdUPP/wgwzD0448/qrCwsN7NW4GBgYqJiZHT6fRodzqdGjZsmNf7MQzD45oaAADw1+bVys1dd92lmTNnasaMGbLZbDrjjDPq9DlyOunIizS9kZqaqokTJyo2NlZxcXHKzs5WYWGhpk2bJunwKaU9e/Zo5cqVkqR//etf6t27t0499VRJh5978+ijj2rmzJleHxMAAFibV+Hm73//u6677jr98MMPGjx4sN566y2FhoY2+eDjxo1TWVmZFi5cKJfLpejoaOXl5SkiIkKS5HK5PFaDamtrNW/ePO3evVtt27ZV37599eCDD+qmm25qci0AAMAavH6IX6dOnRQdHa1nnnlGw4cP99l1LMnJyUpOTq73uxUrVnh8njlzJqs0AADgqEw/ofiGG25ojjoAAAB8wqtw07VrV+3YsUNhYWHq0qXLUW/V3rdvn8+KAwAAMMurcLNo0SJ16tTJ/edjPYcGAADAX7wKN/97Kmry5MnNVQsAAECTeRVuzLyPKTg4uNHFAAAANJVX4aZz585en4oy85wbAAAAX/Mq3Lz77rvuP3///feaO3euJk+erLi4OEnSBx98oH//+99KS0trnioBAAC85FW4GTVqlPvPCxcuVHp6use7nC677DKddtppys7O5lZxAADgV6Zf7f3BBx8oNja2TntsbKw++ugjnxQFAADQWKbDjcPh0JIlS+q0L126VA6HwydFAQAANJbpJxQvWrRIV111ldatW6ezzz5bkrRlyxZ99913ys3N9XmBAAAAZpheuRkzZox27typyy67TPv27VNZWZkuv/xy7dixQ2PGjGmOGgEAALxmeuVGknr16qUHHnjA17UAAAA0WaPCza+//qply5bpq6++ks1m08CBAzV16lSFhIT4uj4AAABTTJ+W2rp1q/r27atFixZp3759Ki0tVXp6uvr27atPPvmkOWoEAADwmumVm9mzZ+uyyy7TU089pbZtDw+vrq5WUlKSUlJStGHDBp8XCQAA4C3T4Wbr1q0ewUaS2rZtqzlz5tT7/BsAAIDjyfRpqeDgYBUWFtZpLyoqUqdOnXxSFAAAQGOZDjfjxo1TYmKicnJyVFRUpB9//FGrV69WUlKSxysZAAAA/MH0aalHH31UNptNkyZNUnV1tSSpXbt2mj59uh588EGfFwgAAGCG6XATGBioxYsXKy0tTd99950Mw1C/fv10wgknNEd9AAAApjTqOTeSdMIJJ+i0007zZS0AAABN5nW4mTp1qlf9li9f3uhiAAAAmsrrcLNixQpFRERo6NChMgyjOWsCAABoNK/DzbRp07R69Wrt2rVLU6dO1YQJE9S1a9fmrA0AAMA0r28Fz8zMlMvl0h133KFXX31VDodD11xzjdatW8dKDgAAaDFMPefGbrfruuuuk9Pp1Pbt2zVo0CAlJycrIiJCBw4caK4aAQAAvGb6IX5H2Gw22Ww2GYah2tpaX9YEAADQaKbCTWVlpVatWqXRo0drwIAB+vzzz/Xkk0+qsLBQHTt2bK4aAQAAvOb1BcXJyclavXq1evfurSlTpmj16tUKDQ1tztoAAABM8zrcLFmyRL1791ZkZKTWr1+v9evX19tvzZo1PisOAADALK/DzaRJk2Sz2ZqzFgAAgCYz9RA/AACAlq7Rd0v5SmZmpiIjIxUUFKSYmBht3Lixwb5r1qzR6NGjdeKJJyo4OFhxcXFat27dcawWAAC0dH4NNzk5OUpJSdH8+fNVUFCgkSNHKiEhQYWFhfX237Bhg0aPHq28vDzl5+frvPPO06WXXqqCgoLjXDkAAGip/Bpu0tPTlZiYqKSkJEVFRSkjI0MOh0NZWVn19s/IyNCcOXN0xhlnqH///nrggQfUv39/vfrqq8e5cgAA0FL5LdxUVVUpPz9f8fHxHu3x8fHavHmzV/uora3V/v37j/qOq8rKSlVUVHhsAADAuvwWbkpLS1VTU6Pw8HCP9vDwcBUXF3u1j8cee0wHDx7UNddc02CftLQ0hYSEuDeHw9GkugEAQMvm9wuK/3x7uWEYXt1yvmrVKt17773KyclRt27dGuw3b948lZeXu7eioqIm1wwAAFour28F97WwsDAFBATUWaUpKSmps5rzZzk5OUpMTNR//vMfXXjhhUfta7fbZbfbm1wvAABoHfy2chMYGKiYmBg5nU6PdqfTqWHDhjU4btWqVZo8ebKef/55XXLJJc1dJgAAaGX8tnIjSampqZo4caJiY2MVFxen7OxsFRYWatq0aZIOn1Las2ePVq5cKelwsJk0aZIWL16ss88+273q0759e4WEhPjtdwAAgJbDr+Fm3LhxKisr08KFC+VyuRQdHa28vDxFRERIklwul8czb5YuXarq6mrdfPPNuvnmm93tN9xwA09QBgAAkvwcbqTDbxtPTk6u97s/B5b33nuv+QsCAACtmt/vlgIAAPAlwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUv4ebzMxMRUZGKigoSDExMdq4cWODfV0ul8aPH68BAwaoTZs2SklJOX6FAgCAVsGv4SYnJ0cpKSmaP3++CgoKNHLkSCUkJKiwsLDe/pWVlTrxxBM1f/58DRky5DhXCwAAWgO/hpv09HQlJiYqKSlJUVFRysjIkMPhUFZWVr39+/Tpo8WLF2vSpEkKCQk5ztUCAIDWwG/hpqqqSvn5+YqPj/doj4+P1+bNm312nMrKSlVUVHhsAADAuvwWbkpLS1VTU6Pw8HCP9vDwcBUXF/vsOGlpaQoJCXFvDofDZ/sGAAAtj98vKLbZbB6fDcOo09YU8+bNU3l5uXsrKiry2b4BAEDL09ZfBw4LC1NAQECdVZqSkpI6qzlNYbfbZbfbfbY/AADQsvlt5SYwMFAxMTFyOp0e7U6nU8OGDfNTVQAAoLXz28qNJKWmpmrixImKjY1VXFycsrOzVVhYqGnTpkk6fEppz549WrlypXvMtm3bJEkHDhzQzz//rG3btikwMFADBw70x08AAAAtjF/Dzbhx41RWVqaFCxfK5XIpOjpaeXl5ioiIkHT4oX1/fubN0KFD3X/Oz8/X888/r4iICH3//ffHs3QAANBC+TXcSFJycrKSk5Pr/W7FihV12gzDaOaKAABAa+b3u6UAAAB8iXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAsxe/hJjMzU5GRkQoKClJMTIw2btx41P7r169XTEyMgoKCdPLJJ2vJkiXHqVIAANAa+DXc5OTkKCUlRfPnz1dBQYFGjhyphIQEFRYW1tt/9+7dGjNmjEaOHKmCggLdeeeduuWWW5Sbm3ucKwcAAC2VX8NNenq6EhMTlZSUpKioKGVkZMjhcCgrK6ve/kuWLFHv3r2VkZGhqKgoJSUlaerUqXr00UePc+UAAKClauuvA1dVVSk/P19z5871aI+Pj9fmzZvrHfPBBx8oPj7eo+2iiy7SsmXL9Mcff6hdu3Z1xlRWVqqystL9uby8XJJUUVHR1J9Qr9rKQ82yXxxbc83pEcyt/zTn3DKv/sO/Wetqjrk9sk/DMI7Z12/hprS0VDU1NQoPD/doDw8PV3Fxcb1jiouL6+1fXV2t0tJSde/evc6YtLQ0LViwoE67w+FoQvVoiUIy/F0Bmgtza03Mq3U159zu379fISEhR+3jt3BzhM1m8/hsGEadtmP1r6/9iHnz5ik1NdX9uba2Vvv27VNoaOhRj/NXU1FRIYfDoaKiIgUHB/u7HPgQc2tdzK01Ma/1MwxD+/fvV48ePY7Z12/hJiwsTAEBAXVWaUpKSuqszhxx0kkn1du/bdu2Cg0NrXeM3W6X3W73aOvcuXPjC7e44OBg/jFZFHNrXcytNTGvdR1rxeYIv11QHBgYqJiYGDmdTo92p9OpYcOG1TsmLi6uTv8333xTsbGx9V5vAwAA/nr8erdUamqqnn76aS1fvlxfffWVZs+ercLCQk2bNk3S4VNKkyZNcvefNm2afvjhB6Wmpuqrr77S8uXLtWzZMt12223++gkAAKCF8es1N+PGjVNZWZkWLlwol8ul6Oho5eXlKSIiQpLkcrk8nnkTGRmpvLw8zZ49W//617/Uo0cPPf7447rqqqv89RMsw26365577qlzCg+tH3NrXcytNTGvTWczvLmnCgAAoJXw++sXAAAAfIlwAwAALIVwAwAALIVwAwAALIVw08ps2LBBl156qXr06CGbzab//ve//i4JPpCWlqYzzjhDnTp1Urdu3TR27Fh98803/i4LPpCVlaXBgwe7H8gWFxenN954w99lwcfS0tJks9mUkpLi71Igwk2rc/DgQQ0ZMkRPPvlksx7njz/+aNb9w9P69et18803a8uWLXI6naqurlZ8fLwOHjzo82Mxt8dXr1699OCDD2rr1q3aunWrzj//fF1++eX68ssvfXoc5tV/Pv74Y2VnZ2vw4MHNsn/mthEMtFqSjJdeeumY/b766itj+PDhht1uN6Kiogyn0+kxdvfu3YYkIycnxxg1apRht9uN5cuXG6Wlpca1115r9OzZ02jfvr0RHR1tPP/88x77HjVqlDFjxgxj1qxZRufOnY1u3boZS5cuNQ4cOGBMnjzZ6Nixo3HyyScbeXl5zfA3YF0lJSWGJGP9+vVH7cfctk5dunQxnn766Qa/Z15bj/379xv9+/c3nE6nMWrUKGPWrFlH7c/cHh+Em1bMm3BTU1NjDBgwwBg9erSxbds2Y+PGjcaZZ55Z7z+mPn36GLm5ucauXbuMPXv2GD/++KPxyCOPGAUFBcZ3331nPP7440ZAQICxZcsW9/5HjRpldOrUybjvvvuMHTt2GPfdd5/Rpk0bIyEhwcjOzjZ27NhhTJ8+3QgNDTUOHjzYjH8b1rJz505DkvH555832Ie5bX2qq6uNVatWGYGBgcaXX35Zbx/mtXWZNGmSkZKSYhiGccxww9weP4SbVsybcPPGG28Ybdu2NVwul7utof9TyMjIOOYxx4wZY9x6663uz6NGjTJGjBjh/lxdXW106NDBmDhxorvN5XIZkowPPvjAy1/211ZbW2tceumlHn+v9WFuW4/PPvvM6NChgxEQEGCEhIQYr7/+eoN9mdfWY9WqVUZ0dLTx22+/GYZx7HDD3B4/XHNjIQ888IA6duzo3goLC/XNN9/I4XDopJNOcvc788wz6x0fGxvr8bmmpkb//Oc/NXjwYIWGhqpjx4568803PV6JIcnjPHNAQIBCQ0N12mmnuduOvOW9pKSkyb/xr2DGjBn67LPPtGrVKncbc9u6DRgwQNu2bdOWLVs0ffp03XDDDdq+fTvz2ooVFRVp1qxZevbZZxUUFFTne+bWv/z6bin41rRp03TNNde4P/fo0UOGYchms3k1vkOHDh6fH3vsMS1atEgZGRk67bTT1KFDB6WkpKiqqsqj35/fyG6z2Tzajhy/trbW1O/5K5o5c6ZeeeUVbdiwQb169XK3M7etW2BgoPr16yfp8H+0Pv74Yy1evFhpaWnMayuVn5+vkpISxcTEuNtqamq0YcMGPfnkk9q7dy9z60eEGwvp2rWrunbt6tF26qmnqrCwUHv37nUn9o8//tir/W3cuFGXX365JkyYIOnwP4adO3cqKirKt4VDhmFo5syZeumll/Tee+8pMjLS43vm1loMw1BlZSXz2opdcMEF+vzzzz3apkyZolNPPVV33HGHQkNDFRoa6vE9c3v8EG5amQMHDujbb791f969e7e2bdumrl27qnfv3nX6jx49Wn379tUNN9yghx9+WPv379f8+fMl6Zj/B9GvXz/l5uZq8+bN6tKli9LT01VcXMw/pmZw88036/nnn9fLL7+sTp06qbi4WJIUEhKi9u3b1zuGuW0d7rzzTiUkJMjhcGj//v1avXq13nvvPa1du7be/sxr69CpUydFR0d7tHXo0EGhoaF12o9gbo8frrlpZbZu3aqhQ4dq6NChkqTU1FQNHTpU//jHP+rtHxAQoP/+9786cOCAzjjjDCUlJemuu+6SpHrPE/+vu+++W6effrouuuginXvuuTrppJM0duxYn/4eHJaVlaXy8nKde+656t69u3vLyclpcAxz2zrs3btXEydO1IABA3TBBRfoww8/1Nq1azV69Oh6+zOv1sXcHj82wzAMfxeB4+v999/XiBEj9O2336pv377+Lgc+xNxaE/NqXcxt8yDc/AW89NJL6tixo/r3769vv/1Ws2bNUpcuXbRp0yZ/l4YmYm6tiXm1Lub2+OCam7+A/fv3a86cOSoqKlJYWJguvPBCPfbYY/4uCz7A3FoT82pdzO3xwcoNAACwFC4oBgAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlvL/AEGIHigLKKERAAAAAElFTkSuQmCC",
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Mocked dataset showing the precision multiplied by the BP for different n-grams\n",
+    "data = {\"1-gram\": 0.8, \"2-gram\": 0.77, \"3-gram\": 0.74, \"4-gram\": 0.71}\n",
+    "\n",
+    "# Plot the datapoints defined above\n",
+    "fig, ax = plt.subplots(1)\n",
+    "bars = ax.bar(*zip(*data.items()))\n",
+    "ax.set(ylabel=\"Modified N-gram precision\")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 3. Example Calculations of the BLEU score\n",
+    "\n",
+    "In this example you will have a reference sentence and 2 candidate sentences. You will tokenize all sentences using the NLTK package. Then you will compare the two candidates to the reference using BLEU score.\n",
+    "\n",
+    "First you define and tokenize the sentences."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The NASA Opportunity rover is battling a massive dust storm on planet Mars. -> ['the', 'nasa', 'opportunity', 'rover', 'is', 'battling', 'a', 'massive', 'dust', 'storm', 'on', 'planet', 'mars', '.']\n",
+      "\n",
+      "\n",
+      "The Opportunity rover is combating a big sandstorm on planet Mars. -> ['the', 'opportunity', 'rover', 'is', 'combating', 'a', 'big', 'sandstorm', 'on', 'planet', 'mars', '.']\n",
+      "\n",
+      "\n",
+      "A NASA rover is fighting a massive storm on planet Mars. -> ['a', 'nasa', 'rover', 'is', 'fighting', 'a', 'massive', 'storm', 'on', 'planet', 'mars', '.']\n"
+     ]
+    }
+   ],
+   "source": [
+    "reference = \"The NASA Opportunity rover is battling a massive dust storm on planet Mars.\"\n",
+    "candidate_1 = \"The Opportunity rover is combating a big sandstorm on planet Mars.\"\n",
+    "candidate_2 = \"A NASA rover is fighting a massive storm on planet Mars.\"\n",
+    "\n",
+    "tokenized_ref = nltk.word_tokenize(reference.lower())\n",
+    "tokenized_cand_1 = nltk.word_tokenize(candidate_1.lower())\n",
+    "tokenized_cand_2 = nltk.word_tokenize(candidate_2.lower())\n",
+    "\n",
+    "print(f\"{reference} -> {tokenized_ref}\")\n",
+    "print(\"\\n\")\n",
+    "print(f\"{candidate_1} -> {tokenized_cand_1}\")\n",
+    "print(\"\\n\")\n",
+    "print(f\"{candidate_2} -> {tokenized_cand_2}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.1 Define the functions to calculate the BLEU score\n",
+    "\n",
+    "### Computing the Brevity Penalty\n",
+    "You will start by defining the function for brevity penalty according to the equation (2) in section 2.1."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def brevity_penalty(candidate, reference):\n",
+    "    \"\"\"\n",
+    "    Calculates the brevity penalty given the candidate and reference sentences.\n",
+    "    \"\"\"\n",
+    "    reference_length = len(reference)\n",
+    "    candidate_length = len(candidate)\n",
+    "\n",
+    "    if reference_length < candidate_length:\n",
+    "        BP = 1\n",
+    "    else:\n",
+    "        penalty = 1 - (reference_length / candidate_length)\n",
+    "        BP = np.exp(penalty)\n",
+    "\n",
+    "    return BP"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computing the clipped Precision\n",
+    "Next, you need to define a function to calculate the geometrically averaged clipped precision. This function calculates how many of the n-grams in the candidate sentence actually appear in the reference sentence. The clipping takes care of overcounting. For example if a certain n-gram appears five times in the candidate sentence, but only twice in the reference, the value is clipped to two."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def average_clipped_precision(candidate, reference):\n",
+    "    \"\"\"\n",
+    "    Calculates the precision given the candidate and reference sentences.\n",
+    "    \"\"\"\n",
+    "\n",
+    "    clipped_precision_score = []\n",
+    "    \n",
+    "    # Loop through values 1, 2, 3, 4. This is the length of n-grams\n",
+    "    for n_gram_length in range(1, 5):\n",
+    "        reference_n_gram_counts = Counter(ngrams(reference, n_gram_length))        \n",
+    "        candidate_n_gram_counts = Counter(ngrams(candidate, n_gram_length))                \n",
+    "\n",
+    "        total_candidate_ngrams = sum(candidate_n_gram_counts.values())       \n",
+    "        \n",
+    "        for ngram in candidate_n_gram_counts: \n",
+    "            # check if it is in the reference n-gram\n",
+    "            if ngram in reference_n_gram_counts:\n",
+    "                # if the count of the candidate n-gram is bigger than the corresponding\n",
+    "                # count in the reference n-gram, then set the count of the candidate n-gram \n",
+    "                # to be equal to the reference n-gram\n",
+    "                \n",
+    "                if candidate_n_gram_counts[ngram] > reference_n_gram_counts[ngram]: \n",
+    "                    candidate_n_gram_counts[ngram] = reference_n_gram_counts[ngram] # t\n",
+    "                                                   \n",
+    "            else:\n",
+    "                candidate_n_gram_counts[ngram] = 0 # else set the candidate n-gram equal to zero\n",
+    "\n",
+    "        clipped_candidate_ngrams = sum(candidate_n_gram_counts.values())\n",
+    "        \n",
+    "        clipped_precision_score.append(clipped_candidate_ngrams / total_candidate_ngrams)\n",
+    "    \n",
+    "    # Calculate the geometric average: take the mean of elemntwise log, then exponentiate\n",
+    "    # This is equivalent to taking the n-th root of the product as shown in equation (1) above\n",
+    "    s = np.exp(np.mean(np.log(clipped_precision_score)))\n",
+    "    \n",
+    "    return s\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computing the BLEU score\n",
+    "Finally, you can compute the BLEU score using the above two functions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def bleu_score(candidate, reference):\n",
+    "    BP = brevity_penalty(candidate, reference)    \n",
+    "    geometric_average_precision = average_clipped_precision(candidate, reference)    \n",
+    "    return BP * geometric_average_precision"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.2 Testing the functions\n",
+    "Now you can test the functions with your Example Reference and Candidates Sentences."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BLEU score of reference versus candidate 1: 27.6\n",
+      "BLEU score of reference versus candidate 2: 35.3\n"
+     ]
+    }
+   ],
+   "source": [
+    "result_candidate_1 = round(bleu_score(tokenized_cand_1, tokenized_ref) * 100, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 1: {result_candidate_1}\")\n",
+    "result_candidate_2 = round(bleu_score(tokenized_cand_2, tokenized_ref) * 100, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 2: {result_candidate_2}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.3 Comparing the Results from your Code with the Sacrebleu Library\n",
+    "Below you will do the same calculation, but using the `sacrebleu` library. Compare them with your implementation above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BLEU score of reference versus candidate 1: 27.6\n",
+      "BLEU score of reference versus candidate 2: 35.3\n"
+     ]
+    }
+   ],
+   "source": [
+    "result_candidate_1 = round(sacrebleu.sentence_bleu(candidate_1, [reference]).score, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 1: {result_candidate_1}\")\n",
+    "result_candidate_2 = round(sacrebleu.sentence_bleu(candidate_2, [reference]).score, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 2: {result_candidate_2}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 4.  BLEU computation on a corpus\n",
+    "\n",
+    "## 4.1 Loading Datasets for Evaluation Using the BLEU Score\n",
+    "\n",
+    "In this section, you will use a simple pipeline for evaluating machine translated text. You will use English to German translations generated by [Google Translate](https://translate.google.com). There are three files you will need:\n",
+    "\n",
+    "1. A source text in English. In this lab, you will use the first 1671 words of the [wmt19](http://statmt.org/wmt19/translation-task.html) evaluation dataset downloaded via SacreBLEU.\n",
+    "2. A reference translation to German of the corresponding first 1671 words from the original English text. This is also provided by SacreBLEU.\n",
+    "3. A candidate machine translation to German from the same 1671 words. This is generated by Google Translate.\n",
+    "\n",
+    "With that, you can now compare the reference and candidate translation to get the BLEU Score."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Loading the raw data\n",
+    "wmt19_src = open(\"data/wmt19_src.txt\", \"r\")\n",
+    "wmt19_src_1 = wmt19_src.read()\n",
+    "wmt19_src.close()\n",
+    "\n",
+    "wmt19_ref = open(\"data/wmt19_ref.txt\", \"r\")\n",
+    "wmt19_ref_1 = wmt19_ref.read()\n",
+    "wmt19_ref.close()\n",
+    "\n",
+    "wmt19_can = open(\"data/wmt19_can.txt\", \"r\")\n",
+    "wmt19_can_1 = wmt19_can.read()\n",
+    "wmt19_can.close()\n",
+    "\n",
+    "tokenized_corpus_src = nltk.word_tokenize(wmt19_src_1.lower())\n",
+    "tokenized_corpus_ref = nltk.word_tokenize(wmt19_ref_1.lower())\n",
+    "tokenized_corpus_cand = nltk.word_tokenize(wmt19_can_1.lower())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that you have your data loaded, you can inspect the first sentence of each dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "English source text:\n",
+      "\n",
+      "Welsh AMs worried about 'looking like muppets'\n",
+      "There is consternation among some AMs at a suggestion their title should change to MWPs (Member of the Welsh Parliament).\n",
+      " -> ['\\ufeffwelsh', 'ams', 'worried', 'about', \"'looking\", 'like', \"muppets'\", 'there', 'is', 'consternation', 'among', 'some', 'ams', 'at', 'a', 'suggestion', 'their', 'title', 'should', 'change', 'to', 'mwps', '(', 'member', 'of', 'the', 'welsh', 'parliament', ')', '.']\n",
+      "\n",
+      "\n",
+      "German reference translation:\n",
+      "\n",
+      "Walisische Ageordnete sorgen sich \"wie Dödel auszusehen\"\n",
+      "Es herrscht Bestürzung unter einigen Mitgliedern der Versammlung über einen Vorschlag, der ihren Titel zu MWPs (Mitglied der walisischen Parlament) ändern soll.\n",
+      " -> ['\\ufeffwalisische', 'ageordnete', 'sorgen', 'sich', '``', 'wie', 'dödel', 'auszusehen', \"''\", 'es', 'herrscht', 'bestürzung', 'unter', 'einigen', 'mitgliedern', 'der', 'versammlung', 'über', 'einen', 'vorschlag', ',', 'der', 'ihren', 'titel', 'zu', 'mwps', '(', 'mitglied', 'der', 'walisischen', 'parlament', ')', 'ändern', 'soll', '.']\n",
+      "\n",
+      "\n",
+      "German machine translation:\n",
+      "\n",
+      "Walisische AMs machten sich Sorgen, dass sie wie Muppets aussehen könnten\n",
+      "Einige AMs sind bestürzt über den Vorschlag, ihren Titel in MWPs (Mitglied des walisischen Parlaments) zu ändern.\n",
+      "Es ist aufg -> ['walisische', 'ams', 'machten', 'sich', 'sorgen', ',', 'dass', 'sie', 'wie', 'muppets', 'aussehen', 'könnten', 'einige', 'ams', 'sind', 'bestürzt', 'über', 'den', 'vorschlag', ',', 'ihren', 'titel', 'in', 'mwps', '(', 'mitglied', 'des', 'walisischen', 'parlaments']\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"English source text:\\n\")\n",
+    "print(f\"{wmt19_src_1[0:170]} -> {tokenized_corpus_src[0:30]}\\n\\n\")\n",
+    "print(\"German reference translation:\\n\")\n",
+    "print(f\"{wmt19_ref_1[0:219]} -> {tokenized_corpus_ref[0:35]}\\n\\n\")\n",
+    "print(\"German machine translation:\\n\")\n",
+    "print(f\"{wmt19_can_1[0:199]} -> {tokenized_corpus_cand[0:29]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And lastly, you can calculate the BLEU score of the translation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BLEU score of the reference versus candidate translation: 43.2\n"
+     ]
+    }
+   ],
+   "source": [
+    "result =  round(sacrebleu.sentence_bleu(wmt19_can_1, [wmt19_ref_1]).score, 1)\n",
+    "print(f\"BLEU score of the reference versus candidate translation: {result}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4.2 BLEU Score Interpretation on a Corpus\n",
+    "The table below (taken from [here](https://cloud.google.com/translate/automl/docs/evaluate)) shows the typical values of BLEU score. You can see that the translation above is of high quality according to this table and in comparison to the given reference sentence. (*if you see \"Hard to get the gist\", please open your workspace, delete `wmt19_can.txt` and get the latest version via the Lab Help button*)\n",
+    "\n",
+    "|Score      | Interpretation                                                |\n",
+    "|:---------:|:-------------------------------------------------------------:|\n",
+    "| < 10      | Almost useless                                                |\n",
+    "| 10 - 19   | Hard to get the gist                                          |\n",
+    "| 20 - 29   | The gist is clear, but has significant grammatical errors     |\n",
+    "| 30 - 40   | Understandable to good translations                           |\n",
+    "| 40 - 50   | High quality translations                                     |\n",
+    "| 50 - 60   | Very high quality, adequate, and fluent translations          |\n",
+    "| > 60      | Quality often better than human                               |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/.ipynb_checkpoints/C4W1_QKV_Attention-checkpoint.ipynb

@@ -0,0 +1,270 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "707052ae",
+   "metadata": {},
+   "source": [
+    "# Scaled Dot-Product Attention: Ungraded Lab\n",
+    "\n",
+    "The 2017 paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762) introduced the Transformer model and scaled dot-product attention, sometimes also called QKV (**Q**ueries, **K**eys, **V**alues) attention. Since then, Transformers have come to dominate large-scale natural language applications. Scaled dot-product attention can be used to improve seq2seq models as well. In this ungraded lab, you'll implement a simplified version of scaled dot-product attention and replicate word alignment between English and French, as shown in [Bhadanau, et al. (2014)](https://arxiv.org/abs/1409.0473).\n",
+    "\n",
+    "The Transformer model learns how to align words in different languages. You won't be training any weights here, so instead you will use [pre-trained aligned word embeddings from here](https://fasttext.cc/docs/en/aligned-vectors.html). Run the cell below to load the embeddings and set up the rest of the notebook.\n",
+    "\n",
+    "This is a practice notebook, where you can train writing your code. All of the solutions are provided at the end of the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "aa4d9f30",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import the libraries\n",
+    "import pickle\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "# Load the word2int dictionaries\n",
+    "with open(\"./data/word2int_en.pkl\", \"rb\") as f:\n",
+    "    en_words = pickle.load(f)\n",
+    "    \n",
+    "with open(\"./data/word2int_fr.pkl\", \"rb\") as f:\n",
+    "    fr_words = pickle.load(f)\n",
+    "\n",
+    "# Load the word embeddings\n",
+    "en_embeddings = np.load(\"./data/embeddings_en.npz\")[\"embeddings\"]\n",
+    "fr_embeddings = np.load(\"./data/embeddings_fr.npz\")[\"embeddings\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a6914081",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define some helper functions\n",
+    "\n",
+    "def tokenize(sentence, token_mapping):\n",
+    "    tokenized = []\n",
+    "    \n",
+    "    for word in sentence.lower().split(\" \"):\n",
+    "        try:\n",
+    "            tokenized.append(token_mapping[word])\n",
+    "        except KeyError:\n",
+    "            # Using -1 to indicate an unknown word\n",
+    "            tokenized.append(-1)\n",
+    "        \n",
+    "    return tokenized\n",
+    "\n",
+    "\n",
+    "def embed(tokens, embeddings):\n",
+    "    embed_size = embeddings.shape[1]\n",
+    "    \n",
+    "    output = np.zeros((len(tokens), embed_size))\n",
+    "    for i, token in enumerate(tokens):\n",
+    "        if token == -1:\n",
+    "            output[i] = np.zeros((1, embed_size))\n",
+    "        else:\n",
+    "            output[i] = embeddings[token]\n",
+    "            \n",
+    "    return output"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6153d4b2",
+   "metadata": {},
+   "source": [
+    "The scaled-dot product attention consists of two matrix multiplications and a softmax scaling as shown in the diagram below from [Vaswani, et al. (2017)](https://arxiv.org/abs/1706.03762). It takes three input matrices, the queries, keys, and values.\n",
+    "\n",
+    "![scaled-dot product attention diagram](./images/attention.png)\n",
+    "\n",
+    "Mathematically, this is expressed as\n",
+    "\n",
+    "$$ \n",
+    "\\large \\mathrm{Attention}\\left(Q, K, V\\right) = \\mathrm{softmax}\\left(\\frac{QK^{\\top}}{\\sqrt{d_k}}\\right)V\n",
+    "$$\n",
+    "\n",
+    "where $Q$, $K$, and $V$ are the queries, keys, and values matrices respectively, and $d_k$ is the dimension of the keys. In practice, Q, K, and V all have the same dimensions. This form of attention is faster and more space-efficient than what you implemented before since it consists of only matrix multiplications instead of a learned feed-forward layer.\n",
+    "\n",
+    "Conceptually, the first matrix multiplication is a measure of the similarity between the queries and the keys. This is transformed into weights using the softmax function. These weights are then applied to the values with the second matrix multiplication resulting in output attention vectors. Typically, decoder states are used as the queries while encoder states are the keys and values.\n",
+    "\n",
+    "### Exercise 1\n",
+    "Implement the softmax function with Numpy and use it to calculate the weights from the queries and keys. Assume the queries and keys are 2D arrays (matrices). Note that since the dot-product of Q and K will be a matrix, you'll need to calculate softmax over a specific axis. See the end of the notebook for solutions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3932b927",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def softmax(x, axis=0):    \n",
+    "    \"\"\" Calculate softmax function for an array x\n",
+    "\n",
+    "        axis=0 calculates softmax across rows which means each column sums to 1 \n",
+    "        axis=1 calculates softmax across columns which means each row sums to 1\n",
+    "    \"\"\"\n",
+    "    # Replace pass with your code.\n",
+    "    pass\n",
+    "\n",
+    "def calculate_weights(queries, keys):\n",
+    "    \"\"\" Calculate the weights for scaled dot-product attention\"\"\"\n",
+    "    # Replace None with your code.\n",
+    "    dot = None\n",
+    "    weights = softmax(dot, axis=1)\n",
+    "    \n",
+    "    assert weights.sum(axis=1)[0] == 1, \"Each row in weights must sum to 1\"\n",
+    "    \n",
+    "    # Replace pass with your code.\n",
+    "    pass"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "51f47450",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Tokenize example sentences in English and French, then get their embeddings\n",
+    "sentence_en = \"The agreement on the European Economic Area was signed in August 1992 .\"\n",
+    "tokenized_en = tokenize(sentence_en, en_words)\n",
+    "embedded_en = embed(tokenized_en, en_embeddings)\n",
+    "\n",
+    "sentence_fr = \"L accord sur la zone économique européenne a été signé en août 1992 .\"\n",
+    "tokenized_fr = tokenize(sentence_fr, fr_words)\n",
+    "embedded_fr = embed(tokenized_fr, fr_embeddings)\n",
+    "\n",
+    "# These weights indicate alignment between words in English and French\n",
+    "alignment = calculate_weights(embedded_fr, embedded_en)\n",
+    "\n",
+    "# Visualize weights to check for alignment\n",
+    "fig, ax = plt.subplots(figsize=(7,7))\n",
+    "ax.imshow(alignment, cmap='gray')\n",
+    "ax.xaxis.tick_top()\n",
+    "ax.set_xticks(np.arange(alignment.shape[1]))\n",
+    "ax.set_xticklabels(sentence_en.split(\" \"), rotation=90, size=16);\n",
+    "ax.set_yticks(np.arange(alignment.shape[0]));\n",
+    "ax.set_yticklabels(sentence_fr.split(\" \"), size=16);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d634f0ec",
+   "metadata": {},
+   "source": [
+    "If you implemented the weights calculations correctly, the alignment matrix should look like this:\n",
+    "\n",
+    "![alignment visualization](./images/alignment.png)\n",
+    "\n",
+    "This is a demonstration of alignment where the model has learned which words in English correspond to words in French. For example, the words *signed* and *signé* have a large weight because they have the same meaning. Typically, these alignments are learned using linear layers in the model, but you've used pre-trained embeddings here.\n",
+    "\n",
+    "### Exercise 2\n",
+    "Complete the implementation of scaled dot-product attention using your `calculate_weights` function (ignore the mask)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fbfc157e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def attention_qkv(queries, keys, values):\n",
+    "    \"\"\" Calculate scaled dot-product attention from queries, keys, and values matrices \"\"\"\n",
+    "    \n",
+    "    # Replace pass with your code.\n",
+    "    pass\n",
+    "\n",
+    "\n",
+    "attention_qkv_result = attention_qkv(embedded_fr, embedded_en, embedded_en)\n",
+    "\n",
+    "print(f\"The shape of the attention_qkv function is {attention_qkv_result.shape}\")\n",
+    "print(f\"Some elements of the attention_qkv function are \\n{attention_qkv_result[0:2,:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f98335f0",
+   "metadata": {},
+   "source": [
+    "**Expected output**\n",
+    "\n",
+    "The shape of the attention_qkv function is `(14, 300)`\n",
+    "\n",
+    "Some elements of the attention_qkv function are \n",
+    "```python\n",
+    "[[-0.04039161 -0.00275749  0.00389873  0.04842744 -0.02472726  0.01435613\n",
+    "  -0.00370253 -0.0619686  -0.00206159  0.01615228]\n",
+    " [-0.04083253 -0.00245985  0.00409068  0.04830341 -0.02479128  0.01447497\n",
+    "  -0.00355203 -0.06196036 -0.00241327  0.01582606]]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f87131fb",
+   "metadata": {},
+   "source": [
+    "## Solutions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8470a024",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "def softmax(x, axis=0):\n",
+    "    \"\"\" Calculate softmax function for an array x\n",
+    "    \n",
+    "        axis=0 calculates softmax across rows which means each column sums to 1 \n",
+    "        axis=1 calculates softmax across columns which means each row sums to 1\n",
+    "    \"\"\"\n",
+    "    y = np.exp(x) \n",
+    "    return y / np.expand_dims(np.sum(y, axis=axis), axis)\n",
+    "\n",
+    "def calculate_weights(queries, keys):\n",
+    "    \"\"\" Calculate the weights for scaled dot-product attention\"\"\"\n",
+    "    dot = np.matmul(queries, keys.T)/np.sqrt(keys.shape[1])\n",
+    "    weights = softmax(dot, axis=1)\n",
+    "    \n",
+    "    assert weights.sum(axis=1)[0] == 1, \"Each row in weights must sum to 1\"\n",
+    "    \n",
+    "    return weights\n",
+    "\n",
+    "def attention_qkv(queries, keys, values):\n",
+    "    \"\"\" Calculate scaled dot-product attention from queries, keys, and values matrices \"\"\"\n",
+    "    weights = calculate_weights(queries, keys)\n",
+    "    return np.matmul(weights, values)\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/C4W1_Basic_Attention.ipynb

@@ -0,0 +1,324 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9c74bac5",
+   "metadata": {},
+   "source": [
+    "# Basic Attention Operation: Ungraded Lab\n",
+    "\n",
+    "As you've learned, attention allows a seq2seq decoder to use information from each encoder step instead of just the final encoder hidden state. In the attention operation, the encoder outputs are weighted based on the decoder hidden state, then combined into one context vector. This vector is then used as input to the decoder to predict the next output step.\n",
+    "\n",
+    "In this ungraded lab, you'll implement a basic attention operation as described in [Bhadanau, et al (2014)](https://arxiv.org/abs/1409.0473) using Numpy.\n",
+    "\n",
+    "This is a practice notebook, where you can train writing your code. All of the solutions are provided at the end of the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "a5288920",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import the libraries and define the functions you will need for this lab\n",
+    "import numpy as np\n",
+    "\n",
+    "def softmax(x, axis=0):\n",
+    "    \"\"\" Calculate softmax function for an array x along specified axis\n",
+    "    \n",
+    "        axis=0 calculates softmax across rows which means each column sums to 1 \n",
+    "        axis=1 calculates softmax across columns which means each row sums to 1\n",
+    "    \"\"\"\n",
+    "    return np.exp(x) / np.expand_dims(np.sum(np.exp(x), axis=axis), axis)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9a6e0293",
+   "metadata": {},
+   "source": [
+    "## 1: Calculating alignment scores\n",
+    "\n",
+    "The first step is to calculate the alignment scores. This is a measure of similarity between the decoder hidden state and each encoder hidden state. From the paper, this operation looks like\n",
+    "\n",
+    "$$\n",
+    "\\large e_{ij} = v_a^\\top \\tanh{\\left(W_a s_{i-1} + U_a h_j\\right)}\n",
+    "$$\n",
+    "\n",
+    "where $W_a \\in \\mathbb{R}^{n\\times m}$, $U_a \\in \\mathbb{R}^{n \\times m}$, and $v_a \\in \\mathbb{R}^m$\n",
+    "are the weight matrices and $n$ is the hidden state size. In practice, this is implemented as a feedforward neural network with two layers, where $m$ is the size of the layers in the alignment network. It looks something like:\n",
+    "\n",
+    "![alignment model](./images/alignment_model_3.jpg)\n",
+    "\n",
+    "Here $h_j$ are the encoder hidden states for each input step $j$ and $s_{i - 1}$ is the decoder hidden state of the previous step. The first layer corresponds to $W_a$ and $U_a$, while the second layer corresponds to $v_a$.\n",
+    "\n",
+    "To implement this, first concatenate the encoder and decoder hidden states to produce an array with size $K \\times 2n$ where $K$ is the number of encoder states/steps. For this, use `np.concatenate` ([docs](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html)). Note that there is only one decoder state so you'll need to reshape it to successfully concatenate the arrays. The easiest way is to use `decoder_state.repeat` ([docs](https://numpy.org/doc/stable/reference/generated/numpy.repeat.html#numpy.repeat)) to match the hidden state array size.\n",
+    "\n",
+    "Then, apply the first layer as a matrix multiplication between the weights and the concatenated input. Use the tanh function to get the activations. Finally, compute the matrix multiplication of the second layer weights and the activations. This returns the alignment scores."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "72857076",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "hidden_size = 16\n",
+    "attention_size = 10\n",
+    "input_length = 5\n",
+    "\n",
+    "np.random.seed(42)\n",
+    "\n",
+    "# Synthetic vectors used to test\n",
+    "encoder_states = np.random.randn(input_length, hidden_size)\n",
+    "decoder_state = np.random.randn(1, hidden_size)\n",
+    "\n",
+    "# Weights for the neural network, these are typically learned through training\n",
+    "# Use these in the alignment function below as the layer weights\n",
+    "layer_1 = np.random.randn(2 * hidden_size, attention_size)\n",
+    "layer_2 = np.random.randn(attention_size, 1)\n",
+    "\n",
+    "# Implement this function. Replace None with your code. Solution at the bottom of the notebook\n",
+    "def alignment(encoder_states, decoder_state):\n",
+    "    # First, concatenate the encoder states and the decoder state\n",
+    "    inputs = np.concatenate((encoder_states, np.repeat(decoder_state, input_length, axis=0)),axis=1)\n",
+    "    assert inputs.shape == (input_length, 2 * hidden_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the concatenated inputs and layer_1, with tanh activation\n",
+    "    activations = np.tanh(np.dot(inputs,layer_1))\n",
+    "    assert activations.shape == (input_length, attention_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the activations with layer_2. Remember that you don't need tanh here\n",
+    "    scores = np.dot(activations, layer_2)\n",
+    "    assert scores.shape == (input_length, 1)\n",
+    "    \n",
+    "    return scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "fb638355",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[[4.35790943]\n",
+      " [5.92373433]\n",
+      " [4.18673175]\n",
+      " [2.11437202]\n",
+      " [0.95767155]]\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Run this to test your alignment function\n",
+    "scores = alignment(encoder_states, decoder_state)\n",
+    "print(scores)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f26aae76",
+   "metadata": {},
+   "source": [
+    "If you implemented the function correctly, you should get these scores:\n",
+    "\n",
+    "```python\n",
+    "[[4.35790943]\n",
+    " [5.92373433]\n",
+    " [4.18673175]\n",
+    " [2.11437202]\n",
+    " [0.95767155]]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "58b8cfa9",
+   "metadata": {},
+   "source": [
+    "## 2: Turning alignment into weights\n",
+    "\n",
+    "The next step is to calculate the weights from the alignment scores. These weights determine the encoder outputs that are the most important for the decoder output. These weights should be between 0 and 1. You can use the softmax function (which is already implemented above) to get these weights from the attention scores. Pass the attention scores vector to the softmax function to get the weights. Mathematically,\n",
+    "\n",
+    "$$\n",
+    "\\large \\alpha_{ij} = \\frac{\\exp{\\left(e_{ij}\\right)}}{\\sum_{k=1}^K \\exp{\\left(e_{ik}\\right)}}\n",
+    "$$\n",
+    "\n",
+    "\n",
+    "\n",
+    "## 3: Weight the encoder output vectors and sum\n",
+    "\n",
+    "The weights tell you the importance of each input word with respect to the decoder state. In this step, you use the weights to modulate the magnitude of the encoder vectors. Words with little importance will be scaled down relative to important words. Multiply each encoder vector by its respective weight to get the alignment vectors, then sum up the weighted alignment vectors to get the context vector. Mathematically,\n",
+    "\n",
+    "$$\n",
+    "\\large c_i = \\sum_{j=1}^K\\alpha_{ij} h_{j}\n",
+    "$$\n",
+    "\n",
+    "Implement these steps in the `attention` function below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "4546cbb5",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[-0.63514569  0.04917298 -0.43930867 -0.9268003   1.01903919 -0.43181409\n",
+      "  0.13365099 -0.84746874 -0.37572203  0.18279832 -0.90452701  0.17872958\n",
+      " -0.58015282 -0.58294027 -0.75457577  1.32985756]\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Implement this function. Replace None with your code.\n",
+    "def attention(encoder_states, decoder_state):\n",
+    "    \"\"\" Example function that calculates attention, returns the context vector \n",
+    "    \n",
+    "        Arguments:\n",
+    "        encoder_vectors: NxM numpy array, where N is the number of vectors and M is the vector length\n",
+    "        decoder_vector: 1xM numpy array, M is the vector length, much be the same M as encoder_vectors\n",
+    "    \"\"\" \n",
+    "    \n",
+    "    # First, calculate the alignment scores\n",
+    "    scores = alignment(encoder_states, decoder_state)\n",
+    "    \n",
+    "    # Then take the softmax of the alignment scores to get a weight distribution\n",
+    "    weights = softmax(scores)\n",
+    "    \n",
+    "    # Multiply each encoder state by its respective weight\n",
+    "    weighted_scores = weights * encoder_states\n",
+    "    \n",
+    "    # Sum up weighted alignment vectors to get the context vector and return it\n",
+    "    context = np.sum(weighted_scores,axis=0)\n",
+    "    return context\n",
+    "\n",
+    "context_vector = attention(encoder_states, decoder_state)\n",
+    "print(context_vector)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d9f3df4",
+   "metadata": {},
+   "source": [
+    "If you implemented the `attention` function correctly, the context vector should be\n",
+    "\n",
+    "```python\n",
+    "[-0.63514569  0.04917298 -0.43930867 -0.9268003   1.01903919 -0.43181409\n",
+    "  0.13365099 -0.84746874 -0.37572203  0.18279832 -0.90452701  0.17872958\n",
+    " -0.58015282 -0.58294027 -0.75457577  1.32985756]\n",
+    "```\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4210899c",
+   "metadata": {},
+   "source": [
+    "## See below for solutions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3ba0d629",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "# Solution\n",
+    "def alignment(encoder_states, decoder_state):\n",
+    "    # First, concatenate the encoder states and the decoder state.\n",
+    "    inputs = np.concatenate((encoder_states, decoder_state.repeat(input_length, axis=0)), axis=1)\n",
+    "    assert inputs.shape == (input_length, 2*hidden_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the concatenated inputs and the first layer, with tanh activation\n",
+    "    activations = np.tanh(np.matmul(inputs, layer_1))\n",
+    "    assert activations.shape == (input_length, attention_size)\n",
+    "    \n",
+    "    # Matrix multiplication of the activations with the second layer. Remember that you don't need tanh here\n",
+    "    scores = np.matmul(activations, layer_2)\n",
+    "    assert scores.shape == (input_length, 1)\n",
+    "    \n",
+    "    return scores\n",
+    "\n",
+    "# Run this to test your alignment function\n",
+    "scores = alignment(encoder_states, decoder_state)\n",
+    "print(scores)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f80faecb",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "# Solution\n",
+    "def attention(encoder_states, decoder_state):\n",
+    "    \"\"\" Example function that calculates attention, returns the context vector \n",
+    "    \n",
+    "        Arguments:\n",
+    "        encoder_vectors: NxM numpy array, where N is the number of vectors and M is the vector length\n",
+    "        decoder_vector: 1xM numpy array, M is the vector length, much be the same M as encoder_vectors\n",
+    "    \"\"\" \n",
+    "    \n",
+    "    # First, calculate the dot product of each encoder vector with the decoder vector\n",
+    "    scores = alignment(encoder_states, decoder_state)\n",
+    "    \n",
+    "    # Then take the softmax of those scores to get a weight distribution\n",
+    "    weights = softmax(scores)\n",
+    "    \n",
+    "    # Multiply each encoder state by its respective weight\n",
+    "    weighted_scores = encoder_states * weights\n",
+    "    \n",
+    "    # Sum up the weights encoder states\n",
+    "    context = np.sum(weighted_scores, axis=0)\n",
+    "    \n",
+    "    return context\n",
+    "\n",
+    "context_vector = attention(encoder_states, decoder_state)\n",
+    "print(context_vector)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16a6caa8",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/C4W1_Bleu_Score.ipynb

@@ -0,0 +1,585 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Calculating the Bilingual Evaluation Understudy (BLEU) score: Ungraded Lab"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this ungraded lab, you will implement a popular metric for evaluating the quality of machine-translated text: the BLEU score proposed by Kishore Papineni, et al. in their 2002 paper [\"BLEU: a Method for Automatic Evaluation of Machine Translation\"](https://www.aclweb.org/anthology/P02-1040.pdf). The BLEU score works by comparing a \"candidate\" text to one or more \"reference\" texts. The score is higher the better the result. In the following sections you will calculate this value using your own implementation as well as using functions from a library."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 1.  Importing the Libraries\n",
+    "\n",
+    "You will start by importing the Python libraries. First, you will implement your own version of the BLEU Score using NumPy. To verify that your implementation is correct, you will compare the results with those generated by the [SacreBLEU library](https://github.com/mjpost/sacrebleu). This package provides hassle-free computation of shareable, comparable, and reproducible BLEU scores. It also knows all the standard test sets and handles downloading, processing, and tokenization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
+      "[nltk_data]   Package punkt is already up-to-date!\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Requirement already satisfied: sacrebleu in /opt/conda/lib/python3.10/site-packages (2.3.1)\n",
+      "Requirement already satisfied: portalocker in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (2.8.2)\n",
+      "Requirement already satisfied: regex in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (2023.10.3)\n",
+      "Requirement already satisfied: tabulate>=0.8.9 in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (0.9.0)\n",
+      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (1.24.3)\n",
+      "Requirement already satisfied: colorama in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (0.4.6)\n",
+      "Requirement already satisfied: lxml in /opt/conda/lib/python3.10/site-packages (from sacrebleu) (4.9.3)\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np                  # import numpy to make numerical computations.\n",
+    "import nltk                         # import NLTK to handle simple NL tasks like tokenization.\n",
+    "nltk.download(\"punkt\")\n",
+    "from nltk.util import ngrams\n",
+    "from collections import Counter     # import a counter.\n",
+    "!pip3 install 'sacrebleu'           # install the sacrebleu package.\n",
+    "import sacrebleu                    # import sacrebleu in order compute the BLEU score.\n",
+    "import matplotlib.pyplot as plt     # import pyplot in order to make some illustrations."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 2.  BLEU score\n",
+    "\n",
+    "## 2.1 Definitions and formulas\n",
+    "\n",
+    "You have seen how to calculate the BLEU score in this week's lectures. Formally, you can express the BLEU score as:\n",
+    "\n",
+    "$$BLEU = BP\\times\\Bigl(\\prod_{i=1}^{n}precision_i\\Bigr)^{(1/n)}.\\tag{1}$$\n",
+    "\n",
+    "\n",
+    "The BLEU score depends on the $BP$, which stands for Brevity Penalty, and the weighted geometric mean precision for different lengths of n-grams, both of which are described below. The product runs from $i=1$ to $i=n$ to account for 1-grams to n-grams and the exponent of $1/n$ is there to calculate the geometrical average. In this notebook, you will use $n=4$\n",
+    "\n",
+    "The **Brevity Penalty** is defined as an exponential decay:\n",
+    "\n",
+    "$$BP = min\\Bigl(1, e^{(1-({len(ref)}/{len(cand)}))}\\Bigr),\\tag{2}$$\n",
+    "\n",
+    "where ${len(ref)}$ and ${len(cand)}$ refer to the length or count of words in the reference and candidate translations. The brevity penalty helps to handle very short translations. \n",
+    "\n",
+    "The **precision** is defined as :\n",
+    "\n",
+    "$$precision_i = \\frac {\\sum_{s_i \\in{cand}}min\\Bigl(C(s_i, cand), C(s_i, ref)\\Bigr)}{\\sum_{s_i \\in{cand}} C(s_i, cand)}.\\tag{3}$$\n",
+    "\n",
+    "The sum goes over all the i-grams $s_i$ in the candidate sentence $cand$. $C(s_i, cand)$ and $C(s_i, ref)$ are the counts of the i-grams in the candidate and reference sentences respectively. So the sum counts all the n-grams in the candidate sentence that also appear in the reference sentence, but only counts them as many times as they appear in the reference sentence and not more. This is then divided by the total number of i-grams in the candidate sentence."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2.2 Visualizing the BLEU score\n",
+    "\n",
+    "### Brevity Penalty:\n",
+    "The brevity penalty penalizes generated translations that are shorter than the reference sentence. It compensates for the fact that the BLEU score has no recall term."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABcvElEQVR4nO3deVwU9eMG8Gd2WW4BEUEEBLxRPBCVwEytxLRMy1LTvLPwzLM0y6vMX5rmrZlXlleKV2UmWZ5oCoIniiIKKoqgcsq1+/n9Ye7XlcNdBAaW5/167eslw8zOswPMPs5+ZkYSQggQERERGQmF3AGIiIiIShLLDRERERkVlhsiIiIyKiw3REREZFRYboiIiMiosNwQERGRUWG5ISIiIqNiIneAsqbRaHDr1i1UqVIFkiTJHYeIiIj0IIRAWloaatasCYWi6GMzla7c3Lp1C25ubnLHICIiomKIj4+Hq6trkfNUunJTpUoVAI82jo2NjcxpiIiISB+pqalwc3PTvo8XpdKVm8cfRdnY2LDcEBERVTD6DCnhgGIiIiIyKiw3REREZFRYboiIiMiosNwQERGRUWG5ISIiIqPCckNERERGheWGiIiIjArLDRERERkVlhsiIiIyKiw3REREZFRkLTeHDh1C165dUbNmTUiShJ07dz5zmYMHD8LX1xfm5uaoXbs2VqxYUfpBiYiIqMKQtdxkZGSgWbNmWLJkiV7zx8bGokuXLmjbti0iIiLw2WefYfTo0QgODi7lpERERFRRyHrjzM6dO6Nz5856z79ixQrUqlULCxYsAAB4eXkhLCwM3377LXr06FFKKfWj1ggkpDyUNQM9Hycbc6iU/KSWiKiiq1B3BT927BgCAwN1pnXq1AmrV69Gbm4uVCpVvmWys7ORnZ2t/To1NbVUsiVnZOPFb/4pleemslHDxhxL+/rA191e7ihERPQcKlS5uX37NpycnHSmOTk5IS8vD0lJSXB2ds63zOzZszFjxowyyWdmwv/1V1RqjcDt1Cz0XnkcU99ohPdfcIckSXLHIiKiYqhQ5QZAvjccIUSB0x+bPHkyxo0bp/06NTUVbm5uJZ7LsYo5Ln2l/0dsVL5kZOfhk21n8PvZBHyx6zwi4h9g9ttNYGailDsaEREZqEIdaqhRowZu376tMy0xMREmJiaoVq1agcuYmZnBxsZG50H0NCszEyzp44MpXbygkIDtp27imz8uyR2LiIiKoUKVG39/f4SEhOhM27dvH1q2bFngeBsiQ0iShKEv1cby930BAGtDYxERd1/mVEREZChZy016ejoiIyMRGRkJ4NGp3pGRkYiLiwPw6COl/v37a+cPCgrC9evXMW7cOERFRWHNmjVYvXo1JkyYIEd8MlKdGtfAWz4uEAKYFHwWOXkauSMREZEBZC03YWFh8PHxgY+PDwBg3Lhx8PHxwdSpUwEACQkJ2qIDAJ6entizZw8OHDiA5s2b48svv8SiRYtkPw2cjM8XbzSCvZUpLt1Jw/cHY+SOQ0REBpDE4xG5lURqaipsbW2RkpLC8TdUpF2RN/Hx5kiYKhXY8/GLqOtYRe5IRESVliHv3xVqzA1RWXqzWU10aFAdOWoNJgWfhUZTqf4fQERUYbHcEBVCkiR89VYTWJkqEXb9Pn7+97rckYiISA8sN0RFcLGzwKTODQEA3/xxETfuZ8qciIiInoXlhugZ+vq5o5VHVWTkqPHZjnOoZMPUiIgqHJYbomdQKCT8X4+mMDVR4FD0XWw/dVPuSEREVASWGyI91KlujbGv1gcAzPztAhLTsmROREREhWG5IdLT0Lae8HaxQcrDXEzbdV7uOEREVAiWGyI9mSgVmNOjGUwUEv44dxt7zibIHYmIiArAckNkgEY1bTC8fR0AwNRd53AvI0fmRERE9DSWGyIDjXi5Luo7WSMpPQczf+XHU0RE5Q3LDZGBzEyUmPNOMygkYGfkLeyPuiN3JCIiegLLDVExNHezw9C2tQEAn+04i5SHuTInIiKix1huiIppbMf68HSwwp3UbMz6/YLccYiI6D8sN0TFZK5SYs47TSFJwC9hN3DgUqLckYiICCw3RM+llYc9BgV4AgAmbz+L1Cx+PEVEJDeWG6LnNLFTA7hXs0RCSha+/j1K7jhERJUeyw3Rc7IwVWLuO80gScDmk/E4FH1X7khERJUayw1RCWjtaY8B/h4AgEnBZ/jxFBGRjFhuiErIJ689+njqVkoWZv3Gj6eIiOTCckNUQixNTbQfT20Ji8c/PHuKiEgWLDdEJai15//OnpoUfAYpmfx4ioiorLHcEJWwiZ0aaC/uN/M3XtyPiKissdwQlTALUyW+fffRxf2CT93AXxd47ykiorLEckNUCnzd7bX3npq84yzuZ+TInIiIqPJguSEqJeM61kddR2vcTcvG1N3n5Y5DRFRpsNwQlRJzlRLz3m0GpULCr6dv4fczCXJHIiKqFFhuiEpRMzc7DG9fBwDw+c6zuJuWLXMiIiLjx3JDVMpGvVwPXs42uJ+Ziyk7zkIIIXckIiKjxnJDVMpMTRSY924zqJQS9l24g+2nbsodiYjIqLHcEJWBRjVtMObV+gCA6bvP49aDhzInIiIyXiw3RGXko5dqw6eWHdKy8zBx22loNPx4ioioNLDcEJURE6UC83s2h7lKgaNXkrH+2DW5IxERGSWWG6Iy5Olghc+6eAEAZv9xETF302VORERkfFhuiMrY+37uaFvPAdl5Goz75TTy1Bq5IxERGRWWG6IyplBImPNOU1QxN8Hp+AdY+k+M3JGIiIwKyw2RDJxtLfBlN28AwKK/L+N0/AN5AxERGRGWGyKZdGteE683dYZaIzD2l0g8zFHLHYmIyCiw3BDJRJIkzOruDccqZrh6NwPf7L0odyQiIqPAckMkIztLU8x9txkAYF3oNRyKvitzIiKiio/lhkhm7epXR39/dwDAxG2n8SAzR+ZEREQVG8sNUTkwubMXajtY4U5qNqbsOMebaxIRPQeWG6JywMJUiQW9m8NEIeH3swnYEcGbaxIRFZfs5WbZsmXw9PSEubk5fH19cfjw4SLnX7p0Kby8vGBhYYEGDRpg/fr1ZZSUqHQ1dbXDmFfrAQCm7jqP+HuZMiciIqqYZC03W7ZswZgxYzBlyhRERESgbdu26Ny5M+Li4gqcf/ny5Zg8eTKmT5+O8+fPY8aMGRgxYgR+/fXXMk5OVDqGta+Llu5VkZ6dh/G/nIaaN9ckIjKYJGT8cN/Pzw8tWrTA8uXLtdO8vLzQvXt3zJ49O9/8AQEBaNOmDebOnaudNmbMGISFheHIkSN6rTM1NRW2trZISUmBjY3N878IohIWfy8TnRceRnp2Hj55rQGGt68rdyQiItkZ8v4t25GbnJwchIeHIzAwUGd6YGAgQkNDC1wmOzsb5ubmOtMsLCxw4sQJ5ObmFrpMamqqzoOoPHOzt8T0NxsDAObvi8bZGykyJyIiqlhkKzdJSUlQq9VwcnLSme7k5ITbt28XuEynTp2watUqhIeHQwiBsLAwrFmzBrm5uUhKSipwmdmzZ8PW1lb7cHNzK/HXQlTSerRwQZcmNZCnEfh4SwQyc/LkjkREVGHIPqBYkiSdr4UQ+aY99sUXX6Bz58544YUXoFKp0K1bNwwcOBAAoFQqC1xm8uTJSElJ0T7i4+NLND9RaZAkCV+/1QQ1bMxx9W4Gvvo9Su5IREQVhmzlxsHBAUqlMt9RmsTExHxHcx6zsLDAmjVrkJmZiWvXriEuLg4eHh6oUqUKHBwcClzGzMwMNjY2Og+iisDO0hTzezaDJAEb/43DvvMFH9EkIiJdspUbU1NT+Pr6IiQkRGd6SEgIAgICilxWpVLB1dUVSqUSmzdvxhtvvAGFQvaDUEQlLqCuAz5sWxsA8GnwGdxJzZI5ERFR+SdrIxg3bhxWrVqFNWvWICoqCmPHjkVcXByCgoIAPPpIqX///tr5o6Oj8fPPP+Py5cs4ceIEevfujXPnzuHrr7+W6yUQlbrxgQ3QuKYN7mfmYsLW09Dw9HAioiKZyLnyXr16ITk5GTNnzkRCQgK8vb2xZ88euLs/us9OQkKCzjVv1Go15s2bh0uXLkGlUqFDhw4IDQ2Fh4eHTK+AqPSZmiiwsLcP3lh8GIcvJ2HN0Vh88N/RHCIiyk/W69zIgde5oYpqw7/XMWXHOaiUEnYMbwNvF1u5IxERlZkKcZ0bIjJMn9a1ENjICblqgdGbeXo4EVFhWG6IKghJkvBNj6ZwsjHD1bsZmPnrBbkjERGVSyw3RBVIVStTfNerOSQJ2HwyHnvOJsgdiYio3GG5IapgAuo4YFi7OgCAScFncPPBQ5kTERGVLyw3RBXQ2I710czNDqlZeRi7OZJ3DyciegLLDVEFpFIqsKh3c1ibmeDEtXtY/PdluSMREZUbLDdEFZR7NSt82f3R3cMX7b+Mf68my5yIiKh8YLkhqsDe8nHF2y1coBHAmC2RuJ+RI3ckIiLZsdwQVXBfdvOGp4MVElKy8EnwGVSy63ISEeXDckNUwVmZmWDxez5QKSWEXLiDn45flzsSEZGsWG6IjIC3iy0mdfYCAHz1exQu3EqVORERkXxYboiMxOA2HniloSNy8jQYuekUMrJ5ewYiqpxYboiMhCRJmPtuM9SwMcfVuxmYuuu83JGIiGTBckNkROytTLGwd3MoJCD41A0Eh9+QOxIRUZljuSEyMn61q2HMq/UBAF/sOoeYu+kyJyIiKlssN0RGaESHugioUw2ZOWqM2HAKWblquSMREZUZlhsiI6RUSFjQqzmqWZni4u00zPo9Su5IRERlhuWGyEg52phjfq/mAICfjl/H72cS5A1ERFRGWG6IjFi7+tUxrH0dAMCnwWdwLSlD5kRERKWP5YbIyI3vWB8t3asiPTsPIzZy/A0RGT+WGyIjZ6JUYHEfH1S1VOH8rVR8vYfjb4jIuLHcEFUCzrYW2vE3649x/A0RGTeWG6JKokMDRwzn+BsiqgRYbogqkXEd66OVx6PxN8N5/RsiMlIsN0SViIlSgcXvtYC9lSkuJKRi5m8X5I5ERFTiWG6IKpkatuZY0Ks5JAnY+G8cdkbclDsSEVGJYrkhqoReql8do16uBwCYvP0sLt9JkzkREVHJYbkhqqQ+fqUeXqzrgIe5agzfcAqZOXlyRyIiKhEsN0SVlFIhYUHv5nCsYobLiemYsuMchBByxyIiem4sN0SVmIO1GZb0aQGlQsKOiJvYdCJe7khERM+N5YaokmvtaY+JnRoAAKbvPo8zNx7IG4iI6Dmx3BARPnqpNjo2ckKOWoNhP5/Cg8wcuSMRERUbyw0RQZIkfPtuM7hXs8TNBw8xdkskNBqOvyGiionlhogAALYWKizv6wszEwX+uXQXS/+5InckIqJiYbkhIq1GNW3wVXdvAMD8v6Jx+PJdmRMRERmO5YaIdLzb0g29W7lBCODjzZG49eCh3JGIiAzCckNE+Ux/szG8XWxwLyMHwzacQnYeb7BJRBUHyw0R5WOuUmJ5X1/YWqhwOv4BZv7KG2wSUcXBckNEBXKzt8TC3o9usLnh3zhsDeMF/oioYmC5IaJCtW/giLGv1gcATNl5DudupsiciIjo2VhuiKhIIzvUxcsNHZGTp0HQz+G4n8EL/BFR+cZyQ0RFUigkfNezOWrZW+LG/Yf4eEsk1LzAHxGVYyw3RPRMtpYqrHjfF+YqBQ5F38WCv6LljkREVCjZy82yZcvg6ekJc3Nz+Pr64vDhw0XOv2HDBjRr1gyWlpZwdnbGoEGDkJycXEZpiSqvRjVt8H9vNwUALP77Cv48f1vmREREBZO13GzZsgVjxozBlClTEBERgbZt26Jz586Ii4srcP4jR46gf//+GDJkCM6fP4+tW7fi5MmT+OCDD8o4OVHl1N3HBYPaeAAAxv9yGlcS0+UNRERUAFnLzfz58zFkyBB88MEH8PLywoIFC+Dm5obly5cXOP/x48fh4eGB0aNHw9PTEy+++CI++ugjhIWFFbqO7OxspKam6jyIqPg+6+IFP097pGfn4cOfwpCWlSt3JCIiHbKVm5ycHISHhyMwMFBnemBgIEJDQwtcJiAgADdu3MCePXsghMCdO3ewbds2vP7664WuZ/bs2bC1tdU+3NzcSvR1EFU2KqUCS/u2gLOtOa7ezcC4X07zDuJEVK7IVm6SkpKgVqvh5OSkM93JyQm3bxf8WX5AQAA2bNiAXr16wdTUFDVq1ICdnR0WL15c6HomT56MlJQU7SM+nhciI3peDtZmWP6+L0yVCoRcuMM7iBNRuSL7gGJJknS+FkLkm/bYhQsXMHr0aEydOhXh4eHYu3cvYmNjERQUVOjzm5mZwcbGRudBRM+vuZudzh3E/754R+ZERESPyFZuHBwcoFQq8x2lSUxMzHc057HZs2ejTZs2mDhxIpo2bYpOnTph2bJlWLNmDRISEsoiNhE9oWcrN/T1q/XoDuKbIhFzlwOMiUh+spUbU1NT+Pr6IiQkRGd6SEgIAgICClwmMzMTCoVuZKVSCeDRER8iKnvTujZGK4+qSMvOw9D1YUjlAGMikpmsH0uNGzcOq1atwpo1axAVFYWxY8ciLi5O+zHT5MmT0b9/f+38Xbt2xfbt27F8+XJcvXoVR48exejRo9G6dWvUrFlTrpdBVKmZmiiwrK8vatj8N8B4SyQHGBORrEzkXHmvXr2QnJyMmTNnIiEhAd7e3tizZw/c3d0BAAkJCTrXvBk4cCDS0tKwZMkSjB8/HnZ2dnj55ZfxzTffyPUSiAhA9SpmWNnfF++sOIa/ohKx4K9ojAtsIHcsIqqkJFHJPs9JTU2Fra0tUlJSOLiYqIQFh9/A+K2nAQAr3m+B17ydZU5ERMbCkPdv2c+WIiLj0cPXFYPbeAIAxv1yGpdup8mciIgqI5YbIipRn3VpiIA61ZCZo8bQ9WG4n5EjdyQiqmRYboioRJkoFVjapwXc7C0Qdy8TwzecQq5aI3csIqpEDC437du3x/r16/Hw4cPSyENERqCqlSlW9W8FK1Mljl1Nxle/XZA7EhFVIgaXG19fX3zyySeoUaMGhg4diuPHj5dGLiKq4BrUqILvejUHAPx47Do2/htX9AJERCXE4HIzb9483Lx5E+vXr8fdu3fx0ksvoVGjRvj2229x5w4vv05E/xPYuAYmBNYHAEzddQ4nYu/JnIiIKoNijblRKpXo1q0bdu7ciZs3b6JPnz744osv4Obmhu7du+Pvv/8u6ZxEVEGN6FAXbzR1Rp5GYNjP4bhxP1PuSERk5J5rQPGJEycwdepUfPvtt3B0dMTkyZPh6OiIrl27YsKECSWVkYgqMEmSMPedZmhc0wbJGTkYuj4cGdl5csciIiNmcLlJTEzEvHnz4O3tjbZt2+Lu3bvYvHkzrl27hhkzZmDlypXYtWsXVqxYURp5iagCsjBV4of+LeFgbYaohFSM5S0aiKgUGVxuXF1dsWrVKgwYMAA3btzAtm3b8Nprr0GSJO08rVu3RqtWrUo0KBFVbDXtLPB9P1+YKhXYd+EO5oVckjsSERkpg2+/cPjwYbRt27a08pQ63n6BSF47Im5g7JZHt2j4rlczvOXjKnMiIqoISvX2C9OmTcODBw8KXOnLL79s6NMRUSXzlo8rhrWvAwD4NPgsTsXdlzkRERkbg8vNwYMHkZOT/3LqWVlZOHz4cImEIiLjNjGwATo2ckJOngYfrg/HzQe8KCgRlRwTfWc8c+YMAEAIgQsXLuD27dva76nVauzduxcuLi4ln5CIjI5CIWFBr+bosTwUF2+n4YMfw7AtyB9WZnrvkoiICqX3mBuFQqEdNFzQIhYWFli8eDEGDx5csglLGMfcEJUfN+5novvSo0hKz0HHRk74/n1fKBTSsxckokrHkPdvvcvN9evXIYRA7dq1ceLECVSvXl37PVNTUzg6OkKpVD5f8jLAckNUvoRfv4f3fvgXOXkafPRSbUzu4iV3JCIqhwx5/9b7GLC7uzsAQKPh3X2JqOT4uttj7jtN8fHmSHx/6Co8HazQu3UtuWMRUQWmV7nZvXu33k/45ptvFjsMEVVO3Zq7IOZuBhbtv4zPd55DLXtLBNR1kDsWEVVQen0spVDod1KVJElQq9XPHao08WMpovJJCIHRmyPx6+lbsDE3wY4RbVCnurXcsYionCjx69xoNBq9HuW92BBR+fXoHlRN4VPLDqlZeRiy7iTuZ+S/7AQR0bM8140ziYhKkrlKiZX9WsLFzgLXkjPx0c/hyMnjOD8iMozBt18AgIyMDBw8eBBxcXH5Lug3evToEgtXGvixFFH5d+l2GnosD0V6dh7e9nHBvJ7NdO5fR0SVT6mcCv5YREQEunTpgszMTGRkZMDe3h5JSUmwtLSEo6Mjrl69+lzhSxvLDVHFcDD6LgavOwm1RmDsq/Xx8av15I5ERDIq1XtLjR07Fl27dsW9e/dgYWGB48eP4/r16/D19cW3335b7NBERE9qV786vuzmDQD47q9o7Ii4IXMiIqooDC43kZGRGD9+PJRKJZRKJbKzs+Hm5oY5c+bgs88+K42MRFRJ9fGrhY9eqg0A+HTbWfx7NVnmRERUERhcblQqlfazbycnJ8TFxQEAbG1ttf8mIiopn77WEJ29ayBHrcGHP4Uj5m663JGIqJwzuNz4+PggLCwMANChQwdMnToVGzZswJgxY9CkSZMSD0hElZtCIeG7Xs3R3M0OKQ9zMXjdSdzjKeJEVASDy83XX38NZ2dnAMCXX36JatWqYdiwYUhMTMTKlStLPCARkblKiVUDWsK1qgWuJ2di6PowZOXyulpEVLBinQpekfFsKaKK60piGt5eForUrDx0aVIDS95rwbuIE1USpXq2FBGRXOo6VsHK/i1hqlRgz9nb+HpPlNyRiKgcMrjc3LlzB/369UPNmjVhYmKiPWvq8YOIqDS9ULsa5r7bFACw6kgs1h6NlTkREZU3et0V/EkDBw5EXFwcvvjiCzg7O/OqoURU5ro1d8HNBw8xZ+8lzPztApxtLfCadw25YxFROWHwmJsqVarg8OHDaN68eSlFKl0cc0NkHIQQmLLzHDb+GwczEwU2ffgCWtSqKncsIiolpTrmxs3NDZVsDDIRlUOSJGHmm43xckNHZOdp8MGPYbiWlCF3LCIqBwwuNwsWLMCkSZNw7dq1UohDRKQ/E6UCi9/zQRMXW9zLyMGAtSeQlJ4tdywikpnBH0tVrVoVmZmZyMvLg6WlJVQqlc737927V6IBSxo/liIyPolpWXh7WShu3H+IZq622Dj0BViZGTykkIjKMUPevw3+61+wYEFxcxERlQrHKub4cXBr9FgeitM3UjBi4yn80L8lVEpe7YKoMuJF/IjIaIRfv4++q44jK1eDni1d8U2Ppjyjk8hIlPpF/GJiYvD555/jvffeQ2JiIgBg7969OH/+fHGejoioRPi6V8Xi91pAIQG/hN3Ad39dljsSEcnA4HJz8OBBNGnSBP/++y+2b9+O9PRHd+g9c+YMpk2bVuIBiYgM0bGRE77q/ugmvov2X8aGf6/LnIiIyprB5WbSpEn46quvEBISAlNTU+30Dh064NixYyUajoioOPr41cLoV+oBAL7YeQ4hF+7InIiIypLB5ebs2bN466238k2vXr06kpOTSyQUEdHzGvtqPfRq6QaNAEZuPIWwa+X7TE4iKjkGlxs7OzskJCTkmx4REQEXFxeDAyxbtgyenp4wNzeHr68vDh8+XOi8AwcOhCRJ+R6NGzc2eL1EZNwkScKst7y1F/kb8mMYou+kyR2LiMqAweWmT58++PTTT3H79m1IkgSNRoOjR49iwoQJ6N+/v0HPtWXLFowZMwZTpkxBREQE2rZti86dOyMuLq7A+RcuXIiEhATtIz4+Hvb29nj33XcNfRlEVAmYKBVY2qcFWtSyQ8rDXPRffQI3HzyUOxYRlTKDTwXPzc3FwIEDsXnzZgghYGJiArVajT59+mDdunUG3Rncz88PLVq0wPLly7XTvLy80L17d8yePfuZy+/cuRNvv/02YmNj4e7uXuA82dnZyM7+3xVLU1NT4ebmxlPBiSqRB5k5eHfFMVxOTEed6lbYGhQAeyvTZy9IROWGIaeCF/s6N1evXsWpU6eg0Wjg4+ODevXqGbR8Tk4OLC0tsXXrVp0xPB9//DEiIyNx8ODBZz5H165dkZ2djX379hU6z/Tp0zFjxox801luiCqXhJSH6LEsFLdSstDczQ4bh/rB0pRXMSaqKErlOjcajQZz585FmzZt0Lp1a6xatQpvvPEGevbsaXCxAYCkpCSo1Wo4OTnpTHdycsLt27efuXxCQgL++OMPfPDBB0XON3nyZKSkpGgf8fHxBmcloorP2dYC64e0hp2lCpHxDzB8wynkqjVyxyKiUqB3ufnmm28wadIkWFlZwdnZGfPnz8fo0aOfO8DTVw8VQuh1RdF169bBzs4O3bt3L3I+MzMz2NjY6DyIqHKq61gFawa2grlKgQOX7uKTbWeg0VSqi7QTVQp6l5t169Zh8eLF2LdvH3bt2oWdO3di/fr1KO7dGxwcHKBUKvMdpUlMTMx3NOdpQgisWbMG/fr107nWDhHRs7SoVRXL+/pCqZCwI+Imvvo9qtj7MSIqn/QuN9evX8cbb7yh/bpTp04QQuDWrVvFWrGpqSl8fX0REhKiMz0kJAQBAQFFLnvw4EFcuXIFQ4YMKda6iahy69DQEXPfaQoAWHM0Fkv+viJzIiIqSXqXm5ycHFhYWGi/liQJpqamOmciGWrcuHFYtWoV1qxZg6ioKIwdOxZxcXEICgoC8Gi8TEGnl69evRp+fn7w9vYu9rqJqHJ7u4Urpr7RCAAwLyQaPx3nbRqIjIVBpwp88cUXsLS01H6dk5ODWbNmwdbWVjtt/vz5ej9fr169kJycjJkzZyIhIQHe3t7Ys2eP9rTuhISEfNe8SUlJQXBwMBYuXGhIdCKifAa/6IkHmTlY9PcVTN11DjbmJujW3PCLkRJR+aL3qeDt27d/5kBfSZLw999/l0iw0mLIqWREZPyEEJi2+zzWH7sOE4WEHwa0RIcGjnLHIqKnlMl1bioqlhsieppGIzBmSyR2n74Fc5UCPw/xQ0sPe7ljEdETSuU6N0RExkqhkDCvZzO0b1AdWbkaDFp3Ehdupcodi4iKieWGiAiASqnA8r6+aOleFWlZeei/5gRikzLkjkVExcByQ0T0HwtTJVYPbAUvZxskpWfj/VX/8kabRBUQyw0R0RNsLVT4aUhr1K5uhZsPHqLfqn9xN634l7wgorLHckNE9BQHazP8PMQPLnYWuJqUgX6r/0VKZq7csYhITwaXGw8PD8ycOTPf9WeIiIxJTTsLbPjAD9WrmOHi7TQMXHcCGdl5csciIj0YXG7Gjx+PXbt2oXbt2ujYsSM2b978XFcpJiIqrzwcrPDTkNawtVAhIu4Bhq4PQ1auWu5YRPQMBpebUaNGITw8HOHh4WjUqBFGjx4NZ2dnjBw5EqdOnSqNjEREsmlYwwY/Dm4NK1MlQmOSMXLjKeSqNXLHIqIiFHvMTbNmzbBw4ULcvHkT06ZNw6pVq9CqVSs0a9YMa9as4V12ichoNHezw6oBrWBmosBfUYkY98tpqDXcxxGVV8UuN7m5ufjll1/w5ptvYvz48WjZsiVWrVqFnj17YsqUKejbt29J5iQikpV/nWpY8b4vVEoJv56+hU+Dz0DDgkNULhl040wAOHXqFNauXYtNmzZBqVSiX79++O6779CwYUPtPIGBgXjppZdKNCgRkdw6NHTEot4+GLkpAtvCb8BcpcCX3byfed89IipbBpebVq1aoWPHjli+fDm6d+8OlUqVb55GjRqhd+/eJRKQiKg86dzEGfPVGozZEomfj8fBzESJz1/3YsEhKkcMLjdXr16Fu7t7kfNYWVlh7dq1xQ5FRFSedWvuguxcDT4JPoPVR2JhoVJiQqcGcsciov8YPOamQ4cOSE5Ozjf9wYMHqF27domEIiIq73q2csPMbo0BAEv+uYIlf1+WORERPWZwubl27RrU6vzXecjOzsbNmzdLJBQRUUXQ398Dn3V5NN7w233RWHX4qsyJiAgw4GOp3bt3a//9559/wtbWVvu1Wq3G/v374eHhUaLhiIjKuw9fqoOsXA3mh0Tjq9+jYGqiQH9/D7ljEVVqepeb7t27AwAkScKAAQN0vqdSqeDh4YF58+aVaDgioopg1Mt1kZWrxrIDMZi66zyUCgl9/Yoem0hEpUfvcqPRPLoip6enJ06ePAkHB4dSC0VEVJFIkoSJnRogTyOw8tBVTNlxDiYKCb1a1ZI7GlGlZPDZUrGxsaWRg4ioQpMkCZM7N0SeWmDN0VhM2n4WSoUC7/i6yh2NqNLRq9wsWrQIH374IczNzbFo0aIi5x09enSJBCMiqmgkScIXb3hBrdHgx2PXMXHbaSgVwFs+LDhEZUkSetwEytPTE2FhYahWrRo8PT0LfzJJwtWr5ftsgdTUVNja2iIlJQU2NjZyxyEiIySEwOc7z2HDv3FQSMCC3j54s1lNuWMRVWiGvH/rdeTmyY+i+LEUEVHRJEnCl928odYIbD4Zj7FbIqGUJLze1FnuaESVgsHXuTl48GBp5CAiMioKhYSv32qCd3xdodYIfLw5An+cTZA7FlGlYHC56dixI2rVqoVJkybh7NmzpZGJiMgoKBQSvunRFG/7uCBPIzByEwsOUVkwuNzcunULn3zyCQ4fPoxmzZqhadOmmDNnDm7cuFEa+YiIKjSlQsLcd5vhbR8XqP8rOHtYcIhKlV4DigsTGxuLjRs3YtOmTbh48SJeeukl/P333yWZr8RxQDERyUGtEZi49TS2R9yEUiFhUW8fjsEhMoAh79/PVW6AR7de+OOPP/DFF1/gzJkzBd53qjxhuSEiuag1AhO3ncb2U48KzsLezfFGU55FRaQPQ96/Df5Y6rGjR49i+PDhcHZ2Rp8+fdC4cWP89ttvxX06IiKjp1RImPtOM/Ro8XiQcSR+PX1L7lhERsfgKxR/9tln2LRpE27duoVXX30VCxYsQPfu3WFpaVka+YiIjIpSIWHOO00hScC28BsYsyUSANCV18EhKjEGl5sDBw5gwoQJ6NWrF+8vRURUDMr/zqKSAGwNv4GPN0dAIwS6NXeROxqRUTC43ISGhpZGDiKiSuVxwQEeFZyxWyKRqxa8FxVRCSjWmJuffvoJbdq0Qc2aNXH9+nUAwIIFC7Br164SDUdEZMweXwfnvdZu0Ahg4rbT2HwiTu5YRBWeweVm+fLlGDduHLp06YIHDx5oz46ys7PDggULSjofEZFRUygkzOreBP393SEEMGn7Wfx07JrcsYgqNIPLzeLFi/HDDz9gypQpUCqV2uktW7bkFYuJiIpBoZAw483GGPLioxsTf7HrPFYf4X38iIrL4HITGxsLHx+ffNPNzMyQkZFRIqGIiCobSZLw+eteGNa+DgDgy98uYPmBGJlTEVVMBpcbT09PREZG5pv+xx9/oFGjRiWRiYioUpIkCZ90aoCPX6kHAPhm70Us2n9Z5lREFY/BZ0tNnDgRI0aMQFZWFoQQOHHiBDZt2oTZs2dj1apVpZGRiKjSkCQJYzvWh6mJAnP/vIT5IdHIydNgfGB9SJIkdzyiCsHgcjNo0CDk5eXhk08+QWZmJvr06QMXFxcsXLgQvXv3Lo2MRESVzogOdaFSSvh6z0Us+ecKHuaq8fnrXiw4RHowqNzk5eVhw4YN6Nq1K4YOHYqkpCRoNBo4OjqWVj4iokrrw5fqwMxEiWm7Hw0wzsxR46vu3lAqWHCIimLQmBsTExMMGzYM2dnZAAAHBwcWGyKiUjQgwANz3mkKhQRsOhGH8b9EIk+tkTsWUblm8IBiPz8/RERElEYWIiIqQM+WbljY2wcmCgk7I29hxMZTyM5Tyx2LqNwyuNwMHz4c48ePx5IlS3Ds2DGcOXNG52GoZcuWwdPTE+bm5vD19cXhw4eLnD87OxtTpkyBu7s7zMzMUKdOHaxZs8bg9RIRVSRdm9XEivd9YWqiwJ/n7+DD9eF4mMOCQ1QQSQghDFlAocjfhyRJghACkiRpr1isjy1btqBfv35YtmwZ2rRpg++//x6rVq3ChQsXUKtWrQKX6datG+7cuYOvvvoKdevWRWJiIvLy8hAQEKDXOlNTU2Fra4uUlBTY2NjonZWIqDw4cjkJQ9eH4WGuGn6e9lg9sBWszQw+N4SowjHk/dvgcvP4XlKFcXd31/u5/Pz80KJFCyxfvlw7zcvLC927d8fs2bPzzb9371707t0bV69ehb29vf6hn8ByQ0QVXdi1exi09iTSsvPQ3M0OPw5qDVtLldyxiEqVIe/fBn8s5e7uXuRDXzk5OQgPD0dgYKDO9MDAwELvPL579260bNkSc+bMgYuLC+rXr48JEybg4cOHha4nOzsbqampOg8iooqspYc9Ng59AXaWKkTGP0CvlceQmJYldyyicsPgcpOcnKz9d3x8PKZOnYqJEyc+c6zM05KSkqBWq+Hk5KQz3cnJCbdv3y5wmatXr+LIkSM4d+4cduzYgQULFmDbtm0YMWJEoeuZPXs2bG1ttQ83NzeDchIRlUdNXG2x5UN/OFYxw8XbaXh3xTHE38uUOxZRuaB3uTl79iw8PDzg6OiIhg0bIjIyEq1atcJ3332HlStXokOHDti5c6fBAZ6+INXjsTsF0Wg0kCQJGzZsQOvWrdGlSxfMnz8f69atK/TozeTJk5GSkqJ9xMfHG5yRiKg8alCjCrYFBcDN3gLXkzPxzopQXL6TJncsItnpXW4++eQTNGnSBAcPHkT79u3xxhtvoEuXLkhJScH9+/fx0Ucf4f/+7//0XrGDgwOUSmW+ozSJiYn5juY85uzsDBcXF9ja2mqneXl5QQiBGzduFLiMmZkZbGxsdB5ERMaiVjVLbAsKQAOnKriTmo13vz+G0/EP5I5FJCu9y83Jkycxa9YsvPjii/j2229x69YtDB8+HAqFAgqFAqNGjcLFixf1XrGpqSl8fX0REhKiMz0kJKTQM5/atGmDW7duIT09XTstOjoaCoUCrq6ueq+biMiYONmYY8tHL6C5mx0eZOaizw/HERqTJHcsItnoXW7u3buHGjVqAACsra1hZWWlc8ZS1apVkZZm2OHQcePGYdWqVVizZg2ioqIwduxYxMXFISgoCMCjj5T69++vnb9Pnz6oVq0aBg0ahAsXLuDQoUOYOHEiBg8eDAsLC4PWTURkTOwsTbHhAz+0qVsNGTlqDFx7EvvOFzx+kcjYGTSg+OmxMM97A7devXphwYIFmDlzJpo3b45Dhw5hz5492rOuEhISEBcXp53f2toaISEhePDgAVq2bIm+ffuia9euWLRo0XPlICIyBlZmJlgzsBU6NXZCTp4GwzacQnB4wR/ZExkzva9zo1Ao0LlzZ5iZmQEAfv31V7z88suwsrIC8OiU67179xp0ET858Do3RGTs8tQaTNp+Ftv+KzZfvNEIQ170lDkV0fMplYv4DRo0SK+Vr127Vq/55MJyQ0SVgUYjMGtPFFYfiQUADGtfB590avDcR9yJ5FKqVyiu6FhuiKiyEEJg+cEYzNl7CQDQs6Urvn6rCUyUBl/ijEh2pXqFYiIiqhgkScLw9nXxTY8mUEjAL2E3EPTzKWTllu/hA0TPi+WGiMjI9WpVC9/3awkzEwX+irqDfqv/RUpmrtyxiEoNyw0RUSXQsZETfhrihyrmJjh57T56fn8Mt1N4PyoyTiw3RESVRGtPe2wNenQ/qkt30tBjeShi7qY/e0GiCoblhoioEmlYwwbBwwJQ28EKNx88xLsrjiGSt2sgI8NyQ0RUybjZW2JrkD+autriXkYO3lt5HH9fvCN3LKISw3JDRFQJVbM2w8ahL6BtPQc8zFVj6PpwbDkZ9+wFiSoAlhsiokrK+r/bNfRo4Qq1RuDT4LP4LiQalezyZ2SEWG6IiCoxlVKBb99tilEv1wUALNx/GZOCzyJPrZE5GVHxsdwQEVVykiRhfGADzHrLGwoJ2BIWj6Hrw5CZkyd3NKJiYbkhIiIAQF8/d3zfryXMVQr8c+kueq88jqT0bLljERmM5YaIiLQ6NnLCxqEvoKqlCmdupKDH8lBcS8qQOxaRQVhuiIhIR4taVRE8LABu9ha4npyJt5eHIvz6fbljEemN5YaIiPKpXd0a24e1QROXR9fC6fPDcew5myB3LCK9sNwQEVGBqlcxw+YPX8CrXo7IztNg+IZT+P5gDE8Vp3KP5YaIiAplZWaC7/u1xMAADwDA7D8u4vOd53iqOJVrLDdERFQkpULC9DcbY+objSBJwIZ/4zB0fRjSs3mqOJVPLDdERKSXwS96YsX7vtpTxXuuOIbbKVlyxyLKh+WGiIj01qlxDWz+0B8O1qa4kJCK7kuP4sKtVLljEelguSEiIoM0d7PDjuFtUNfRGrdTs/DuilAcuJQodywiLZYbIiIymJu9JYKDAuBfuxoyctQYvO4k1h+7JncsIgAsN0REVEy2lir8OLg1erRwhUYAU3edx7RdPJOK5MdyQ0RExWZq8uiu4p++1hAA8OOx6xj8YxhSs3JlTkaVGcsNERE9F0mSMKx9Hax43xcWKiUORd9Fj2WhiEvOlDsaVVIsN0REVCJe866BrUH+cLIxw+XEdHRfdhQnr92TOxZVQiw3RERUYrxdbLFrxIvwdrHBvYwc9P3hXwSH35A7FlUyLDdERFSiatia45eP/PFa4xrIUWswfutpzP3zIjQa3pOKygbLDRERlThLUxMs69sCw9vXAQAs/ScGwzecQmYOb9lApY/lhoiISoVCIeGT1xri23ebQaWUsPf8bfRYfgw37nOgMZUulhsiIipV7/i6YtPQF+BgbYqohFR0W3IUJ2I50JhKD8sNERGVupYe9tg18kU0rmmD5Iwc9F11HJtOxMkdi4wUyw0REZUJFzsLbAsKwOtNnZGrFpi8/Sym7TqHXF7RmEoYyw0REZUZC1MllrzngwmB9QE8uqLxgDUncD8jR+ZkZExYboiIqExJkoSRL9fD9/18YWmqRGhMMrovO4roO2lyRyMjwXJDRESy6NS4BrYPD4BrVQtcT87EW0uP4q8Ld+SORUaA5YaIiGTTsIYNdo98EX6e9sjIUWPoT2FYvP8yL/hHz4XlhoiIZGVvZYqfP/BDvxfcIQQwLyQaQT+HI413FqdiYrkhIiLZqZQKfNndG9/0aAJTpQL7LtxB96VHEXM3Xe5oVAGx3BARUbnRq1Ut/BLkjxo25oi5m4HuS44ihONwyEAsN0REVK40d7PDr6NeRGsPe6Rl52Ho+jAs+Cua43BIb7KXm2XLlsHT0xPm5ubw9fXF4cOHC533wIEDkCQp3+PixYtlmJiIiEpb9Spm2DDUDwP83QEAC/66jA9/Ckcqx+GQHmQtN1u2bMGYMWMwZcoUREREoG3btujcuTPi4oq+JPelS5eQkJCgfdSrV6+MEhMRUVlRKRWY0c0bc99pClMTBf6KejQO50oix+FQ0SQhhGzH+fz8/NCiRQssX75cO83Lywvdu3fH7Nmz881/4MABdOjQAffv34ednV2x1pmamgpbW1ukpKTAxsamuNGJiKgMnbnxAB/9FI6ElCxYm5lgXs9m6NS4htyxqAwZ8v4t25GbnJwchIeHIzAwUGd6YGAgQkNDi1zWx8cHzs7OeOWVV/DPP/8UOW92djZSU1N1HkREVLE0dX00DsfP0x7p2Xn46Kdw/N8fF5HH+1JRAWQrN0lJSVCr1XByctKZ7uTkhNu3bxe4jLOzM1auXIng4GBs374dDRo0wCuvvIJDhw4Vup7Zs2fD1tZW+3BzcyvR10FERGXDwdoMP3/gh8FtPAEAKw7GoN/qE7ibli1zMipvZPtY6tatW3BxcUFoaCj8/f2102fNmoWffvpJ70HCXbt2hSRJ2L17d4Hfz87ORnb2/37xU1NT4ebmxo+liIgqsN/O3MKn284gI0cNJxszLOvbAr7u9nLHolJUIT6WcnBwgFKpzHeUJjExMd/RnKK88MILuHz5cqHfNzMzg42Njc6DiIgqtjea1sSukW1Q19Ead1Kz0ev741h7NBYyDiOlckS2cmNqagpfX1+EhIToTA8JCUFAQIDezxMREQFnZ+eSjkdEROVcXccq2DWiDd5o6ow8jcCMXy9g9OZIZGTnyR2NZGYi58rHjRuHfv36oWXLlvD398fKlSsRFxeHoKAgAMDkyZNx8+ZNrF+/HgCwYMECeHh4oHHjxsjJycHPP/+M4OBgBAcHy/kyiIhIJlZmJlj8ng983ati1u9R+PX0LUQlpGLF+76o62gtdzySiazlplevXkhOTsbMmTORkJAAb29v7NmzB+7ujy7alJCQoHPNm5ycHEyYMAE3b96EhYUFGjdujN9//x1dunSR6yUQEZHMJEnCoDaeaOJiixEbT+FKYjq6LTmCOe80w+tNeWS/MpL1Ojdy4HVuiIiM1920bIzadArHr94DAAxu44lJnRvC1ET2C/LTc6oQA4qJiIhKWvUqZvh5iB+C2tUBAKw5Got3vz+G+HuZMiejssRyQ0RERsVEqcCkzg2xqn9L2FqocDr+AV5fdJh3F69EWG6IiMgovdrICb+PfhHN3OyQmvXo7uKzfr+AXF7V2Oix3BARkdFyrWqJrR/544MXH13V+IfDsej5/THcfPBQ5mRUmlhuiIjIqJmaKPD5G42wsp8vbMxNEBH3AF0WHsb+KH5MZaxYboiIqFIIbFwDv49ui2autkh5mIshP4Zh9p4ofkxlhFhuiIio0nCzt8TWoAAMauMBAPj+0FX0Xnkct/gxlVFhuSEiokrF1ESBaV0bY8X7vqhiboLw6/fRhWdTGRWWGyIiqpRe866B30e1RVNXWzzIzMXQ9WGYtuscsnLVckej58RyQ0RElVatapbYFhSAoW0fnU3147HreGtZKK4kpsucjJ4Hyw0REVVqpiYKTHm9EdYOaoVqVqaISkhF18VHsOVkHCrZHYqMBssNERERgA4NHPHHmLZ4sa4DHuaq8WnwWYzaFIHUrFy5o5GBWG6IiIj+41jFHOsHt8anrzWEiULCb2cS8Pqiw4iIuy93NDIAyw0REdETFAoJw9rXwdYgf7jZWyD+3kO8u+IYlh24Ao2GH1NVBCw3REREBfCpVRW/j26LN5o6I08jMGfvJfRfcwKJqVlyR6NnYLkhIiIqhI25Covf88GcHk1hoVLiyJUkvLbwMPadvy13NCoCyw0REVERJElCz1Zu+HVUGzRytsG9jBx8+FM4Jm8/g8ycPLnjUQFYboiIiPRQ17EKdowIwEcv1YYkAZtOxOP1RUdwOv6B3NHoKSw3REREejIzUWJyFy9s+MAPzrbmiE3KQI/loVjy92WoOdi43GC5ISIiMlBAHQfs/fglvP7fYONv90Wj1/fHEH8vU+5oBJYbIiKiYrG1VGHJez6Y37MZrM1MEHb9PjovPIztp27wysYyY7khIiIqJkmS8HYLV/zxcVu0dK+K9Ow8jPvlNEZuikBKJq9sLBeWGyIioufkZm+JzR++gAmB9WGikPD7mQS8tvAQQq8kyR2tUmK5ISIiKgEmSgVGvlwPwcMC4OlghYSULPRZ9S+m7z6PhzlqueNVKiw3REREJaiZmx1+G/Ui+vjVAgCsC73G+1OVMZYbIiKiEmZlZoKv32qCdYNawcnGDFf/O2X82z8vISdPI3c8o8dyQ0REVEraN3DEvjHt0L15TWgEsOSfK+i+9Cgu3k6VO5pRY7khIiIqRbaWKizo7YNlfVugqqUKFxJS0XXxESw/EMML/5USlhsiIqIy0KWJM/aNbYdXvZyQqxb4Zu9F9Pz+GK4lZcgdzeiw3BAREZWR6lXM8EN/X3z7bjNUMTNB+H8X/lt/7Bo0PIpTYlhuiIiIypAkSXjH1xV7x76ENnWr4WGuGlN3nUf/NSdw88FDueMZBZYbIiIiGbjYWeCnwX6Y8WZjmKsUOHIlCYHzD+Ln49d5FOc5sdwQERHJRKGQMCDAA3tGt0Urj6rIyFHj853n0HfVv4hL5k04i4vlhoiISGa1q1tjy4f+mNa1ESxUShy7moxOCw5h3dFYHsUpBpYbIiKickChkDCojSf2jmmLF2rb42GuGtN/vYDeK48jlmdUGYTlhoiIqBxxr2aFjR+8gC+7e8PKVIkT1+7htQWHsOrwVV4XR08sN0REROWMQiGh3wvu+HPsS2hbzwHZeRp89XsU3lkRiiuJaXLHK/dYboiIiMop16qWWD+4Nf7v7SaoYmaCiLgH6LLoCJYduII8Ne9RVRiWGyIionJMkiT0bl0L+8a9hA4NqiMnT4M5ey/h7eWhuHCL96gqCMsNERFRBeBsa4E1A1th3rvNYGNugjM3UtB1yRF8s/cisnLVcscrV1huiIiIKghJktDD1xV/jWuHLk1qQK0RWH4gBp0WHMLRK0lyxys3WG6IiIgqGEcbcyzr64sf+rdEDRtzXE/ORN9V/2LC1tO4n5EjdzzZyV5uli1bBk9PT5ibm8PX1xeHDx/Wa7mjR4/CxMQEzZs3L92ARERE5VTHRk4IGfcS+vu7Q5KAbeE38Or8g9gVeRNCVN7TxmUtN1u2bMGYMWMwZcoUREREoG3btujcuTPi4uKKXC4lJQX9+/fHK6+8UkZJiYiIyqcq5irM7OaNbUEBqO9kjeSMHHy8ORKD1p3EjfuV8xYOkpCx2vn5+aFFixZYvny5dpqXlxe6d++O2bNnF7pc7969Ua9ePSiVSuzcuRORkZF6rzM1NRW2trZISUmBjY3N88QnIiIqV3LyNFhxMAZL/r6CHLUGlqZKjA9sgIEBHlAqJLnjPRdD3r9lO3KTk5OD8PBwBAYG6kwPDAxEaGhoocutXbsWMTExmDZtml7ryc7ORmpqqs6DiIjIGJmaKDD6lXrY83FbtPawR2aOGl/+dgFvLztaqU4bl63cJCUlQa1Ww8nJSWe6k5MTbt++XeAyly9fxqRJk7BhwwaYmJjotZ7Zs2fD1tZW+3Bzc3vu7EREROVZXUdrbP7wBcx+uwmqmJvg9H+njc/eE4XMnDy545U62QcUS5LuYTIhRL5pAKBWq9GnTx/MmDED9evX1/v5J0+ejJSUFO0jPj7+uTMTERGVdwqFhPda18L+J04b//7QVbw67yD+PH/bqAcc63f4oxQ4ODhAqVTmO0qTmJiY72gOAKSlpSEsLAwREREYOXIkAECj0UAIARMTE+zbtw8vv/xyvuXMzMxgZmZWOi+CiIionHt82vj+qDuYtvs8btx/iI9+CscrDR0x/c3GcLO3lDtiiZPtyI2pqSl8fX0REhKiMz0kJAQBAQH55rexscHZs2cRGRmpfQQFBaFBgwaIjIyEn59fWUUnIiKqcF7xckLI2HYY0aEOVEoJ+y8mouN3B7H0nyvIyTOu+1TJduQGAMaNG4d+/fqhZcuW8Pf3x8qVKxEXF4egoCAAjz5SunnzJtavXw+FQgFvb2+d5R0dHWFubp5vOhEREeVnYarExE4N8ZaPCz7feQ7Hr97D3D8vYUfETXzZzRv+darJHbFEyFpuevXqheTkZMycORMJCQnw9vbGnj174O7uDgBISEh45jVviIiIyDB1Hatg09AXsCPiJmb9HoUriel474fjeNvHBZ+97gUH64o9nEPW69zIgde5ISIi+p+UzFzM+fMiNp6IgxCAjbkJPnmtIfq0rgVFObo2jiHv3yw3REREhMj4B5iy4yzO/3c9nGZudpjV3RveLrYyJ3uE5aYILDdEREQFy1Nr8NPx65i3Lxrp2XlQSMD7L7hjXMf6sLM0lTVbhbhCMREREZUvJkoFBrXxxN/j26Frs5rQCGD9sevo8O0BbPw3DmpNxTgewiM3REREVKDQK0mY/ut5RN9JBwB4u9hgxpve8HWvWuZZ+LFUEVhuiIiI9Jer1uCnY9fx3V/RSMt6dOuGt1u4YFLnhnCsYl5mOVhuisByQ0REZLik9GzM2XsRv4TdAABYm5ng41fqYUCAB0xNSn+UC8tNEVhuiIiIii8y/gGm7TqH0zdSAAB1qlth+puN0bZe9VJdL8tNEVhuiIiIno9GI7At/Aa+2XsRyRk5AIBOjZ3w+euNSu1eVSw3RWC5ISIiKhkpD3Ox4K9orD92HWqNgJmJAkHt6mBY+zowVylLdF0sN0VguSEiIipZl26nYfru8zh2NRkA4GJngR0jAkp0wDGvc0NERERlpkGNKtg41A9L+7RATVtz1K5uheoy3p9K1htnEhERkXGQJAmvN3XGyw0dkZqVC0mS775ULDdERERUYixMlbAwLdnxNobix1JERERkVFhuiIiIyKiw3BAREZFRYbkhIiIio8JyQ0REREaF5YaIiIiMCssNERERGRWWGyIiIjIqLDdERERkVFhuiIiIyKiw3BAREZFRYbkhIiIio8JyQ0REREal0t0VXAgBAEhNTZU5CREREenr8fv24/fxolS6cpOcnAwAcHNzkzkJERERGSotLQ22trZFzlPpyo29vT0AIC4u7pkbpzJITU2Fm5sb4uPjYWNjI3cc2XF7/A+3hS5uD13cHv/DbaGrtLaHEAJpaWmoWbPmM+etdOVGoXg0zMjW1pa/hE+wsbHh9ngCt8f/cFvo4vbQxe3xP9wWukpje+h7UIIDiomIiMiosNwQERGRUal05cbMzAzTpk2DmZmZ3FHKBW4PXdwe/8NtoYvbQxe3x/9wW+gqD9tDEvqcU0VERERUQVS6IzdERERk3FhuiIiIyKiw3BAREZFRYbkhIiIio2KU5WbZsmXw9PSEubk5fH19cfjw4SLnz87OxpQpU+Du7g4zMzPUqVMHa9asKaO0pc/Q7bFhwwY0a9YMlpaWcHZ2xqBBg7S3rajIDh06hK5du6JmzZqQJAk7d+585jIHDx6Er68vzM3NUbt2baxYsaL0g5YRQ7fH9u3b0bFjR1SvXh02Njbw9/fHn3/+WTZhS1lxfjceO3r0KExMTNC8efNSy1fWirM9jHk/WpztYaz70dmzZ6NVq1aoUqUKHB0d0b17d1y6dOmZy5X1vtToys2WLVswZswYTJkyBREREWjbti06d+6MuLi4Qpfp2bMn9u/fj9WrV+PSpUvYtGkTGjZsWIapS4+h2+PIkSPo378/hgwZgvPnz2Pr1q04efIkPvjggzJOXvIyMjLQrFkzLFmyRK/5Y2Nj0aVLF7Rt2xYRERH47LPPMHr0aAQHB5dy0rJh6PY4dOgQOnbsiD179iA8PBwdOnRA165dERERUcpJS5+h2+KxlJQU9O/fH6+88kopJZNHcbaHMe9HDd0exrwfPXjwIEaMGIHjx48jJCQEeXl5CAwMREZGRqHLyLIvFUamdevWIigoSGdaw4YNxaRJkwqc/48//hC2trYiOTm5LOKVOUO3x9y5c0Xt2rV1pi1atEi4urqWWkY5ABA7duwocp5PPvlENGzYUGfaRx99JF544YVSTCYPfbZHQRo1aiRmzJhR8oFkZMi26NWrl/j888/FtGnTRLNmzUo1l1z02R7Gvh99kj7bo7LsR4UQIjExUQAQBw8eLHQeOfalRnXkJicnB+Hh4QgMDNSZHhgYiNDQ0AKX2b17N1q2bIk5c+bAxcUF9evXx4QJE/Dw4cOyiFyqirM9AgICcOPGDezZswdCCNy5cwfbtm3D66+/XhaRy5Vjx47l23adOnVCWFgYcnNzZUpVfmg0GqSlpWlvRlvZrF27FjExMZg2bZrcUWRnzPvR4qhM+9GUlBQAKHI/IMe+1KhunJmUlAS1Wg0nJyed6U5OTrh9+3aBy1y9ehVHjhyBubk5duzYgaSkJAwfPhz37t2r8J8XF2d7BAQEYMOGDejVqxeysrKQl5eHN998E4sXLy6LyOXK7du3C9x2eXl5SEpKgrOzs0zJyod58+YhIyMDPXv2lDtKmbt8+TImTZqEw4cPw8TEqHajxWLM+9HiqCz7USEExo0bhxdffBHe3t6FzifHvtSojtw8JkmSztdCiHzTHtNoNJAkCRs2bEDr1q3RpUsXzJ8/H+vWrTOa/3UYsj0uXLiA0aNHY+rUqQgPD8fevXsRGxuLoKCgsoha7hS07QqaXtls2rQJ06dPx5YtW+Do6Ch3nDKlVqvRp08fzJgxA/Xr15c7TrlQGfajhqgs+9GRI0fizJkz2LRp0zPnLet9qVH9l8PBwQFKpTLfUYnExMR8rfExZ2dnuLi46NxG3cvLC0II3LhxA/Xq1SvVzKWpONtj9uzZaNOmDSZOnAgAaNq0KaysrNC2bVt89dVXlepoRY0aNQrcdiYmJqhWrZpMqeS3ZcsWDBkyBFu3bsWrr74qd5wyl5aWhrCwMERERGDkyJEAHr25CyFgYmKCffv24eWXX5Y5Zdky5v1ocVSG/eioUaOwe/duHDp0CK6urkXOK8e+1KiO3JiamsLX1xchISE600NCQhAQEFDgMm3atMGtW7eQnp6unRYdHQ2FQvHMH1h5V5ztkZmZCYVC99dCqVQC+F/Triz8/f3zbbt9+/ahZcuWUKlUMqWS16ZNmzBw4EBs3LjRKMcP6MPGxgZnz55FZGSk9hEUFIQGDRogMjISfn5+ckcsc8a8Hy0OY96PCiEwcuRIbN++HX///Tc8PT2fuYws+9JSG6osk82bNwuVSiVWr14tLly4IMaMGSOsrKzEtWvXhBBCTJo0SfTr1087f1pamnB1dRXvvPOOOH/+vDh48KCoV6+e+OCDD+R6CSXK0O2xdu1aYWJiIpYtWyZiYmLEkSNHRMuWLUXr1q3legklJi0tTURERIiIiAgBQMyfP19ERESI69evCyHyb4urV68KS0tLMXbsWHHhwgWxevVqoVKpxLZt2+R6CSXK0O2xceNGYWJiIpYuXSoSEhK0jwcPHsj1EkqModviacZ2tpSh28PY96OGbg9j3o8OGzZM2NraigMHDujsBzIzM7XzlId9qdGVGyGEWLp0qXB3dxempqaiRYsWOqeoDRgwQLRr105n/qioKPHqq68KCwsL4erqKsaNG6fzg6roDN0eixYtEo0aNRIWFhbC2dlZ9O3bV9y4caOMU5e8f/75RwDI9xgwYIAQouBtceDAAeHj4yNMTU2Fh4eHWL58edkHLyWGbo927doVOX9FVpzfjScZW7kpzvYw5v1ocbaHse5HC9oOAMTatWu185SHfan0X1giIiIio2BUY26IiIiIWG6IiIjIqLDcEBERkVFhuSEiIiKjwnJDRERERoXlhoiIiIwKyw0REREZFZYbIiIiMiosN1Soa9euQZIkREZGlup6MjMz0aNHD9jY2ECSJDx48EDvZSVJws6dO0s0z7p162BnZ1eiz/k8PDw8sGDBghJ/3qNHj6JJkyZQqVTo3r273suVt+0jhMCHH34Ie3v7Mvl9NTZl9XdeVqZPn47mzZtrvx44cOAzf7/bt2+PMWPGlGouKlssNxXcwIEDIUkSJEmCiYkJatWqhWHDhuH+/fsGP8/TOwA3NzckJCTA29u7BBPn9+OPP+Lw4cMIDQ1FQkKCzp2FH3t6h2WMyro0jBs3Ds2bN0dsbCzWrVtX4DylVaxK0t69e7Fu3Tr89ttvZfL7KoeSKvH6vNEbm4ULFxb6+11cBw4cMPg/YoUprb+xivC3W5pM5A5Az++1117D2rVrkZeXhwsXLmDw4MF48OABNm3a9FzPq1QqUaNGjRJKWbiYmBh4eXkZ5ZtSeRYTE4OgoKBye9fmnJwcmJqaPnO+mJgYODs7F3qne30IIaBWq2Fiwl2isSnoP0tUCZTqnauo1A0YMEB069ZNZ9q4ceOEvb299uu8vDwxePBg4eHhIczNzUX9+vXFggULtN+fNm1avpug/fPPPyI2NlYAEBEREdp5Dxw4IFq1aiVMTU1FjRo1xKeffipyc3OLzLht2zbRqFEjYWpqKtzd3cW3336r/d7TN2Ms6OaEa9euLfQmbQDEDz/8ILp37y4sLCxE3bp1xa5du3SWP3/+vOjcubOwsrISjo6O4v333xd3794tNO/atWuFra2tzrTdu3eLFi1aCDMzM+Hp6SmmT5+u87r1ybFr1y5Rt25dYW5uLtq3by/WrVsnAIj79+8XeGO+adOmCSGEcHd3F7NmzRKDBg0S1tbWws3NTXz//fdFbvOsrCwxatQoUb16dWFmZibatGkjTpw4IYQQ2p9rQdvzSQXdKPPJ7bN3717RsGFDYWVlJTp16iRu3bqls/yaNWtEw4YNhZmZmWjQoIFYunRpkZnbtWsnRowYIcaOHSuqVasmXnrpJSFE0T+/AQMG6ORzd3cXQgih0WjEN998Izw9PYW5ublo2rSp2Lp1q3Zdj7f33r17ha+vr1CpVOLvv//We7m//vpL+Pr6CgsLC+Hv7y8uXryY72ft6+srzMzMRLVq1cRbb72l/V52draYOHGiqFmzprC0tBStW7cW//zzT6Hbxd3dvcDXKIQQy5YtE7Vr1xYqlUrUr19frF+/vtDnedbfeXBwsGjfvr2wsLAQTZs2FaGhoTrLHz16VLRt21aYm5sLV1dXMWrUKJGenl7o+p61HX766Sfh6+srrK2thZOTk3jvvffEnTt3DN7Ws2fPFo6OjsLa2loMHjxYfPrppzo3MX16H5meni769esnrKysRI0aNcS3334r2rVrJz7++GO9shX09/P4BprP+v15WmF/Y8/a3j/++KOwsrIS0dHR2vlHjhwp6tWrJ9LT04t83sqi8r1iI/P0H25MTIxo1KiRcHJy0k7LyckRU6dOFSdOnBBXr14VP//8s7C0tBRbtmwRQgiRlpYmevbsKV577TXt7euzs7PzlZsbN24IS0tLMXz4cBEVFSV27NghHBwctG/CBQkLCxMKhULMnDlTXLp0Saxdu1ZYWFho30yTk5PF0KFDhb+/v0hISBDJycn5niMzM1OMHz9eNG7cWJvv8d2GAQhXV1exceNGcfnyZTF69GhhbW2tfZ5bt24JBwcHMXnyZBEVFSVOnTolOnbsKDp06FBo5qfLzd69e4WNjY1Yt26diImJEfv27RMeHh5i+vTp2nmelSM2NlaoVCoxYcIEcfHiRbFp0ybh4uKiLTfZ2dliwYIFwsbGRvsa09LShBCP3tzs7e3F0qVLxeXLl8Xs2bOFQqEQUVFRhb6G0aNHi5o1a4o9e/aI8+fPiwEDBoiqVauK5ORkkZeXJxISEoSNjY1YsGCBzvZ8UnJysnB1dRUzZ87UZnq8fVQqlXj11VfFyZMnRXh4uPDy8hJ9+vTRLrty5Urh7OwsgoODxdWrV0VwcLCwt7cX69atKzRzu3bthLW1tZg4caK4ePGiiIqKeubP78GDB2LmzJnC1dVVJCQkiMTERCGEEJ999plo2LCh2Lt3r4iJiRFr164VZmZm4sCBA0KI/71xNm3aVOzbt09cuXJFJCUl6b2cn5+fOHDggDh//rxo27atCAgI0L6O3377TSiVSjF16lRx4cIFERkZKWbNmqX9fp8+fURAQIA4dOiQuHLlipg7d64wMzPTeaN6UmJioraAPvkat2/fLlQqlVi6dKm4dOmSmDdvnlAqleLvv/8u8Hme9XfesGFD8dtvv4lLly6Jd955R7i7u2sL/JkzZ4S1tbX47rvvRHR0tDh69Kjw8fERAwcOLPTn+aztsHr1arFnzx4RExMjjh07Jl544QXRuXNn7ff12dZbtmwRpqam4ocffhAXL14UU6ZMEVWqVCmy3AwbNky4urqKffv2iTNnzog33nhDWFtb65SborLl5eWJ4OBgAUBcunRJJCQkiAcPHgghnv1797TC/sb02d7vvvuuaNWqlcjNzRV//PGHUKlU2v/AFPa8lQnLTQU3YMAAoVQqhZWVlTA3N9e29Pnz5xe53PDhw0WPHj10nufpI0BPl5vPPvtMNGjQQGg0Gu08S5cuFdbW1kKtVhe4nj59+oiOHTvqTJs4caJo1KiR9uuPP/64wCM2T5o2bZrODusxAOLzzz/Xfp2eni4kSRJ//PGHEEKIL774QgQGBuosEx8fr90xFeTpctO2bVvx9ddf68zz008/CWdnZ71zfPrpp8Lb21vnOaZMmaItNwWt9zF3d3fx/vvva7/WaDTC0dFRLF++vMD86enpQqVSiQ0bNmin5eTkiJo1a4o5c+Zop9na2hZ4xObpdX/33Xc60x4fSbty5Yp22tKlS3UKtZubm9i4caPOcl9++aXw9/cvdF3t2rUTzZs315mmz8/vu+++0zmakZ6eLszNzfMdeRgyZIh47733hBD/e+PcuXNnsZb766+/tN///fffBQDx8OFDIYQQ/v7+om/fvgW+xitXrghJksTNmzd1pr/yyiti8uTJBW8Y8ej3a8eOHTrTAgICxNChQ3Wmvfvuu6JLly6FPk9Rf+erVq3STjt//rwAoC3Q/fr1Ex9++KHOcocPHxYKhUL7up9W1HYoyIkTJwQAbanXd1sHBQXpPI+fn1+h5SYtLU2YmpqKzZs3a7+fnJwsLCwsdMqNvtke/+0Kod/vT0EK+hvTZ3vfu3dPuLq6imHDhgknJyfx1VdfPfN5KxN+wGwEOnTogOXLlyMzMxOrVq1CdHQ0Ro0apTPPihUrsGrVKly/fh0PHz5ETk6OwQN0o6Ki4O/vD0mStNPatGmD9PR03LhxA7Vq1SpwmW7duulMa9OmDRYsWAC1Wg2lUmlQhoI0bdpU+28rKytUqVIFiYmJAIDw8HD8888/sLa2zrdcTEwM6tev/8znDw8Px8mTJzFr1iztNLVajaysLGRmZsLS0vKZOS5duoRWrVrpPG/r1q2L9RolSUKNGjW0z13Q68rNzUWbNm2001QqFVq3bo2oqCi911kUS0tL1KlTR/u1s7OzNs/du3cRHx+PIUOGYOjQodp58vLynjn+oWXLljpfF+fnd+HCBWRlZaFjx44603NycuDj41Po+gxZ7smfh7OzMwAgMTERtWrVQmRkpM7rftKpU6cghMiXOzs7G9WqVStwmcJERUXhww8/1JnWpk0bLFy40KDneayw19SwYUOEh4fjypUr2LBhg3YeIQQ0Gg1iY2Ph5eWV7/mK2g4AEBERgenTpyMyMhL37t2DRqMBAMTFxaFRo0bPzFWrVi1ERUUhKChI53n9/f3xzz//FLjOmJgY5OTkwN/fXzvN3t4eDRo0KFa2Jxny+/Ms+mzvqlWrYvXq1ejUqRMCAgIwadIkg9Zh7FhujICVlRXq1q0LAFi0aBE6dOiAGTNm4MsvvwQA/PLLLxg7dizmzZsHf39/VKlSBXPnzsW///5r0HqEEDrF5vE0APmm67NMSVGpVDpfS5Kk3RlpNBp07doV33zzTb7lHu8on0Wj0WDGjBl4++23833P3NxcrxzPux2Keu6nFfYzKShDcRWU5/F6H+f64Ycf4OfnpzPfs8qslZWVztfF+fk9Xv/vv/8OFxcXne+ZmZkVuj5Dlnvy9T/epo+Xt7CwKDDX43mUSiXCw8PzbYuCCtyzlOTPuKjXpNFo8NFHH2H06NH5livoPzVA0dshIyMDgYGBCAwMxM8//4zq1asjLi4OnTp1Qk5Ojt65DKXP35wh2Z5kyO/Ps+i7vQ8dOgSlUolbt24hIyMDNjY2Bq3HmLHcGKFp06ahc+fOGDZsGGrWrInDhw8jICAAw4cP184TExOjs4ypqSnUanWRz9uoUSMEBwfr7EBDQ0NRpUqVfH/MTy5z5MgRnWmhoaGoX7++QUdt9MlXkBYtWiA4OBgeHh7FPhOmRYsWuHTpkrZAFkfDhg2xZ88enWlhYWE6Xxf3NT6tbt26MDU1xZEjR9CnTx8AQG5uLsLCwgy+lkdxMjk5OcHFxQVXr15F3759DVr2acX5+TVq1AhmZmaIi4tDu3bt9F5XcZd7WtOmTbF//34MGjQo3/d8fHygVquRmJiItm3b6v2cKpUq38/By8sLR44cQf/+/bXTQkNDCzyK8tjz/B2dP3/eoL+BorbDxYsXkZSUhP/7v/+Dm5sbgPx/D/rw8vLC8ePHdbbB8ePHC52/bt26UKlUOH78uLYk3L9/H9HR0dqfuT7ZHp/F9+S2LO7vT0E/E322d2hoKObMmYNff/0VkyZNwqhRo/Djjz8W+byVCa9zY4Tat2+Pxo0b4+uvvwbw6A86LCwMf/75J6Kjo/HFF1/g5MmTOst4eHjgzJkzuHTpEpKSkpCbm5vveYcPH474+HiMGjUKFy9exK5duzBt2jSMGzcOCkXBv0rjx4/H/v378eWXXyI6Oho//vgjlixZggkTJhj0mjw8PBAbG4vIyEgkJSUhOztbr+VGjBiBe/fu4b333sOJEydw9epV7Nu3D4MHD9b7D3/q1KlYv349pk+fjvPnzyMqKgpbtmzB559/rnf+jz76CBcvXsSnn36K6Oho/PLLL9prbzwuih4eHkhPT8f+/fuRlJSEzMxMvZ//SVZWVhg2bBgmTpyIvXv34sKFCxg6dCgyMzMxZMgQg57Lw8MDhw4dws2bN5GUlKT3ctOnT8fs2bOxcOFCREdH4+zZs1i7di3mz59v0PqL8/OrUqUKJkyYgLFjx+LHH39ETEwMIiIisHTpUp2df0kt97Rp06Zh06ZNmDZtGqKionD27FnMmTMHAFC/fn307dsX/fv3x/bt2xEbG4uTJ0/im2++yVd+n+Th4YH9+/fj9u3b2mtYTZw4EevWrcOKFStw+fJlzJ8/H9u3by/yb0ufv/OCfPrppzh27BhGjBiByMhIXL58Gbt378738be+26FWrVowNTXF4sWLcfXqVezevVt7pNkQH3/8MdasWYM1a9YgOjoa06ZNw/nz5wud39raGkOGDMHEiROxf/9+nDt3DgMHDtTZf+mTzd3dHZIk4bfffsPdu3eRnp5e7N+fgv7GnrW909LS0K9fP4waNQqdO3fGxo0b8csvv2Dr1q1FPm+lIsM4HypBBQ0QFEKIDRs2CFNTUxEXFyeysrLEwIEDha2trbCzsxPDhg0TkyZN0hl0l5iYKDp27Cisra1L7VRwlUolatWqJebOnavzfX0GFGdlZYkePXoIOzu7fKeCPz3Q8umBstHR0eKtt94SdnZ2wsLCQjRs2FCMGTNGZ2D0kwoa2Lt3714REBAgLCwshI2NjWjdurVYuXKl9vv65Hh8KriZmZlo3769WL58uc7gSCGECAoKEtWqVct3KvjTAwObNWtW5FlqDx8+FKNGjRIODg75TgUvLF9Bjh07Jpo2bSrMzMzynQr+pB07duQ73XTDhg2iefPmwtTUVFStWlW89NJLYvv27YWu6+nTcR971s/v6QHFQjwadL1w4ULRoEEDoVKpRPXq1UWnTp3EwYMHhRAFDwgt7nIRERECgIiNjdVOCw4O1r52BwcH8fbbb2u/9/jsRQ8PD6FSqUSNGjXEW2+9Jc6cOVPottm9e7eoW7euMDExKfap4ELo/3d+//597fcfO3HihHZZKysr0bRpU52znwpS1HbYuHGj8PDwEGZmZsLf31/s3r1bJ4e+23rWrFnCwcFBWFtbiwEDBohPPvmkyLOl0tLSxPvvvy8sLS2Fk5OTmDNnTr7fvWdlE0KImTNniho1aghJknROBS/q96cgBf2NPWt7Dxo0SDRp0kRkZWVp51+4cKGwt7cXN27cKPJ5KwtJiBIeAEFEepk1axZWrFiB+Ph4uaMQERkVjrkhKiPLli1Dq1atUK1aNRw9ehRz587FyJEj5Y5FRGR0WG6Iysjly5fx1Vdf4d69e6hVqxbGjx+PyZMnyx2LiMjo8GMpIiIiMio8W4qIiIiMCssNERERGRWWGyIiIjIqLDdERERkVFhuiIiIyKiw3BAREZFRYbkhIiIio8JyQ0REREbl/wEzBPVvSKSe5gAAAABJRU5ErkJggg==",
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "reference_length = 1\n",
+    "candidate_length = np.linspace(1.5, 0.5, 100)\n",
+    "\n",
+    "length_ratio = reference_length / candidate_length\n",
+    "BP = np.minimum(1, np.exp(1 - length_ratio))\n",
+    "\n",
+    "# Plot the data\n",
+    "fig, ax = plt.subplots(1)\n",
+    "lines = ax.plot(length_ratio, BP)\n",
+    "ax.set(\n",
+    "    xlabel=\"Ratio of the length of the reference to the candidate text\",\n",
+    "    ylabel=\"Brevity Penalty\",\n",
+    ")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### N-Gram Precision:\n",
+    "The n-gram precision counts how many n-grams (in your case unigrams, bigrams, trigrams, and four-grams for i =1 , ... , 4) match their n-gram counterpart in the reference translations. This term acts as a precision metric. Unigrams account for adequacy while longer n-grams account for fluency of the translation. To avoid overcounting, the n-gram counts are clipped to the maximal n-gram count occurring in the reference ($m_{n}^{ref}$). Typically precision shows exponential decay with the degree of the n-gram."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGdCAYAAADuR1K7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAseElEQVR4nO3de1BV9d7H8c8WZGMomJB4aYuUphTZMbBzwDyWF4qcTj3TxW6aCnMiyiTqlGTlrRN2I7pB+qjHPJVxyuwykrVPpWJkTyJOPWlXtU20kcAOoBYErOcPxz3PDlSWbNiwfL9m1kzrt35rre/2N46ffutmMwzDEAAAgEX08HcBAAAAvkS4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlhLo7wI6W3Nzs3788Uf16dNHNpvN3+UAAIA2MAxDdXV1GjRokHr0OPbczEkXbn788Uc5HA5/lwEAAE5AWVmZTj/99GP2OenCTZ8+fSQd/sMJDQ31czUAAKAtamtr5XA4PP+OH8tJF26OXIoKDQ0l3AAA0M205ZYSbigGAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACW4vdwk5eXp+joaAUHBysuLk5FRUXH7P/SSy/pvPPO0ymnnKKBAwdq5syZqq6u7qRqAQBAV+fXcFNQUKCMjAzNmzdPpaWlGjdunJKTk+VyuVrtv2XLFk2fPl0pKSn64osv9Oqrr+rTTz9VampqJ1cOAAC6Kr+Gm5ycHKWkpCg1NVUxMTHKzc2Vw+FQfn5+q/23bt2qoUOH6o477lB0dLQuvPBC3XLLLdq2bVsnVw4AALoqv4WbhoYGlZSUKCkpyas9KSlJxcXFre6TmJioH374QYWFhTIMQ/v27dNrr72mKVOmHPU89fX1qq2t9VoAAIB1BfrrxFVVVWpqalJkZKRXe2RkpCoqKlrdJzExUS+99JKmTp2qX3/9VY2NjfrLX/6iZ5555qjnyc7O1sKFC31a+7EMnbu+084Fb3uXHD3kAgBOHn6/odhms3mtG4bRou2InTt36o477tCDDz6okpISbdiwQXv27FFaWtpRj5+VlaWamhrPUlZW5tP6AQBA1+K3mZuIiAgFBAS0mKWprKxsMZtzRHZ2tsaOHau//e1vkqRRo0YpJCRE48aN00MPPaSBAwe22Mdut8tut/v+BwAAgC7JbzM3QUFBiouLk9Pp9Gp3Op1KTExsdZ9Dhw6pRw/vkgMCAiQdnvEBAADw62WpzMxMLV++XCtXrtSuXbt05513yuVyeS4zZWVlafr06Z7+l19+uV5//XXl5+dr9+7d+uijj3THHXfoggsu0KBBg/z1MwAAQBfit8tSkjR16lRVV1dr0aJFcrvdio2NVWFhoaKioiRJbrfb6503M2bMUF1dnZ599lnddddd6tu3ryZMmKBHHnnEXz8BAAB0MTbjJLueU1tbq7CwMNXU1Cg0NNTnx+dpKf/haSkAsC4z/377/WkpAAAAXyLcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAASyHcAAAAS/F7uMnLy1N0dLSCg4MVFxenoqKio/adMWOGbDZbi+Wcc87pxIoBAEBX5tdwU1BQoIyMDM2bN0+lpaUaN26ckpOT5XK5Wu3/1FNPye12e5aysjL169dP11xzTSdXDgAAuiq/hpucnBylpKQoNTVVMTExys3NlcPhUH5+fqv9w8LCNGDAAM+ybds2/fzzz5o5c2YnVw4AALoqv4WbhoYGlZSUKCkpyas9KSlJxcXFbTrGihUrNGnSJEVFRR21T319vWpra70WAABgXYH+OnFVVZWampoUGRnp1R4ZGamKiorj7u92u/XOO+/o5ZdfPma/7OxsLVy4sF21ApI0dO56f5dw0tq7ZIq/SwDQjfj9hmKbzea1bhhGi7bWrFq1Sn379tWVV155zH5ZWVmqqanxLGVlZe0pFwAAdHF+m7mJiIhQQEBAi1maysrKFrM5v2cYhlauXKlp06YpKCjomH3tdrvsdnu76wUAAN2D32ZugoKCFBcXJ6fT6dXudDqVmJh4zH03bdqkb7/9VikpKR1ZIgAA6Ib8NnMjSZmZmZo2bZri4+OVkJCgZcuWyeVyKS0tTdLhS0rl5eVavXq1134rVqzQH//4R8XGxvqjbAAA0IX5NdxMnTpV1dXVWrRokdxut2JjY1VYWOh5+sntdrd4501NTY3Wrl2rp556yh8lAwCALs6v4UaS0tPTlZ6e3uq2VatWtWgLCwvToUOHOrgqAADQXfn9aSkAAABfItwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABL8Xu4ycvLU3R0tIKDgxUXF6eioqJj9q+vr9e8efMUFRUlu92uM888UytXruykagEAQFcX6M+TFxQUKCMjQ3l5eRo7dqyWLl2q5ORk7dy5U0OGDGl1n2uvvVb79u3TihUrNGzYMFVWVqqxsbGTKwcAAF2VX8NNTk6OUlJSlJqaKknKzc3Vu+++q/z8fGVnZ7fov2HDBm3atEm7d+9Wv379JElDhw7tzJIBAEAX57fLUg0NDSopKVFSUpJXe1JSkoqLi1vd56233lJ8fLweffRRDR48WGeddZbuvvtu/fLLL0c9T319vWpra70WAABgXX6buamqqlJTU5MiIyO92iMjI1VRUdHqPrt379aWLVsUHBysdevWqaqqSunp6dq/f/9R77vJzs7WwoULfV4/AGsYOne9v0s4ae1dMsXfJcCi/H5Dsc1m81o3DKNF2xHNzc2y2Wx66aWXdMEFF+iyyy5TTk6OVq1addTZm6ysLNXU1HiWsrIyn/8GAADQdfht5iYiIkIBAQEtZmkqKytbzOYcMXDgQA0ePFhhYWGetpiYGBmGoR9++EHDhw9vsY/dbpfdbvdt8QAAoMvy28xNUFCQ4uLi5HQ6vdqdTqcSExNb3Wfs2LH68ccfdeDAAU/b119/rR49euj000/v0HoBAED34NfLUpmZmVq+fLlWrlypXbt26c4775TL5VJaWpqkw5eUpk+f7ul/ww03KDw8XDNnztTOnTu1efNm/e1vf9OsWbPUq1cvf/0MAADQhfj1UfCpU6equrpaixYtktvtVmxsrAoLCxUVFSVJcrvdcrlcnv69e/eW0+nU7NmzFR8fr/DwcF177bV66KGH/PUTAABAF+PXcCNJ6enpSk9Pb3XbqlWrWrSNHDmyxaUsAACAI/z+tBQAAIAvEW4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClBJrd4eDBg1qyZInef/99VVZWqrm52Wv77t27fVYcAACAWabDTWpqqjZt2qRp06Zp4MCBstlsHVEXAADACTEdbt555x2tX79eY8eO7Yh6AAAA2sX0PTennnqq+vXr1xG1AAAAtJvpcLN48WI9+OCDOnToUEfUAwAA0C6mL0s98cQT+u677xQZGamhQ4eqZ8+eXtu3b9/us+IAAADMMh1urrzyyg4oAwAAwDdMh5v58+d3RB0AAAA+YTrcHFFSUqJdu3bJZrPp7LPP1ujRo31ZFwAAwAkxHW4qKyt13XXXaePGjerbt68Mw1BNTY0uvvhivfLKKzrttNM6ok4AAIA2Mf201OzZs1VbW6svvvhC+/fv188//6z//d//VW1tre64446OqBEAAKDNTM/cbNiwQf/+978VExPjaTv77LP13HPPKSkpyafFAQAAmGV65qa5ubnF49+S1LNnzxbfmQIAAOhspsPNhAkTNGfOHP3444+etvLyct15552aOHGiT4sDAAAwy3S4efbZZ1VXV6ehQ4fqzDPP1LBhwxQdHa26ujo988wzHVEjAABAm5m+58bhcGj79u1yOp368ssvZRiGzj77bE2aNKkj6gMAADDlhN9zM3nyZE2ePNmXtQAAALRbm8LN008/rb/+9a8KDg7W008/fcy+PA4OAAD8qU3h5sknn9SNN96o4OBgPfnkk0ftZ7PZTIebvLw8PfbYY3K73TrnnHOUm5urcePGtdp348aNuvjii1u079q1SyNHjjR1XgAAYE1tCjd79uxp9b/bq6CgQBkZGcrLy9PYsWO1dOlSJScna+fOnRoyZMhR9/vqq68UGhrqWeetyAAA4AjTT0v9XlNTk3bs2KGff/7Z9L45OTlKSUlRamqqYmJilJubK4fDofz8/GPu179/fw0YMMCzBAQEnGj5AADAYkyHm4yMDK1YsULS4WDz5z//Weeff74cDoc2btzY5uM0NDSopKSkxVuNk5KSVFxcfMx9R48erYEDB2rixIn68MMPj9m3vr5etbW1XgsAALAu009Lvfbaa7rpppskSW+//bb27t2rL7/8UqtXr9a8efP00Ucftek4VVVVampqUmRkpFd7ZGSkKioqWt1n4MCBWrZsmeLi4lRfX69//vOfmjhxojZu3Kg///nPre6TnZ2thQsXmviFAAArGDp3vb9LOGntXTLFr+c3HW6qqqo0YMAASVJhYaGuueYanXXWWUpJSTnuk1StsdlsXuuGYbRoO2LEiBEaMWKEZz0hIUFlZWV6/PHHjxpusrKylJmZ6Vmvra2Vw+EwXScAAOgeTF+WioyM1M6dO9XU1KQNGzZ4Xt536NAhU/e+REREKCAgoMUsTWVlZYvZnGP505/+pG+++eao2+12u0JDQ70WAABgXabDzcyZM3XttdcqNjZWNpvN8yK/Tz75xNTj2EFBQYqLi5PT6fRqdzqdSkxMbPNxSktLNXDgwDb3BwAA1mb6stSCBQsUGxursrIyXXPNNbLb7ZKkgIAAzZ0719SxMjMzNW3aNMXHxyshIUHLli2Ty+VSWlqapMOXlMrLy7V69WpJUm5uroYOHapzzjlHDQ0NevHFF7V27VqtXbvW7M8AAAAWdUKfX7j66qtbtN18882mjzN16lRVV1dr0aJFcrvdio2NVWFhoaKioiRJbrdbLpfL07+hoUF33323ysvL1atXL51zzjlav369LrvsshP5GQAAwIL8/vmF9PR0paent7pt1apVXuv33HOP7rnnHlPHBwAAJxe/f34BAADAl/z6+QUAAABfa/fnFwAAALoS0+Hm6quv1pIlS1q0P/bYY7rmmmt8UhQAAMCJMh1uNm3apClTWr5W+dJLL9XmzZt9UhQAAMCJMh1uDhw4oKCgoBbtPXv25KOUAADA70yHm9jYWBUUFLRof+WVV3T22Wf7pCgAAIATZfolfg888ICuuuoqfffdd5owYYIk6f3339eaNWv06quv+rxAAAAAM0yHm7/85S9644039PDDD+u1115Tr169NGrUKP373//W+PHjO6JGAACANjuhzy9MmTKl1ZuKAQAA/O2E3nPzn//8R8uXL9d9992n/fv3S5K2b9+u8vJynxYHAABglumZm88++0yTJk1SWFiY9u7dq9TUVPXr10/r1q3T999/7/mCNwAAgD+YnrnJzMzUjBkz9M033yg4ONjTnpyczHtuAACA35kON59++qluueWWFu2DBw9WRUWFT4oCAAA4UabDTXBwcKsv6/vqq6902mmn+aQoAACAE2U63FxxxRVatGiRfvvtN0mSzWaTy+XS3LlzddVVV/m8QAAAADNMh5vHH39cP/30k/r3769ffvlF48eP17Bhw9SnTx/9/e9/74gaAQAA2sz001KhoaHasmWLPvjgA23fvl3Nzc06//zzNWnSpI6oDwAAwBRT4aaxsVHBwcHasWOHJkyY4Pn8AgAAQFdh6rJUYGCgoqKi1NTU1FH1AAAAtIvpe27uv/9+ZWVled5MDAAA0JWYvufm6aef1rfffqtBgwYpKipKISEhXtu3b9/us+IAAADMMh1urrzyyg4oAwAAwDdMh5v58+d3RB0AAAA+YTrcHLFt2zbt2rVLNptNMTExiouL82VdAAAAJ8R0uPnhhx90/fXX66OPPlLfvn0lSf/5z3+UmJioNWvWyOFw+LpGAACANjP9tNSsWbP022+/adeuXdq/f7/279+vXbt2yTAMpaSkdESNAAAAbWZ65qaoqEjFxcUaMWKEp23EiBF65plnNHbsWJ8WBwAAYJbpmZshQ4Z4Ppr5/zU2Nmrw4ME+KQoAAOBEmQ43jz76qGbPnq1t27bJMAxJh28unjNnjh5//HGfFwgAAGCG6ctSM2bM0KFDh/THP/5RgYGHd29sbFRgYKBmzZqlWbNmefryFmMAANDZTIeb3NzcDigDAADAN0yHm5tvvrkj6gAAAPAJ0/fc+FpeXp6io6MVHBysuLg4FRUVtWm/jz76SIGBgfrDH/7QsQUCAIBuxa/hpqCgQBkZGZo3b55KS0s1btw4JScny+VyHXO/mpoaTZ8+XRMnTuykSgEAQHfh13CTk5OjlJQUpaamKiYmRrm5uXI4HMrPzz/mfrfccotuuOEGJSQkdFKlAACgu/BbuGloaFBJSYmSkpK82pOSklRcXHzU/f7xj3/ou+++a/MHPOvr61VbW+u1AAAA6/JbuKmqqlJTU5MiIyO92iMjI1VRUdHqPt98843mzp2rl156yfMY+vFkZ2crLCzMs/DtKwAArM3001K//vqrnnnmGX344YeqrKxUc3Oz1/bt27ebOp7NZvNaNwyjRZskNTU16YYbbtDChQt11llntfn4WVlZyszM9KzX1tYScAAAsDDT4WbWrFlyOp26+uqrdcEFF7QaRNoiIiJCAQEBLWZpKisrW8zmSFJdXZ22bdum0tJS3X777ZKk5uZmGYahwMBAvffee5owYUKL/ex2u+x2+wnVCAAAuh/T4Wb9+vUqLCxs90cyg4KCFBcXJ6fTqf/6r//ytDudTl1xxRUt+oeGhurzzz/3asvLy9MHH3yg1157TdHR0e2qBwAAWIPpcDN48GD16dPHJyfPzMzUtGnTFB8fr4SEBC1btkwul0tpaWmSDl9SKi8v1+rVq9WjRw/FxsZ67d+/f38FBwe3aAcAACcv0+HmiSee0L333qvnn39eUVFR7Tr51KlTVV1drUWLFsntdis2NlaFhYWe47rd7uO+8wYAAOD/Mx1u4uPj9euvv+qMM87QKaecop49e3ptN/uxzPT0dKWnp7e6bdWqVcfcd8GCBVqwYIGp8wEAAGszHW6uv/56lZeX6+GHH1ZkZOQJ31AMAADQEUyHm+LiYn388cc677zzOqIeAACAdjH9Er+RI0fql19+6YhaAAAA2s10uFmyZInuuusubdy4UdXV1XzaAAAAdCmmL0tdeumlktTii9xH3izc1NTkm8oAAABOgOlw8+GHH3ZEHQAAAD5hOtyMHz++I+oAAADwCdPh5ohDhw7J5XKpoaHBq33UqFHtLgoAAOBEmQ43P/30k2bOnKl33nmn1e3ccwMAAPzJ9NNSGRkZ+vnnn7V161b16tVLGzZs0AsvvKDhw4frrbfe6ogaAQAA2sz0zM0HH3ygN998U2PGjFGPHj0UFRWlyZMnKzQ0VNnZ2ZoyZUpH1AkAANAmpmduDh48qP79+0uS+vXrp59++kmSdO6552r79u2+rQ4AAMAk0+FmxIgR+uqrryRJf/jDH7R06VKVl5fr+eef18CBA31eIAAAgBmmL0tlZGTI7XZLkubPn69LLrlEL730koKCgo77FW8AAICOZjrc3HjjjZ7/Hj16tPbu3asvv/xSQ4YMUUREhE+LAwAAMMvUZanffvtNZ5xxhnbu3OlpO+WUU3T++ecTbAAAQJdgKtz07NlT9fX1stlsHVUPAABAu5i+oXj27Nl65JFH1NjY2BH1AAAAtIvpe24++eQTvf/++3rvvfd07rnnKiQkxGv766+/7rPiAAAAzDIdbvr27aurrrqqI2oBAABoN9Ph5h//+EdH1AEAAOATpu+5AQAA6MpMz9yMHj261aelbDabgoODNWzYMM2YMUMXX3yxTwoEAAAww/TMzaWXXqrdu3crJCREF198sS666CL17t1b3333ncaMGSO3261JkybpzTff7Ih6AQAAjsn0zE1VVZXuuusuPfDAA17tDz30kL7//nu99957mj9/vhYvXqwrrrjCZ4UCAAC0hemZm3/961+6/vrrW7Rfd911+te//iVJuv766z0f1wQAAOhMpsNNcHCwiouLW7QXFxcrODhYktTc3Cy73d7+6gAAAEwyfVlq9uzZSktLU0lJicaMGSObzab/+Z//0fLly3XfffdJkt59912NHj3a58UCAAAcj+lwc//99ys6OlrPPvus/vnPf0qSRowYof/+7//WDTfcIElKS0vTrbfe6ttKAQAA2sB0uJGkG2+8UTfeeONRt/fq1euECwIAAGiPdr3ELz09XVVVVb6qBQAAoN3aFW5efPFF1dbW+qoWAACAdmtXuDEMw1d1AAAA+ITfvy2Vl5en6OhoBQcHKy4uTkVFRUftu2XLFo0dO1bh4eHq1auXRo4cqSeffLITqwUAAF3dCd1QfERdXV27Tl5QUKCMjAzl5eVp7NixWrp0qZKTk7Vz504NGTKkRf+QkBDdfvvtGjVqlEJCQrRlyxbdcsstCgkJ0V//+td21QIAAKzBrzM3OTk5SklJUWpqqmJiYpSbmyuHw6H8/PxW+48ePVrXX3+9zjnnHA0dOlQ33XSTLrnkkmPO9gAAgJNLm8NNjx49FBAQcMwlMLDtE0ENDQ0qKSlRUlKSV3tSUlKrb0BuTWlpqYqLizV+/Pij9qmvr1dtba3XAgAArKvNaWTdunVH3VZcXKxnnnnG1A3GVVVVampqUmRkpFd7ZGSkKioqjrnv6aefrp9++kmNjY1asGCBUlNTj9o3OztbCxcubHNdAACge2tzuGntC99ffvmlsrKy9Pbbb+vGG2/U4sWLTRdgs9m81g3DaNH2e0VFRTpw4IC2bt2quXPnatiwYa1+zFOSsrKylJmZ6Vmvra2Vw+EwXScAAOgeTuiG4h9//FHz58/XCy+8oEsuuUQ7duxQbGysqWNEREQoICCgxSxNZWVli9mc34uOjpYknXvuudq3b58WLFhw1HBjt9v5iCcAACcRUzcU19TU6N5779WwYcP0xRdf6P3339fbb79tOthIUlBQkOLi4uR0Or3anU6nEhMT23wcwzBUX19v+vwAAMCa2jxz8+ijj+qRRx7RgAEDtGbNmlYvU5mVmZmpadOmKT4+XgkJCVq2bJlcLpfS0tIkHb6kVF5ertWrV0uSnnvuOQ0ZMkQjR46UdPi9N48//rhmz57d7loAAIA1tDnczJ07V7169dKwYcP0wgsv6IUXXmi13+uvv97mk0+dOlXV1dVatGiR3G63YmNjVVhYqKioKEmS2+2Wy+Xy9G9ublZWVpb27NmjwMBAnXnmmVqyZIluueWWNp8TAABYW5vDzfTp0497o++JSE9PV3p6eqvbVq1a5bU+e/ZsZmkAAMAxtTnc/D5oAAAAdEV+/7YUAACALxFuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApRBuAACApfg93OTl5Sk6OlrBwcGKi4tTUVHRUfu+/vrrmjx5sk477TSFhoYqISFB7777bidWCwAAujq/hpuCggJlZGRo3rx5Ki0t1bhx45ScnCyXy9Vq/82bN2vy5MkqLCxUSUmJLr74Yl1++eUqLS3t5MoBAEBX5ddwk5OTo5SUFKWmpiomJka5ublyOBzKz89vtX9ubq7uuecejRkzRsOHD9fDDz+s4cOH6+233+7kygEAQFflt3DT0NCgkpISJSUlebUnJSWpuLi4Tcdobm5WXV2d+vXrd9Q+9fX1qq2t9VoAAIB1+S3cVFVVqampSZGRkV7tkZGRqqioaNMxnnjiCR08eFDXXnvtUftkZ2crLCzMszgcjnbVDQAAuja/31Bss9m81g3DaNHWmjVr1mjBggUqKChQ//79j9ovKytLNTU1nqWsrKzdNQMAgK4r0F8njoiIUEBAQItZmsrKyhazOb9XUFCglJQUvfrqq5o0adIx+9rtdtnt9nbXCwAAuge/zdwEBQUpLi5OTqfTq93pdCoxMfGo+61Zs0YzZszQyy+/rClTpnR0mQAAoJvx28yNJGVmZmratGmKj49XQkKCli1bJpfLpbS0NEmHLymVl5dr9erVkg4Hm+nTp+upp57Sn/70J8+sT69evRQWFua33wEAALoOv4abqVOnqrq6WosWLZLb7VZsbKwKCwsVFRUlSXK73V7vvFm6dKkaGxt122236bbbbvO033zzzVq1alVnlw8AALogv4YbSUpPT1d6enqr234fWDZu3NjxBQEAgG7N709LAQAA+BLhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWArhBgAAWIrfw01eXp6io6MVHBysuLg4FRUVHbWv2+3WDTfcoBEjRqhHjx7KyMjovEIBAEC34NdwU1BQoIyMDM2bN0+lpaUaN26ckpOT5XK5Wu1fX1+v0047TfPmzdN5553XydUCAIDuwK/hJicnRykpKUpNTVVMTIxyc3PlcDiUn5/fav+hQ4fqqaee0vTp0xUWFtbJ1QIAgO7Ab+GmoaFBJSUlSkpK8mpPSkpScXGxz85TX1+v2tparwUAAFiX38JNVVWVmpqaFBkZ6dUeGRmpiooKn50nOztbYWFhnsXhcPjs2AAAoOvx+w3FNpvNa90wjBZt7ZGVlaWamhrPUlZW5rNjAwCArifQXyeOiIhQQEBAi1maysrKFrM57WG322W32312PAAA0LX5beYmKChIcXFxcjqdXu1Op1OJiYl+qgoAAHR3fpu5kaTMzExNmzZN8fHxSkhI0LJly+RyuZSWlibp8CWl8vJyrV692rPPjh07JEkHDhzQTz/9pB07digoKEhnn322P34CAADoYvwabqZOnarq6motWrRIbrdbsbGxKiwsVFRUlKTDL+37/TtvRo8e7fnvkpISvfzyy4qKitLevXs7s3QAANBF+TXcSFJ6errS09Nb3bZq1aoWbYZhdHBFAACgO/P701IAAAC+RLgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACWQrgBAACW4vdwk5eXp+joaAUHBysuLk5FRUXH7L9p0ybFxcUpODhYZ5xxhp5//vlOqhQAAHQHfg03BQUFysjI0Lx581RaWqpx48YpOTlZLper1f579uzRZZddpnHjxqm0tFT33Xef7rjjDq1du7aTKwcAAF2VX8NNTk6OUlJSlJqaqpiYGOXm5srhcCg/P7/V/s8//7yGDBmi3NxcxcTEKDU1VbNmzdLjjz/eyZUDAICuKtBfJ25oaFBJSYnmzp3r1Z6UlKTi4uJW9/n444+VlJTk1XbJJZdoxYoV+u2339SzZ88W+9TX16u+vt6zXlNTI0mqra1t709oVXP9oQ45Lo6vo8b0CMbWfzpybBlX/+HvrHV1xNgeOaZhGMft67dwU1VVpaamJkVGRnq1R0ZGqqKiotV9KioqWu3f2NioqqoqDRw4sMU+2dnZWrhwYYt2h8PRjurRFYXl+rsCdBTG1poYV+vqyLGtq6tTWFjYMfv4LdwcYbPZvNYNw2jRdrz+rbUfkZWVpczMTM96c3Oz9u/fr/Dw8GOe52RTW1srh8OhsrIyhYaG+rsc+BBja12MrTUxrq0zDEN1dXUaNGjQcfv6LdxEREQoICCgxSxNZWVli9mZIwYMGNBq/8DAQIWHh7e6j91ul91u92rr27fviRducaGhofxlsijG1roYW2tiXFs63ozNEX67oTgoKEhxcXFyOp1e7U6nU4mJia3uk5CQ0KL/e++9p/j4+FbvtwEAACcfvz4tlZmZqeXLl2vlypXatWuX7rzzTrlcLqWlpUk6fElp+vTpnv5paWn6/vvvlZmZqV27dmnlypVasWKF7r77bn/9BAAA0MX49Z6bqVOnqrq6WosWLZLb7VZsbKwKCwsVFRUlSXK73V7vvImOjlZhYaHuvPNOPffccxo0aJCefvppXXXVVf76CZZht9s1f/78Fpfw0P0xttbF2FoT49p+NqMtz1QBAAB0E37//AIAAIAvEW4AAIClEG4AAIClEG4AAIClEG66mc2bN+vyyy/XoEGDZLPZ9MYbb/i7JPhAdna2xowZoz59+qh///668sor9dVXX/m7LPhAfn6+Ro0a5XkhW0JCgt555x1/lwUfy87Ols1mU0ZGhr9LgQg33c7Bgwd13nnn6dlnn+3Q8/z2228denx427Rpk2677TZt3bpVTqdTjY2NSkpK0sGDB31+Lsa2c51++ulasmSJtm3bpm3btmnChAm64oor9MUXX/j0PIyr/3z66adatmyZRo0a1SHHZ2xPgIFuS5Kxbt264/bbtWuXMXbsWMNutxsxMTGG0+n02nfPnj2GJKOgoMAYP368YbfbjZUrVxpVVVXGddddZwwePNjo1auXERsba7z88stexx4/frxx++23G3PmzDH69u1r9O/f31i6dKlx4MABY8aMGUbv3r2NM844wygsLOyAPwHrqqysNCQZmzZtOmY/xrZ7OvXUU43ly5cfdTvj2n3U1dUZw4cPN5xOpzF+/Hhjzpw5x+zP2HYOwk031pZw09TUZIwYMcKYPHmysWPHDqOoqMi44IILWv3LNHToUGPt2rXG7t27jfLycuOHH34wHnvsMaO0tNT47rvvjKefftoICAgwtm7d6jn++PHjjT59+hiLFy82vv76a2Px4sVGjx49jOTkZGPZsmXG119/bdx6661GeHi4cfDgwQ7807CWb775xpBkfP7550ftw9h2P42NjcaaNWuMoKAg44svvmi1D+PavUyfPt3IyMgwDMM4brhhbDsP4aYba0u4eeedd4zAwEDD7XZ72o72fwq5ubnHPedll11m3HXXXZ718ePHGxdeeKFnvbGx0QgJCTGmTZvmaXO73YYk4+OPP27jLzu5NTc3G5dffrnXn2trGNvu47PPPjNCQkKMgIAAIywszFi/fv1R+zKu3ceaNWuM2NhY45dffjEM4/jhhrHtPNxzYyEPP/ywevfu7VlcLpe++uorORwODRgwwNPvggsuaHX/+Ph4r/Wmpib9/e9/16hRoxQeHq7evXvrvffe8/okhiSv68wBAQEKDw/Xueee62k78pX3ysrKdv/Gk8Htt9+uzz77TGvWrPG0Mbbd24gRI7Rjxw5t3bpVt956q26++Wbt3LmTce3GysrKNGfOHL344osKDg5usZ2x9S+/flsKvpWWlqZrr73Wsz5o0CAZhiGbzdam/UNCQrzWn3jiCT355JPKzc3Vueeeq5CQEGVkZKihocGr3++/yG6z2bzajpy/ubnZ1O85Gc2ePVtvvfWWNm/erNNPP93Tzth2b0FBQRo2bJikw/9offrpp3rqqaeUnZ3NuHZTJSUlqqysVFxcnKetqalJmzdv1rPPPqt9+/Yxtn5EuLGQfv36qV+/fl5tI0eOlMvl0r59+zyJ/dNPP23T8YqKinTFFVfopptuknT4L8M333yjmJgY3xYOGYah2bNna926ddq4caOio6O9tjO21mIYhurr6xnXbmzixIn6/PPPvdpmzpypkSNH6t5771V4eLjCw8O9tjO2nYdw080cOHBA3377rWd9z5492rFjh/r166chQ4a06D958mSdeeaZuvnmm/Xoo4+qrq5O8+bNk6Tj/h/EsGHDtHbtWhUXF+vUU09VTk6OKioq+MvUAW677Ta9/PLLevPNN9WnTx9VVFRIksLCwtSrV69W92Fsu4f77rtPycnJcjgcqqur0yuvvKKNGzdqw4YNrfZnXLuHPn36KDY21qstJCRE4eHhLdqPYGw7D/fcdDPbtm3T6NGjNXr0aElSZmamRo8erQcffLDV/gEBAXrjjTd04MABjRkzRqmpqbr//vslqdXrxP/fAw88oPPPP1+XXHKJLrroIg0YMEBXXnmlT38PDsvPz1dNTY0uuugiDRw40LMUFBQcdR/GtnvYt2+fpk2bphEjRmjixIn65JNPtGHDBk2ePLnV/oyrdTG2ncdmGIbh7yLQuT766CNdeOGF+vbbb3XmmWf6uxz4EGNrTYyrdTG2HYNwcxJYt26devfureHDh+vbb7/VnDlzdOqpp2rLli3+Lg3txNhaE+NqXYxt5+Cem5NAXV2d7rnnHpWVlSkiIkKTJk3SE0884e+y4AOMrTUxrtbF2HYOZm4AAIClcEMxAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwFMINAACwlP8DyN1FcQKWPaEAAAAASUVORK5CYII=",
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Mocked dataset showing the precision for different n-grams\n",
+    "data = {\"1-gram\": 0.8, \"2-gram\": 0.7, \"3-gram\": 0.6, \"4-gram\": 0.5}\n",
+    "\n",
+    "# Plot the datapoints defined above\n",
+    "fig, ax = plt.subplots(1)\n",
+    "bars = ax.bar(*zip(*data.items()))\n",
+    "ax.set(ylabel=\"N-gram precision\")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### N-gram BLEU score:\n",
+    "When the n-gram precision is normalized by the brevity penalty (BP), then the exponential decay of n-grams is almost fully compensated. The BLEU score corresponds to a geometric average of this modified n-gram precision."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGdCAYAAADuR1K7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAyCklEQVR4nO3de1RU9f7/8deIMpgKKiR5GZHUDCU9Bl3wkt2ksFVZrbJMTYVOSppIZZp1SutEV8TqgFKax2+ldMJON9Kmm5fMSsJuVlpakA0SWOClIGD//nA5vzMBOhsGB3bPx1p7Leczn8/e7/GzXL367JvNMAxDAAAAFtHG3wUAAAD4EuEGAABYCuEGAABYCuEGAABYCuEGAABYCuEGAABYCuEGAABYCuEGAABYSlt/F3C81dbW6qefflKnTp1ks9n8XQ4AAPCCYRjav3+/evTooTZtjr4285cLNz/99JMcDoe/ywAAAI1QVFSkXr16HbXPXy7cdOrUSdLhv5zg4GA/VwMAALxRUVEhh8Ph/u/40fzlws2RU1HBwcGEGwAAWhlvLinhgmIAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGAphBsAAGApfg83mZmZioyMVFBQkGJiYrRx48aj9n/uuec0ZMgQnXDCCerevbumTJmisrKy41QtAABo6fwabnJycpSSkqL58+eroKBAI0eOVEJCggoLC+vtv2nTJk2aNEmJiYn68ssv9Z///Ecff/yxkpKSjnPlAACgpfJruElPT1diYqKSkpIUFRWljIwMORwOZWVl1dt/y5Yt6tOnj2655RZFRkZqxIgRuummm7R169bjXDkAAGip/BZuqqqqlJ+fr/j4eI/2+Ph4bd68ud4xw4YN048//qi8vDwZhqG9e/fqxRdf1CWXXNLgcSorK1VRUeGxAQAA62rrrwOXlpaqpqZG4eHhHu3h4eEqLi6ud8ywYcP03HPPady4cfr9999VXV2tyy67TE888USDx0lLS9OCBQt8WvvR9Jn7+nE7Fjx9/2DDIRcA8Nfh9wuKbTabx2fDMOq0HbF9+3bdcsst+sc//qH8/HytXbtWu3fv1rRp0xrc/7x581ReXu7eioqKfFo/AABoWfy2chMWFqaAgIA6qzQlJSV1VnOOSEtL0/Dhw3X77bdLkgYPHqwOHTpo5MiRuv/++9W9e/c6Y+x2u+x2u+9/AAAAaJH8tnITGBiomJgYOZ1Oj3an06lhw4bVO+bQoUNq08az5ICAAEmHV3wAAAD8eloqNTVVTz/9tJYvX66vvvpKs2fPVmFhofs007x58zRp0iR3/0svvVRr1qxRVlaWdu3apffff1+33HKLzjzzTPXo0cNfPwMAALQgfjstJUnjxo1TWVmZFi5cKJfLpejoaOXl5SkiIkKS5HK5PJ55M3nyZO3fv19PPvmkbr31VnXu3Fnnn3++HnroIX/9BAAA0MLYjL/Y+ZyKigqFhISovLxcwcHBPt8/d0v5D3dLAYB1mfnvt9/vlgIAAPAlwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUvz7nBmhNuM3ff7jNH4AZrNwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABL4a3gAP7SeNu7//C2dzQXVm4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAICl+P3FmZmZmXrkkUfkcrk0aNAgZWRkaOTIkfX2nTx5sv7973/XaR84cKC+/PLL5i4VANCK8FJU//H3S1H9unKTk5OjlJQUzZ8/XwUFBRo5cqQSEhJUWFhYb//FixfL5XK5t6KiInXt2lVXX331ca4cAAC0VH4NN+np6UpMTFRSUpKioqKUkZEhh8OhrKysevuHhITopJNOcm9bt27VL7/8oilTphznygEAQEvlt3BTVVWl/Px8xcfHe7THx8dr8+bNXu1j2bJluvDCCxUREdFgn8rKSlVUVHhsAADAuvwWbkpLS1VTU6Pw8HCP9vDwcBUXFx9zvMvl0htvvKGkpKSj9ktLS1NISIh7czgcTaobAAC0bH6/W8pms3l8NgyjTlt9VqxYoc6dO2vs2LFH7Tdv3jyVl5e7t6KioqaUCwAAWji/3S0VFhamgICAOqs0JSUldVZz/swwDC1fvlwTJ05UYGDgUfva7XbZ7fYm1wsAAFoHv63cBAYGKiYmRk6n06Pd6XRq2LBhRx27fv16ffvtt0pMTGzOEgEAQCvk1+fcpKamauLEiYqNjVVcXJyys7NVWFioadOmSTp8SmnPnj1auXKlx7hly5bprLPOUnR0tD/KBgAALZhfw824ceNUVlamhQsXyuVyKTo6Wnl5ee67n1wuV51n3pSXlys3N1eLFy/2R8kAAKCF8/sTipOTk5WcnFzvdytWrKjTFhISokOHDjVzVQAAoLXy+91SAAAAvkS4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAltLW7ICamhqtWLFCb7/9tkpKSlRbW+vx/TvvvOOz4gAAAMwyHW5mzZqlFStW6JJLLlF0dLRsNltz1AUAANAopsPN6tWr9cILL2jMmDHNUQ8AAECTmL7mJjAwUP369WuOWgAAAJrMdLi59dZbtXjxYhmG0Rz1AAAANInp01KbNm3Su+++qzfeeEODBg1Su3btPL5fs2aNz4oDAAAwy/TKTefOnXXFFVdo1KhRCgsLU0hIiMdmVmZmpiIjIxUUFKSYmBht3LjxqP0rKys1f/58RUREyG63q2/fvlq+fLnp4wIAAGsyvXLzzDPP+OzgOTk5SklJUWZmpoYPH66lS5cqISFB27dvV+/evesdc80112jv3r1atmyZ+vXrp5KSElVXV/usJgAA0LqZDjdH/Pzzz/rmm29ks9l0yimn6MQTTzS9j/T0dCUmJiopKUmSlJGRoXXr1ikrK0tpaWl1+q9du1br16/Xrl271LVrV0lSnz59GvsTAACABZk+LXXw4EFNnTpV3bt31znnnKORI0eqR48eSkxM1KFDh7zeT1VVlfLz8xUfH+/RHh8fr82bN9c75pVXXlFsbKwefvhh9ezZU6eccopuu+02/fbbbw0ep7KyUhUVFR4bAACwLtPhJjU1VevXr9err76qX3/9Vb/++qtefvllrV+/XrfeeqvX+yktLVVNTY3Cw8M92sPDw1VcXFzvmF27dmnTpk364osv9NJLLykjI0Mvvviibr755gaPk5aW5nFNkMPh8LpGAADQ+pgON7m5uVq2bJkSEhIUHBys4OBgjRkzRk899ZRefPFF0wX8+QnHhmE0+NTj2tpa2Ww2PffcczrzzDM1ZswYpaena8WKFQ2u3sybN0/l5eXuraioyHSNAACg9TB9zc2hQ4fqrLZIUrdu3UydlgoLC1NAQECdVZqSkpJ69y9J3bt3V8+ePT3uyoqKipJhGPrxxx/Vv3//OmPsdrvsdrvXdQEAgNbN9MpNXFyc7rnnHv3+++/utt9++00LFixQXFyc1/sJDAxUTEyMnE6nR7vT6dSwYcPqHTN8+HD99NNPOnDggLttx44datOmjXr16mXylwAAACsyvXKzePFiXXzxxerVq5eGDBkim82mbdu2KSgoSOvWrTO1r9TUVE2cOFGxsbGKi4tTdna2CgsLNW3aNEmHTynt2bNHK1eulCSNHz9e9913n6ZMmaIFCxaotLRUt99+u6ZOnar27dub/SkAAMCCTIeb6Oho7dy5U88++6y+/vprGYaha6+9Vtdff73pgDFu3DiVlZVp4cKFcrlcio6OVl5eniIiIiRJLpdLhYWF7v4dO3aU0+nUzJkzFRsbq9DQUF1zzTW6//77zf4MAABgUY16zk379u114403+qSA5ORkJScn1/vdihUr6rSdeuqpdU5lAQAAHOFVuHnllVeUkJCgdu3a6ZVXXjlq38suu8wnhQEAADSGV+Fm7NixKi4uVrdu3TR27NgG+9lsNtXU1PiqNgAAANO8Cje1tbX1/hkAAKClMX0reH1+/fVXX+wGAACgyUyHm4ceekg5OTnuz1dffbW6du2qnj176tNPP/VpcQAAAGaZDjdLly51v5/J6XTqrbfe0tq1a5WQkKDbb7/d5wUCAACYYfpWcJfL5Q43r732mq655hrFx8erT58+Ouuss3xeIAAAgBmmV266dOnifvnk2rVrdeGFF0o6/MJL7pQCAAD+Znrl5sorr9T48ePVv39/lZWVKSEhQZK0bds29evXz+cFAgAAmGE63CxatEh9+vRRUVGRHn74YXXs2FHS4dNVDT1pGAAA4HgxHW7atWun2267rU57SkqKL+oBAABoEl6/AAAALIXXLwAAAEvh9QsAAMBSfPL6BQAAgJbCdLi55ZZb9Pjjj9dpf/LJJ7moGAAA+J3pcJObm6vhw4fXaR82bJhefPFFnxQFAADQWKbDTVlZmUJCQuq0BwcHq7S01CdFAQAANJbpcNOvXz+tXbu2Tvsbb7yhk08+2SdFAQAANJbph/ilpqZqxowZ+vnnn3X++edLkt5++2099thjysjI8HV9AAAAppgON1OnTlVlZaX++c9/6r777pMk9enTR1lZWZo0aZLPCwQAADDDdLiRpOnTp2v69On6+eef1b59e/f7pQAAAPytUc+5qa6u1ltvvaU1a9bIMAxJ0k8//aQDBw74tDgAAACzTK/c/PDDD7r44otVWFioyspKjR49Wp06ddLDDz+s33//XUuWLGmOOgEAALxieuVm1qxZio2N1S+//KL27du726+44gq9/fbbPi0OAADALNMrN5s2bdL777+vwMBAj/aIiAjt2bPHZ4UBAAA0humVm9ra2nrf/P3jjz+qU6dOPikKAACgsUyHm9GjR3s8z8Zms+nAgQO65557NGbMGF/WBgAAYJrp01Lp6ek6//zzNXDgQP3+++8aP368du7cqbCwMK1atao5agQAAPCa6XDTs2dPbdu2TatXr1Z+fr5qa2uVmJio66+/3uMCYwAAAH8wFW7++OMPDRgwQK+99pqmTJmiKVOmNFddAAAAjWLqmpt27dqpsrJSNputueoBAABoEtMXFM+cOVMPPfSQqqurm6MeAACAJjEdbj788EOtWbNGvXv31kUXXaQrr7zSYzMrMzNTkZGRCgoKUkxMjDZu3Nhg3/fee082m63O9vXXX5s+LgAAsCbTFxR37txZV111lU8OnpOTo5SUFGVmZmr48OFaunSpEhIStH37dvXu3bvBcd98842Cg4Pdn0888USf1AMAAFo/0+HmmWee8dnB09PTlZiYqKSkJElSRkaG1q1bp6ysLKWlpTU4rlu3burcubPP6gAAANbRqLeCS1JJSYk2btyoTZs2qaSkxPT4qqoq5efnKz4+3qM9Pj5emzdvPurYoUOHqnv37rrgggv07rvvHrVvZWWlKioqPDYAAGBdpsNNRUWFJk6cqJ49e2rUqFE655xz1LNnT02YMEHl5eVe76e0tFQ1NTUKDw/3aA8PD1dxcXG9Y7p3767s7Gzl5uZqzZo1GjBggC644AJt2LChweOkpaUpJCTEvTkcDq9rBAAArY/pcJOUlKQPP/xQr732mn799VeVl5frtdde09atW3XjjTeaLuDPt5UbhtHgreYDBgzQjTfeqNNPP11xcXHKzMzUJZdcokcffbTB/c+bN0/l5eXuraioyHSNAACg9TB9zc3rr7+udevWacSIEe62iy66SE899ZQuvvhir/cTFhamgICAOqs0JSUldVZzjubss8/Ws88+2+D3drtddrvd6/0BAIDWzfTKTWhoqEJCQuq0h4SEqEuXLl7vJzAwUDExMXI6nR7tTqdTw4YN83o/BQUF6t69u9f9AQCAtZleubnrrruUmpqqlStXukNFcXGxbr/9dt19992m9pWamqqJEycqNjZWcXFxys7OVmFhoaZNmybp8CmlPXv2aOXKlZIO303Vp08fDRo0SFVVVXr22WeVm5ur3Nxcsz8DAABYlOlwk5WVpW+//VYRERHuZ9EUFhbKbrfr559/1tKlS919P/nkk6Pua9y4cSorK9PChQvlcrkUHR2tvLw8RURESJJcLpcKCwvd/auqqnTbbbdpz549at++vQYNGqTXX39dY8aMMfszAACARZkON2PHjvVpAcnJyUpOTq73uxUrVnh8njNnjubMmePT4wMAAGsxHW7uueee5qgDAADAJxr9ED8AAICWiHADAAAshXADAAAshXADAAAshXADAAAsxfTdUoZh6MUXX9S7776rkpIS1dbWeny/Zs0anxUHAABglulwM2vWLGVnZ+u8885TeHh4gy+5BAAA8AfT4ebZZ5/VmjVreCowAABokUxfcxMSEqKTTz65OWoBAABoMtPh5t5779WCBQv022+/NUc9AAAATWL6tNTVV1+tVatWqVu3burTp4/atWvn8f2xXpYJAADQnEyHm8mTJys/P18TJkzggmIAANDimA43r7/+utatW6cRI0Y0Rz0AAABNYvqaG4fDoeDg4OaoBQAAoMlMh5vHHntMc+bM0ffff98M5QAAADSN6dNSEyZM0KFDh9S3b1+dcMIJdS4o3rdvn8+KAwAAMMt0uMnIyGiGMgAAAHzDdLi54YYbmqMOAAAAnzAdbv7Xb7/9pj/++MOjjYuNAQCAP5m+oPjgwYOaMWOGunXrpo4dO6pLly4eGwAAgD+ZDjdz5szRO++8o8zMTNntdj399NNasGCBevTooZUrVzZHjQAAAF4zfVrq1Vdf1cqVK3Xuuedq6tSpGjlypPr166eIiAg999xzuv7665ujTgAAAK+YXrnZt2+fIiMjJR2+vubIrd8jRozQhg0bfFsdAACASabDzcknn+x+gN/AgQP1wgsvSDq8otO5c2df1gYAAGCa6XAzZcoUffrpp5KkefPmua+9mT17tm6//XafFwgAAGCG6WtuZs+e7f7zeeedp6+//lpbt25V3759NWTIEJ8WBwAAYJaplZs//vhD5513nnbs2OFu6927t6688kqCDQAAaBFMhZt27drpiy++kM1ma656AAAAmsT0NTeTJk3SsmXLmqMWAACAJjN9zU1VVZWefvppOZ1OxcbGqkOHDh7fp6en+6w4AAAAs0yHmy+++EKnn366JHlceyOJ01UAAMDvTIebd999tznqAAAA8AnT19z4WmZmpiIjIxUUFKSYmBht3LjRq3Hvv/++2rZtq7/97W/NWyAAAGhVTK/cXHHFFfWefrLZbAoKClK/fv00fvx4DRgw4Jj7ysnJUUpKijIzMzV8+HAtXbpUCQkJ2r59u3r37t3guPLyck2aNEkXXHCB9u7da/YnAAAACzO9chMSEqJ33nlHn3zyiTvkFBQU6J133lF1dbVycnI0ZMgQvf/++8fcV3p6uhITE5WUlKSoqChlZGTI4XAoKyvrqONuuukmjR8/XnFxcWbLBwAAFmc63Jx00kkaP368du3apdzcXK1Zs0bfffedJkyYoL59++qrr77SDTfcoDvuuOOo+6mqqlJ+fr7i4+M92uPj47V58+YGxz3zzDP67rvvdM8993hVb2VlpSoqKjw2AABgXabDzbJly5SSkqI2bf7/0DZt2mjmzJnKzs6WzWbTjBkz9MUXXxx1P6WlpaqpqVF4eLhHe3h4uIqLi+sds3PnTs2dO1fPPfec2rb17oxaWlqaQkJC3JvD4fBqHAAAaJ1Mh5vq6mp9/fXXddq//vpr1dTUSJKCgoK8vi38z/0Mw6h3bE1NjcaPH68FCxbolFNO8breefPmqby83L0VFRV5PRYAALQ+pi8onjhxohITE3XnnXfqjDPOkM1m00cffaQHHnhAkyZNkiStX79egwYNOup+wsLCFBAQUGeVpqSkpM5qjiTt379fW7duVUFBgWbMmCFJqq2tlWEYatu2rd58802df/75dcbZ7XbZ7XazPxMAALRSpsPNokWLFB4erocffth9p1J4eLhmz57tvs4mPj5eF1988VH3ExgYqJiYGDmdTl1xxRXudqfTqcsvv7xO/+DgYH3++ecebZmZmXrnnXf04osvKjIy0uxPAQAAFmQ63AQEBGj+/PmaP3++++Lc4OBgjz5Hu437f6WmpmrixImKjY1VXFycsrOzVVhYqGnTpkk6fEppz549Wrlypdq0aaPo6GiP8d26dVNQUFCddgAA8NdlOtz8r8zMTHcQaYxx48aprKxMCxculMvlUnR0tPLy8hQRESFJcrlcKiwsbEqJAADgL6ZJTyh+4IEHtG/fviYVkJycrO+//16VlZXKz8/XOeec4/5uxYoVeu+99xoce++992rbtm1NOj4AALCWJoUbwzB8VQcAAIBP+P3dUgAAAL7UpGtutm/frh49eviqFgAAgCZrUrjhab8AAKCl8TrcREZGHvOpwzabTd99912TiwIAAGgsr8NNSkpKg999//33Wrp0qSorK31REwAAQKN5HW5mzZpVp23fvn267777lJWVpbPOOksPPfSQT4sDAAAwq1HX3Pz2229KT0/XI488oj59+mjNmjUaM2aMr2sDAAAwzVS4qamp0VNPPaUFCxYoKChITzzxhCZMmOD1G8ABAACam9fh5oUXXtBdd92l8vJy3XnnnZo+fboCAwObszYAAADTvA431157rdq3b6/rrrtOP/zwg+bOnVtvv/T0dJ8VBwAAYJbX4eacc8455q3enJ4CAAD+5nW4OdoLLAEAAFoK3i0FAAAshXADAAAshXADAAAshXADAAAshXADAAAsxau7pT777DOvdzh48OBGFwMAANBUXoWbv/3tb7LZbDIM45jPsqmpqfFJYQAAAI3h1Wmp3bt3a9euXdq9e7dyc3MVGRmpzMxMFRQUqKCgQJmZmerbt69yc3Obu14AAICj8mrlJiIiwv3nq6++Wo8//rjHW8AHDx4sh8Ohu+++W2PHjvV5kQAAAN4yfUHx559/rsjIyDrtkZGR2r59u0+KAgAAaCzT4SYqKkr333+/fv/9d3dbZWWl7r//fkVFRfm0OAAAALO8frfUEUuWLNGll14qh8OhIUOGSJI+/fRT2Ww2vfbaaz4vEAAAwAzT4ebMM8/U7t279eyzz+rrr7+WYRgaN26cxo8frw4dOjRHjQAAAF4zHW4k6YQTTtDf//53X9cCAADQZI16QvH//d//acSIEerRo4d++OEHSdKiRYv08ssv+7Q4AAAAs0yHm6ysLKWmpiohIUG//PKL+6F9Xbp0UUZGhq/rAwAAMMV0uHniiSf01FNPaf78+Wrb9v+f1YqNjdXnn3/u0+IAAADMMh1udu/eraFDh9Zpt9vtOnjwoE+KAgAAaCzT4SYyMlLbtm2r0/7GG29o4MCBvqgJAACg0UzfLXX77bfr5ptv1u+//y7DMPTRRx9p1apVSktL09NPP90cNQIAAHjN9MrNlClTdM8992jOnDk6dOiQxo8fryVLlmjx4sW69tprTReQmZmpyMhIBQUFKSYmRhs3bmyw76ZNmzR8+HCFhoaqffv2OvXUU7Vo0SLTxwQAANbVqOfc3HjjjbrxxhtVWlqq2tpadevWrVEHz8nJUUpKijIzMzV8+HAtXbpUCQkJ2r59u3r37l2nf4cOHTRjxgwNHjxYHTp00KZNm3TTTTepQ4cOPHcHAABIauRzbo4ICwtrdLCRpPT0dCUmJiopKUlRUVHKyMiQw+FQVlZWvf2HDh2q6667ToMGDVKfPn00YcIEXXTRRUdd7QEAAH8tXq3cnH766Xr77bfVpUsXDR06VDabrcG+n3zyiVcHrqqqUn5+vubOnevRHh8fr82bN3u1j4KCAm3evFn3339/g30qKytVWVnp/lxRUeHVvgEAQOvkVbi5/PLLZbfbJUljx471yYFLS0tVU1Oj8PBwj/bw8HAVFxcfdWyvXr30888/q7q6Wvfee6+SkpIa7JuWlqYFCxb4pGYAANDyeRVuunTpojZtDp/BmjJlinr16uX+3FR/XgUyDOOoK0OStHHjRh04cEBbtmzR3Llz1a9fP1133XX19p03b55SU1PdnysqKuRwOJpeOAAAaJG8Cjepqam69tprFRQUpMjISLlcriZdayMdvl4nICCgzipNSUlJndWcP4uMjJQknXbaadq7d6/uvffeBsON3W53rzoBAADr82r5pUePHsrNzdUPP/wgwzD0448/qrCwsN7NW4GBgYqJiZHT6fRodzqdGjZsmNf7MQzD45oaAADw1+bVys1dd92lmTNnasaMGbLZbDrjjDPq9DlyOunIizS9kZqaqokTJyo2NlZxcXHKzs5WYWGhpk2bJunwKaU9e/Zo5cqVkqR//etf6t27t0499VRJh5978+ijj2rmzJleHxMAAFibV+Hm73//u6677jr98MMPGjx4sN566y2FhoY2+eDjxo1TWVmZFi5cKJfLpejoaOXl5SkiIkKS5HK5PFaDamtrNW/ePO3evVtt27ZV37599eCDD+qmm25qci0AAMAavH6IX6dOnRQdHa1nnnlGw4cP99l1LMnJyUpOTq73uxUrVnh8njlzJqs0AADgqEw/ofiGG25ojjoAAAB8wqtw07VrV+3YsUNhYWHq0qXLUW/V3rdvn8+KAwAAMMurcLNo0SJ16tTJ/edjPYcGAADAX7wKN/97Kmry5MnNVQsAAECTeRVuzLyPKTg4uNHFAAAANJVX4aZz585en4oy85wbAAAAX/Mq3Lz77rvuP3///feaO3euJk+erLi4OEnSBx98oH//+99KS0trnioBAAC85FW4GTVqlPvPCxcuVHp6use7nC677DKddtppys7O5lZxAADgV6Zf7f3BBx8oNja2TntsbKw++ugjnxQFAADQWKbDjcPh0JIlS+q0L126VA6HwydFAQAANJbpJxQvWrRIV111ldatW6ezzz5bkrRlyxZ99913ys3N9XmBAAAAZpheuRkzZox27typyy67TPv27VNZWZkuv/xy7dixQ2PGjGmOGgEAALxmeuVGknr16qUHHnjA17UAAAA0WaPCza+//qply5bpq6++ks1m08CBAzV16lSFhIT4uj4AAABTTJ+W2rp1q/r27atFixZp3759Ki0tVXp6uvr27atPPvmkOWoEAADwmumVm9mzZ+uyyy7TU089pbZtDw+vrq5WUlKSUlJStGHDBp8XCQAA4C3T4Wbr1q0ewUaS2rZtqzlz5tT7/BsAAIDjyfRpqeDgYBUWFtZpLyoqUqdOnXxSFAAAQGOZDjfjxo1TYmKicnJyVFRUpB9//FGrV69WUlKSxysZAAAA/MH0aalHH31UNptNkyZNUnV1tSSpXbt2mj59uh588EGfFwgAAGCG6XATGBioxYsXKy0tTd99950Mw1C/fv10wgknNEd9AAAApjTqOTeSdMIJJ+i0007zZS0AAABN5nW4mTp1qlf9li9f3uhiAAAAmsrrcLNixQpFRERo6NChMgyjOWsCAABoNK/DzbRp07R69Wrt2rVLU6dO1YQJE9S1a9fmrA0AAMA0r28Fz8zMlMvl0h133KFXX31VDodD11xzjdatW8dKDgAAaDFMPefGbrfruuuuk9Pp1Pbt2zVo0CAlJycrIiJCBw4caK4aAQAAvGb6IX5H2Gw22Ww2GYah2tpaX9YEAADQaKbCTWVlpVatWqXRo0drwIAB+vzzz/Xkk0+qsLBQHTt2bK4aAQAAvOb1BcXJyclavXq1evfurSlTpmj16tUKDQ1tztoAAABM8zrcLFmyRL1791ZkZKTWr1+v9evX19tvzZo1PisOAADALK/DzaRJk2Sz2ZqzFgAAgCYz9RA/AACAlq7Rd0v5SmZmpiIjIxUUFKSYmBht3Lixwb5r1qzR6NGjdeKJJyo4OFhxcXFat27dcawWAAC0dH4NNzk5OUpJSdH8+fNVUFCgkSNHKiEhQYWFhfX237Bhg0aPHq28vDzl5+frvPPO06WXXqqCgoLjXDkAAGip/Bpu0tPTlZiYqKSkJEVFRSkjI0MOh0NZWVn19s/IyNCcOXN0xhlnqH///nrggQfUv39/vfrqq8e5cgAA0FL5LdxUVVUpPz9f8fHxHu3x8fHavHmzV/uora3V/v37j/qOq8rKSlVUVHhsAADAuvwWbkpLS1VTU6Pw8HCP9vDwcBUXF3u1j8cee0wHDx7UNddc02CftLQ0hYSEuDeHw9GkugEAQMvm9wuK/3x7uWEYXt1yvmrVKt17773KyclRt27dGuw3b948lZeXu7eioqIm1wwAAFour28F97WwsDAFBATUWaUpKSmps5rzZzk5OUpMTNR//vMfXXjhhUfta7fbZbfbm1wvAABoHfy2chMYGKiYmBg5nU6PdqfTqWHDhjU4btWqVZo8ebKef/55XXLJJc1dJgAAaGX8tnIjSampqZo4caJiY2MVFxen7OxsFRYWatq0aZIOn1Las2ePVq5cKelwsJk0aZIWL16ss88+273q0759e4WEhPjtdwAAgJbDr+Fm3LhxKisr08KFC+VyuRQdHa28vDxFRERIklwul8czb5YuXarq6mrdfPPNuvnmm93tN9xwA09QBgAAkvwcbqTDbxtPTk6u97s/B5b33nuv+QsCAACtmt/vlgIAAPAlwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUwg0AALAUv4ebzMxMRUZGKigoSDExMdq4cWODfV0ul8aPH68BAwaoTZs2SklJOX6FAgCAVsGv4SYnJ0cpKSmaP3++CgoKNHLkSCUkJKiwsLDe/pWVlTrxxBM1f/58DRky5DhXCwAAWgO/hpv09HQlJiYqKSlJUVFRysjIkMPhUFZWVr39+/Tpo8WLF2vSpEkKCQk5ztUCAIDWwG/hpqqqSvn5+YqPj/doj4+P1+bNm312nMrKSlVUVHhsAADAuvwWbkpLS1VTU6Pw8HCP9vDwcBUXF/vsOGlpaQoJCXFvDofDZ/sGAAAtj98vKLbZbB6fDcOo09YU8+bNU3l5uXsrKiry2b4BAEDL09ZfBw4LC1NAQECdVZqSkpI6qzlNYbfbZbfbfbY/AADQsvlt5SYwMFAxMTFyOp0e7U6nU8OGDfNTVQAAoLXz28qNJKWmpmrixImKjY1VXFycsrOzVVhYqGnTpkk6fEppz549WrlypXvMtm3bJEkHDhzQzz//rG3btikwMFADBw70x08AAAAtjF/Dzbhx41RWVqaFCxfK5XIpOjpaeXl5ioiIkHT4oX1/fubN0KFD3X/Oz8/X888/r4iICH3//ffHs3QAANBC+TXcSFJycrKSk5Pr/W7FihV12gzDaOaKAABAa+b3u6UAAAB8iXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAsxe/hJjMzU5GRkQoKClJMTIw2btx41P7r169XTEyMgoKCdPLJJ2vJkiXHqVIAANAa+DXc5OTkKCUlRfPnz1dBQYFGjhyphIQEFRYW1tt/9+7dGjNmjEaOHKmCggLdeeeduuWWW5Sbm3ucKwcAAC2VX8NNenq6EhMTlZSUpKioKGVkZMjhcCgrK6ve/kuWLFHv3r2VkZGhqKgoJSUlaerUqXr00UePc+UAAKClauuvA1dVVSk/P19z5871aI+Pj9fmzZvrHfPBBx8oPj7eo+2iiy7SsmXL9Mcff6hdu3Z1xlRWVqqystL9uby8XJJUUVHR1J9Qr9rKQ82yXxxbc83pEcyt/zTn3DKv/sO/Wetqjrk9sk/DMI7Z12/hprS0VDU1NQoPD/doDw8PV3Fxcb1jiouL6+1fXV2t0tJSde/evc6YtLQ0LViwoE67w+FoQvVoiUIy/F0Bmgtza03Mq3U159zu379fISEhR+3jt3BzhM1m8/hsGEadtmP1r6/9iHnz5ik1NdX9uba2Vvv27VNoaOhRj/NXU1FRIYfDoaKiIgUHB/u7HPgQc2tdzK01Ma/1MwxD+/fvV48ePY7Z12/hJiwsTAEBAXVWaUpKSuqszhxx0kkn1du/bdu2Cg0NrXeM3W6X3W73aOvcuXPjC7e44OBg/jFZFHNrXcytNTGvdR1rxeYIv11QHBgYqJiYGDmdTo92p9OpYcOG1TsmLi6uTv8333xTsbGx9V5vAwAA/nr8erdUamqqnn76aS1fvlxfffWVZs+ercLCQk2bNk3S4VNKkyZNcvefNm2afvjhB6Wmpuqrr77S8uXLtWzZMt12223++gkAAKCF8es1N+PGjVNZWZkWLlwol8ul6Oho5eXlKSIiQpLkcrk8nnkTGRmpvLw8zZ49W//617/Uo0cPPf7447rqqqv89RMsw26365577qlzCg+tH3NrXcytNTGvTWczvLmnCgAAoJXw++sXAAAAfIlwAwAALIVwAwAALIVwAwAALIVw08ps2LBBl156qXr06CGbzab//ve//i4JPpCWlqYzzjhDnTp1Urdu3TR27Fh98803/i4LPpCVlaXBgwe7H8gWFxenN954w99lwcfS0tJks9mUkpLi71Igwk2rc/DgQQ0ZMkRPPvlksx7njz/+aNb9w9P69et18803a8uWLXI6naqurlZ8fLwOHjzo82Mxt8dXr1699OCDD2rr1q3aunWrzj//fF1++eX68ssvfXoc5tV/Pv74Y2VnZ2vw4MHNsn/mthEMtFqSjJdeeumY/b766itj+PDhht1uN6Kiogyn0+kxdvfu3YYkIycnxxg1apRht9uN5cuXG6Wlpca1115r9OzZ02jfvr0RHR1tPP/88x77HjVqlDFjxgxj1qxZRufOnY1u3boZS5cuNQ4cOGBMnjzZ6Nixo3HyyScbeXl5zfA3YF0lJSWGJGP9+vVH7cfctk5dunQxnn766Qa/Z15bj/379xv9+/c3nE6nMWrUKGPWrFlH7c/cHh+Em1bMm3BTU1NjDBgwwBg9erSxbds2Y+PGjcaZZ55Z7z+mPn36GLm5ucauXbuMPXv2GD/++KPxyCOPGAUFBcZ3331nPP7440ZAQICxZcsW9/5HjRpldOrUybjvvvuMHTt2GPfdd5/Rpk0bIyEhwcjOzjZ27NhhTJ8+3QgNDTUOHjzYjH8b1rJz505DkvH555832Ie5bX2qq6uNVatWGYGBgcaXX35Zbx/mtXWZNGmSkZKSYhiGccxww9weP4SbVsybcPPGG28Ybdu2NVwul7utof9TyMjIOOYxx4wZY9x6663uz6NGjTJGjBjh/lxdXW106NDBmDhxorvN5XIZkowPPvjAy1/211ZbW2tceumlHn+v9WFuW4/PPvvM6NChgxEQEGCEhIQYr7/+eoN9mdfWY9WqVUZ0dLTx22+/GYZx7HDD3B4/XHNjIQ888IA6duzo3goLC/XNN9/I4XDopJNOcvc788wz6x0fGxvr8bmmpkb//Oc/NXjwYIWGhqpjx4568803PV6JIcnjPHNAQIBCQ0N12mmnuduOvOW9pKSkyb/xr2DGjBn67LPPtGrVKncbc9u6DRgwQNu2bdOWLVs0ffp03XDDDdq+fTvz2ooVFRVp1qxZevbZZxUUFFTne+bWv/z6bin41rRp03TNNde4P/fo0UOGYchms3k1vkOHDh6fH3vsMS1atEgZGRk67bTT1KFDB6WkpKiqqsqj35/fyG6z2Tzajhy/trbW1O/5K5o5c6ZeeeUVbdiwQb169XK3M7etW2BgoPr16yfp8H+0Pv74Yy1evFhpaWnMayuVn5+vkpISxcTEuNtqamq0YcMGPfnkk9q7dy9z60eEGwvp2rWrunbt6tF26qmnqrCwUHv37nUn9o8//tir/W3cuFGXX365JkyYIOnwP4adO3cqKirKt4VDhmFo5syZeumll/Tee+8pMjLS43vm1loMw1BlZSXz2opdcMEF+vzzzz3apkyZolNPPVV33HGHQkNDFRoa6vE9c3v8EG5amQMHDujbb791f969e7e2bdumrl27qnfv3nX6jx49Wn379tUNN9yghx9+WPv379f8+fMl6Zj/B9GvXz/l5uZq8+bN6tKli9LT01VcXMw/pmZw88036/nnn9fLL7+sTp06qbi4WJIUEhKi9u3b1zuGuW0d7rzzTiUkJMjhcGj//v1avXq13nvvPa1du7be/sxr69CpUydFR0d7tHXo0EGhoaF12o9gbo8frrlpZbZu3aqhQ4dq6NChkqTU1FQNHTpU//jHP+rtHxAQoP/+9786cOCAzjjjDCUlJemuu+6SpHrPE/+vu+++W6effrouuuginXvuuTrppJM0duxYn/4eHJaVlaXy8nKde+656t69u3vLyclpcAxz2zrs3btXEydO1IABA3TBBRfoww8/1Nq1azV69Oh6+zOv1sXcHj82wzAMfxeB4+v999/XiBEj9O2336pv377+Lgc+xNxaE/NqXcxt8yDc/AW89NJL6tixo/r3769vv/1Ws2bNUpcuXbRp0yZ/l4YmYm6tiXm1Lub2+OCam7+A/fv3a86cOSoqKlJYWJguvPBCPfbYY/4uCz7A3FoT82pdzO3xwcoNAACwFC4oBgAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlvL/AEGIHigLKKERAAAAAElFTkSuQmCC",
+      "text/plain": [
+       "<Figure size 640x480 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Mocked dataset showing the precision multiplied by the BP for different n-grams\n",
+    "data = {\"1-gram\": 0.8, \"2-gram\": 0.77, \"3-gram\": 0.74, \"4-gram\": 0.71}\n",
+    "\n",
+    "# Plot the datapoints defined above\n",
+    "fig, ax = plt.subplots(1)\n",
+    "bars = ax.bar(*zip(*data.items()))\n",
+    "ax.set(ylabel=\"Modified N-gram precision\")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 3. Example Calculations of the BLEU score\n",
+    "\n",
+    "In this example you will have a reference sentence and 2 candidate sentences. You will tokenize all sentences using the NLTK package. Then you will compare the two candidates to the reference using BLEU score.\n",
+    "\n",
+    "First you define and tokenize the sentences."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The NASA Opportunity rover is battling a massive dust storm on planet Mars. -> ['the', 'nasa', 'opportunity', 'rover', 'is', 'battling', 'a', 'massive', 'dust', 'storm', 'on', 'planet', 'mars', '.']\n",
+      "\n",
+      "\n",
+      "The Opportunity rover is combating a big sandstorm on planet Mars. -> ['the', 'opportunity', 'rover', 'is', 'combating', 'a', 'big', 'sandstorm', 'on', 'planet', 'mars', '.']\n",
+      "\n",
+      "\n",
+      "A NASA rover is fighting a massive storm on planet Mars. -> ['a', 'nasa', 'rover', 'is', 'fighting', 'a', 'massive', 'storm', 'on', 'planet', 'mars', '.']\n"
+     ]
+    }
+   ],
+   "source": [
+    "reference = \"The NASA Opportunity rover is battling a massive dust storm on planet Mars.\"\n",
+    "candidate_1 = \"The Opportunity rover is combating a big sandstorm on planet Mars.\"\n",
+    "candidate_2 = \"A NASA rover is fighting a massive storm on planet Mars.\"\n",
+    "\n",
+    "tokenized_ref = nltk.word_tokenize(reference.lower())\n",
+    "tokenized_cand_1 = nltk.word_tokenize(candidate_1.lower())\n",
+    "tokenized_cand_2 = nltk.word_tokenize(candidate_2.lower())\n",
+    "\n",
+    "print(f\"{reference} -> {tokenized_ref}\")\n",
+    "print(\"\\n\")\n",
+    "print(f\"{candidate_1} -> {tokenized_cand_1}\")\n",
+    "print(\"\\n\")\n",
+    "print(f\"{candidate_2} -> {tokenized_cand_2}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.1 Define the functions to calculate the BLEU score\n",
+    "\n",
+    "### Computing the Brevity Penalty\n",
+    "You will start by defining the function for brevity penalty according to the equation (2) in section 2.1."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def brevity_penalty(candidate, reference):\n",
+    "    \"\"\"\n",
+    "    Calculates the brevity penalty given the candidate and reference sentences.\n",
+    "    \"\"\"\n",
+    "    reference_length = len(reference)\n",
+    "    candidate_length = len(candidate)\n",
+    "\n",
+    "    if reference_length < candidate_length:\n",
+    "        BP = 1\n",
+    "    else:\n",
+    "        penalty = 1 - (reference_length / candidate_length)\n",
+    "        BP = np.exp(penalty)\n",
+    "\n",
+    "    return BP"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computing the clipped Precision\n",
+    "Next, you need to define a function to calculate the geometrically averaged clipped precision. This function calculates how many of the n-grams in the candidate sentence actually appear in the reference sentence. The clipping takes care of overcounting. For example if a certain n-gram appears five times in the candidate sentence, but only twice in the reference, the value is clipped to two."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def average_clipped_precision(candidate, reference):\n",
+    "    \"\"\"\n",
+    "    Calculates the precision given the candidate and reference sentences.\n",
+    "    \"\"\"\n",
+    "\n",
+    "    clipped_precision_score = []\n",
+    "    \n",
+    "    # Loop through values 1, 2, 3, 4. This is the length of n-grams\n",
+    "    for n_gram_length in range(1, 5):\n",
+    "        reference_n_gram_counts = Counter(ngrams(reference, n_gram_length))        \n",
+    "        candidate_n_gram_counts = Counter(ngrams(candidate, n_gram_length))                \n",
+    "\n",
+    "        total_candidate_ngrams = sum(candidate_n_gram_counts.values())       \n",
+    "        \n",
+    "        for ngram in candidate_n_gram_counts: \n",
+    "            # check if it is in the reference n-gram\n",
+    "            if ngram in reference_n_gram_counts:\n",
+    "                # if the count of the candidate n-gram is bigger than the corresponding\n",
+    "                # count in the reference n-gram, then set the count of the candidate n-gram \n",
+    "                # to be equal to the reference n-gram\n",
+    "                \n",
+    "                if candidate_n_gram_counts[ngram] > reference_n_gram_counts[ngram]: \n",
+    "                    candidate_n_gram_counts[ngram] = reference_n_gram_counts[ngram] # t\n",
+    "                                                   \n",
+    "            else:\n",
+    "                candidate_n_gram_counts[ngram] = 0 # else set the candidate n-gram equal to zero\n",
+    "\n",
+    "        clipped_candidate_ngrams = sum(candidate_n_gram_counts.values())\n",
+    "        \n",
+    "        clipped_precision_score.append(clipped_candidate_ngrams / total_candidate_ngrams)\n",
+    "    \n",
+    "    # Calculate the geometric average: take the mean of elemntwise log, then exponentiate\n",
+    "    # This is equivalent to taking the n-th root of the product as shown in equation (1) above\n",
+    "    s = np.exp(np.mean(np.log(clipped_precision_score)))\n",
+    "    \n",
+    "    return s\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Computing the BLEU score\n",
+    "Finally, you can compute the BLEU score using the above two functions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def bleu_score(candidate, reference):\n",
+    "    BP = brevity_penalty(candidate, reference)    \n",
+    "    geometric_average_precision = average_clipped_precision(candidate, reference)    \n",
+    "    return BP * geometric_average_precision"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.2 Testing the functions\n",
+    "Now you can test the functions with your Example Reference and Candidates Sentences."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BLEU score of reference versus candidate 1: 27.6\n",
+      "BLEU score of reference versus candidate 2: 35.3\n"
+     ]
+    }
+   ],
+   "source": [
+    "result_candidate_1 = round(bleu_score(tokenized_cand_1, tokenized_ref) * 100, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 1: {result_candidate_1}\")\n",
+    "result_candidate_2 = round(bleu_score(tokenized_cand_2, tokenized_ref) * 100, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 2: {result_candidate_2}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.3 Comparing the Results from your Code with the Sacrebleu Library\n",
+    "Below you will do the same calculation, but using the `sacrebleu` library. Compare them with your implementation above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BLEU score of reference versus candidate 1: 27.6\n",
+      "BLEU score of reference versus candidate 2: 35.3\n"
+     ]
+    }
+   ],
+   "source": [
+    "result_candidate_1 = round(sacrebleu.sentence_bleu(candidate_1, [reference]).score, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 1: {result_candidate_1}\")\n",
+    "result_candidate_2 = round(sacrebleu.sentence_bleu(candidate_2, [reference]).score, 1)\n",
+    "print(f\"BLEU score of reference versus candidate 2: {result_candidate_2}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 4.  BLEU computation on a corpus\n",
+    "\n",
+    "## 4.1 Loading Datasets for Evaluation Using the BLEU Score\n",
+    "\n",
+    "In this section, you will use a simple pipeline for evaluating machine translated text. You will use English to German translations generated by [Google Translate](https://translate.google.com). There are three files you will need:\n",
+    "\n",
+    "1. A source text in English. In this lab, you will use the first 1671 words of the [wmt19](http://statmt.org/wmt19/translation-task.html) evaluation dataset downloaded via SacreBLEU.\n",
+    "2. A reference translation to German of the corresponding first 1671 words from the original English text. This is also provided by SacreBLEU.\n",
+    "3. A candidate machine translation to German from the same 1671 words. This is generated by Google Translate.\n",
+    "\n",
+    "With that, you can now compare the reference and candidate translation to get the BLEU Score."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Loading the raw data\n",
+    "wmt19_src = open(\"data/wmt19_src.txt\", \"r\")\n",
+    "wmt19_src_1 = wmt19_src.read()\n",
+    "wmt19_src.close()\n",
+    "\n",
+    "wmt19_ref = open(\"data/wmt19_ref.txt\", \"r\")\n",
+    "wmt19_ref_1 = wmt19_ref.read()\n",
+    "wmt19_ref.close()\n",
+    "\n",
+    "wmt19_can = open(\"data/wmt19_can.txt\", \"r\")\n",
+    "wmt19_can_1 = wmt19_can.read()\n",
+    "wmt19_can.close()\n",
+    "\n",
+    "tokenized_corpus_src = nltk.word_tokenize(wmt19_src_1.lower())\n",
+    "tokenized_corpus_ref = nltk.word_tokenize(wmt19_ref_1.lower())\n",
+    "tokenized_corpus_cand = nltk.word_tokenize(wmt19_can_1.lower())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that you have your data loaded, you can inspect the first sentence of each dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "English source text:\n",
+      "\n",
+      "Welsh AMs worried about 'looking like muppets'\n",
+      "There is consternation among some AMs at a suggestion their title should change to MWPs (Member of the Welsh Parliament).\n",
+      " -> ['\\ufeffwelsh', 'ams', 'worried', 'about', \"'looking\", 'like', \"muppets'\", 'there', 'is', 'consternation', 'among', 'some', 'ams', 'at', 'a', 'suggestion', 'their', 'title', 'should', 'change', 'to', 'mwps', '(', 'member', 'of', 'the', 'welsh', 'parliament', ')', '.']\n",
+      "\n",
+      "\n",
+      "German reference translation:\n",
+      "\n",
+      "Walisische Ageordnete sorgen sich \"wie Dödel auszusehen\"\n",
+      "Es herrscht Bestürzung unter einigen Mitgliedern der Versammlung über einen Vorschlag, der ihren Titel zu MWPs (Mitglied der walisischen Parlament) ändern soll.\n",
+      " -> ['\\ufeffwalisische', 'ageordnete', 'sorgen', 'sich', '``', 'wie', 'dödel', 'auszusehen', \"''\", 'es', 'herrscht', 'bestürzung', 'unter', 'einigen', 'mitgliedern', 'der', 'versammlung', 'über', 'einen', 'vorschlag', ',', 'der', 'ihren', 'titel', 'zu', 'mwps', '(', 'mitglied', 'der', 'walisischen', 'parlament', ')', 'ändern', 'soll', '.']\n",
+      "\n",
+      "\n",
+      "German machine translation:\n",
+      "\n",
+      "Walisische AMs machten sich Sorgen, dass sie wie Muppets aussehen könnten\n",
+      "Einige AMs sind bestürzt über den Vorschlag, ihren Titel in MWPs (Mitglied des walisischen Parlaments) zu ändern.\n",
+      "Es ist aufg -> ['walisische', 'ams', 'machten', 'sich', 'sorgen', ',', 'dass', 'sie', 'wie', 'muppets', 'aussehen', 'könnten', 'einige', 'ams', 'sind', 'bestürzt', 'über', 'den', 'vorschlag', ',', 'ihren', 'titel', 'in', 'mwps', '(', 'mitglied', 'des', 'walisischen', 'parlaments']\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"English source text:\\n\")\n",
+    "print(f\"{wmt19_src_1[0:170]} -> {tokenized_corpus_src[0:30]}\\n\\n\")\n",
+    "print(\"German reference translation:\\n\")\n",
+    "print(f\"{wmt19_ref_1[0:219]} -> {tokenized_corpus_ref[0:35]}\\n\\n\")\n",
+    "print(\"German machine translation:\\n\")\n",
+    "print(f\"{wmt19_can_1[0:199]} -> {tokenized_corpus_cand[0:29]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And lastly, you can calculate the BLEU score of the translation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BLEU score of the reference versus candidate translation: 43.2\n"
+     ]
+    }
+   ],
+   "source": [
+    "result =  round(sacrebleu.sentence_bleu(wmt19_can_1, [wmt19_ref_1]).score, 1)\n",
+    "print(f\"BLEU score of the reference versus candidate translation: {result}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4.2 BLEU Score Interpretation on a Corpus\n",
+    "The table below (taken from [here](https://cloud.google.com/translate/automl/docs/evaluate)) shows the typical values of BLEU score. You can see that the translation above is of high quality according to this table and in comparison to the given reference sentence. (*if you see \"Hard to get the gist\", please open your workspace, delete `wmt19_can.txt` and get the latest version via the Lab Help button*)\n",
+    "\n",
+    "|Score      | Interpretation                                                |\n",
+    "|:---------:|:-------------------------------------------------------------:|\n",
+    "| < 10      | Almost useless                                                |\n",
+    "| 10 - 19   | Hard to get the gist                                          |\n",
+    "| 20 - 29   | The gist is clear, but has significant grammatical errors     |\n",
+    "| 30 - 40   | Understandable to good translations                           |\n",
+    "| 40 - 50   | High quality translations                                     |\n",
+    "| 50 - 60   | Very high quality, adequate, and fluent translations          |\n",
+    "| > 60      | Quality often better than human                               |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/C4W1_QKV_Attention.ipynb

@@ -0,0 +1,281 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "707052ae",
+   "metadata": {},
+   "source": [
+    "# Scaled Dot-Product Attention: Ungraded Lab\n",
+    "\n",
+    "The 2017 paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762) introduced the Transformer model and scaled dot-product attention, sometimes also called QKV (**Q**ueries, **K**eys, **V**alues) attention. Since then, Transformers have come to dominate large-scale natural language applications. Scaled dot-product attention can be used to improve seq2seq models as well. In this ungraded lab, you'll implement a simplified version of scaled dot-product attention and replicate word alignment between English and French, as shown in [Bhadanau, et al. (2014)](https://arxiv.org/abs/1409.0473).\n",
+    "\n",
+    "The Transformer model learns how to align words in different languages. You won't be training any weights here, so instead you will use [pre-trained aligned word embeddings from here](https://fasttext.cc/docs/en/aligned-vectors.html). Run the cell below to load the embeddings and set up the rest of the notebook.\n",
+    "\n",
+    "This is a practice notebook, where you can train writing your code. All of the solutions are provided at the end of the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "aa4d9f30",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import the libraries\n",
+    "import pickle\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "# Load the word2int dictionaries\n",
+    "with open(\"./data/word2int_en.pkl\", \"rb\") as f:\n",
+    "    en_words = pickle.load(f)\n",
+    "    \n",
+    "with open(\"./data/word2int_fr.pkl\", \"rb\") as f:\n",
+    "    fr_words = pickle.load(f)\n",
+    "\n",
+    "# Load the word embeddings\n",
+    "en_embeddings = np.load(\"./data/embeddings_en.npz\")[\"embeddings\"]\n",
+    "fr_embeddings = np.load(\"./data/embeddings_fr.npz\")[\"embeddings\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a6914081",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define some helper functions\n",
+    "\n",
+    "def tokenize(sentence, token_mapping):\n",
+    "    tokenized = []\n",
+    "    \n",
+    "    for word in sentence.lower().split(\" \"):\n",
+    "        try:\n",
+    "            tokenized.append(token_mapping[word])\n",
+    "        except KeyError:\n",
+    "            # Using -1 to indicate an unknown word\n",
+    "            tokenized.append(-1)\n",
+    "        \n",
+    "    return tokenized\n",
+    "\n",
+    "\n",
+    "def embed(tokens, embeddings):\n",
+    "    embed_size = embeddings.shape[1]\n",
+    "    \n",
+    "    output = np.zeros((len(tokens), embed_size))\n",
+    "    for i, token in enumerate(tokens):\n",
+    "        if token == -1:\n",
+    "            output[i] = np.zeros((1, embed_size))\n",
+    "        else:\n",
+    "            output[i] = embeddings[token]\n",
+    "            \n",
+    "    return output"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6153d4b2",
+   "metadata": {},
+   "source": [
+    "The scaled-dot product attention consists of two matrix multiplications and a softmax scaling as shown in the diagram below from [Vaswani, et al. (2017)](https://arxiv.org/abs/1706.03762). It takes three input matrices, the queries, keys, and values.\n",
+    "\n",
+    "![scaled-dot product attention diagram](./images/attention.png)\n",
+    "\n",
+    "Mathematically, this is expressed as\n",
+    "\n",
+    "$$ \n",
+    "\\large \\mathrm{Attention}\\left(Q, K, V\\right) = \\mathrm{softmax}\\left(\\frac{QK^{\\top}}{\\sqrt{d_k}}\\right)V\n",
+    "$$\n",
+    "\n",
+    "where $Q$, $K$, and $V$ are the queries, keys, and values matrices respectively, and $d_k$ is the dimension of the keys. In practice, Q, K, and V all have the same dimensions. This form of attention is faster and more space-efficient than what you implemented before since it consists of only matrix multiplications instead of a learned feed-forward layer.\n",
+    "\n",
+    "Conceptually, the first matrix multiplication is a measure of the similarity between the queries and the keys. This is transformed into weights using the softmax function. These weights are then applied to the values with the second matrix multiplication resulting in output attention vectors. Typically, decoder states are used as the queries while encoder states are the keys and values.\n",
+    "\n",
+    "### Exercise 1\n",
+    "Implement the softmax function with Numpy and use it to calculate the weights from the queries and keys. Assume the queries and keys are 2D arrays (matrices). Note that since the dot-product of Q and K will be a matrix, you'll need to calculate softmax over a specific axis. See the end of the notebook for solutions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "3932b927",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def softmax(x, axis):    \n",
+    "    \"\"\" Calculate softmax function for an array x\n",
+    "\n",
+    "        axis=0 calculates softmax across rows which means each column sums to 1 \n",
+    "        axis=1 calculates softmax across columns which means each row sums to 1\n",
+    "    \"\"\"\n",
+    "    # Replace pass with your code.\n",
+    "    return np.exp(x) / np.sum(np.exp(x), axis=axis, keepdims=True)\n",
+    "\n",
+    "def calculate_weights(queries, keys):\n",
+    "    \"\"\" Calculate the weights for scaled dot-product attention\"\"\"\n",
+    "    # Replace None with your code.\n",
+    "    dot = queries.dot(keys.T) / np.sqrt(keys.shape[-1])\n",
+    "    weights = softmax(dot, axis=1)\n",
+    "    \n",
+    "    assert weights.sum(axis=1)[0] == 1, \"Each row in weights must sum to 1\"\n",
+    "    \n",
+    "    # Replace pass with your code.\n",
+    "    return weights"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "51f47450",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApoAAAKyCAYAAAB1836kAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACdaUlEQVR4nOzdeVxU1f8/8NcddmRVERBRFkVFwRWXwn0hc9fc11wy00+aZmb6KS1Ns0xLS80lc0vN/WOumYBL7oioKKCyiCiCwICyc35/+GO+jswgBDN30Nfz8ZhHce+5974GEN6ce865khBCgIiIiIionCnkDkBEREREryYWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSYRERER6QQLTSIiIiLSCRaaRERERKQTLDSJiIiISCdYaBIRERGRTrDQJCqD4OBghIaGlqjt1atXERwcrONEREREhkMSQgi5QxBVVAqFAm3atEFQUNBL23bo0AEnT55EXl6eHpIRERHJjz2aRGVUmr/V+HcdERG9TlhoEulJcnIyLCws5I5BRESkN8ZyByCqSJRKJVJTU9W2ZWdnIy4uTmtvZWZmJoKCgnDt2jU0atRIDymJiIgMAwtNolJYunQpvvzyS7VtFy9ehJubW4mOHzt2rA5SERERGSYWmkSlYGdnh5o1a6o+jo2NhampKZycnDS2lyQJFhYW8PDwwKBBgzB8+HB9RSUiIpIdZ50TlYFCoYC/vz+XLSIiItKAPZpEZfDrr7/C0dFR7hhEREQGiT2aRERERKQT7NEkKkcpKSnIyMgodr3M58d4EhERvcpYaBKVUUREBObOnYvDhw8jLS2t2LaSJPHJQERE9NpgoUlUBleuXEG7du1UvZjm5uZwcHCAQsFnIRAREbHQJCqDzz77DOnp6ejUqROWLl2Khg0byh2JiIjIYHAyEFEZ2NnZoaCgAAkJCahUqZLccYiIiAwK7+8RlUFBQQHq1q3LIpOIiEgDFppEZdC4cWMkJCTIHYNeA+np6QgODsatW7eKbXfr1i0EBwcjIyNDT8mIiLRjoUlUBrNmzUJCQgI2bdokdxR6xa1atQodOnTAqVOnim136tQpdOjQAWvWrNFTMiIi7ThGk6iMVq9ejY8//hjjxo3D2LFj4enpCQsLC7lj0SvmzTffxKVLl5Camgpzc3Ot7TIzM2FnZ4cWLVrg5MmTekxIRFQUC02iMjAyMipVe66jafhyc3Px66+/4tChQ7hz506xC/BLkoTbt2/rJZeTkxNsbGwQERHx0rZ169ZFeno67t+/r4dkRETacXkjojIo7d9p/LvOsCUlJaFjx464fv16ib5WkiTpIdUzqampJX6qlK2tLWJiYnSciIjo5VhoEpVBQUGB3BGoHH366ae4du0aatSogU8++QR+fn6oVq2aQSzA7+joiMjISOTn5xfbk56Xl4fIyEhUrVpVj+mIiDRjoUlE9P8dOHAAJiYm+Pvvv1G7dm2546hp06YNfv/9d6xYsQJTpkzR2m7lypVIS0vDW2+9pcd0RESayf9nOhGRgUhLS0PdunUNrsgEgKlTpwIAZsyYga+//hpPnjxR2//kyRMsXLgQ06dPh0KhwEcffSRDSiIidZwMRFQOHj58iLVr1yIoKAjx8fHIyspSmySyd+9eJCYmYuTIkcXOGCZ5+fj4IDc3Fzdv3pQ7ikaLFy/Gp59+CkmSYGpqCm9vb9jZ2SE1NRU3btxATk4OhBBYtGgRPvnkE7njEhHx1jlRWe3duxejR49Genq6agLJi5NEbty4gf/+979wcHBA37595YhJJTBu3DhMmzYNly5dQrNmzeSOU8Qnn3yCunXr4rPPPkN4eDhCQkLU9jds2BDz589Hr169ZEpIRKSOPZpEZXDlyhW0bNkSQghMmTIFvXr1wrRp03D58mXk5+er2t29exeenp4YOnQoNm/eLGNiKo4QAiNGjEBQUBBWrFiB3r17yx1Jq9u3byM8PBxKpRLW1tZo0KABPDw85I5FRKSGPZpEZfD1118jLy8Pa9euxbvvvgsAGm+Nu7u7w9HREVevXtV3RCqFTp06AQASExPRr18/2Nvbw9PTU+uz7CVJwvHjx/UZUcXT0xOenp6yXJuIqKTYo0lUBk5OTigoKEBiYqJqW5s2bXDmzBm1Hk0AaNGiBaKiovD48WN9x6QSKu0yRpIkFfk6ExHR/2GPJlEZpKSkwMfHp0RthRDIzs7WcSIqixMnTsgdAQAQHBwMALC0tETz5s3VtpVG27ZtyzUXEVFpsUeTqAxq1KiB7OxsPHr0SLVNU49mfn4+KleujGrVqiEyMlKOqFSBKBQKSJKEunXr4saNG2rbSoqPOyUiQ8AeTaIy8Pf3xx9//IF9+/YVO3Fkw4YNSE9Px+DBg/WYjiqqtm3bQpIktUdOFm4jIqpI2KNJVAYXLlxAq1atULVqVaxfvx7du3cv0qO5ceNGfPDBB8jJycGVK1fg7e0tc2oiIiL9YKFJVEZLly7Fxx9/DACoVq0asrKyoFQq8eabbyI8PFw1+WfFihWYOHGinFGphO7evYvt27cjNDQUjx8/Rm5ursZ2cs46JyKqCFhoEpWDw4cPY/bs2UUW0AaeLaL9zTffoFu3bjIko9L69ttvMXv2bOTl5aluVT//Y/L5bZx1TkRUPBaaROUoNjYWYWFhSEtLg5WVFby9vQ3yudmk2cGDB9GjRw84Ozvjq6++wrJly3D9+nUcPXoUcXFxCA0Nxbp165Cfn49FixbB19cX7dq103vO1NRU3L17FxkZGSjuRzhnnROR3FhoEhH9f926dcPRo0cRHByMN998U+MKAsnJyejfvz+uXLmCCxcuoE6dOnrL9/fff2P27Nk4f/78S9ty1jkRGQIWmkRE/1+1atVgamqKe/fuAdC++H5CQgJq1aqFgQMH6u2RogcPHkSfPn2Ql5cHc3NzuLu7w8HBodiZ6IayLigRvb64vBFROUhLS0NgYCDu3LlT7O1MSZLw3//+V8/pqKSUSiUaNWqk+rjwcaJKpRI2Njaq7c7OzmjYsKFeC7nPP/8c+fn5mDBhAhYtWgRbW1u9XZuI6N9ioUlURvPmzcM333yjeuqPpiJTkiTV5BEWmoarWrVqUCqVah8DwK1bt+Dn56fWNiMjA8nJyXrLduPGDVStWhUrV67U2zWJiMqKhSZRGXz77beYN28eAKBVq1Zo0qTJS29nkuHy9PTE5cuXVR+3bNkSv//+O1auXKlWaB4/fhxRUVFwd3fXWzZ7e3u4uLjo7XpEROWBhSZRGaxevRqSJGHLli186s8r4K233kJwcDAuXLgAPz8/DB06FF988QV+++03REREoHXr1nj48CF27NgBSZIwYsQIvWXr2rUrdu7ciSdPnqBSpUp6uy4RUVlwMhBRGVhYWKB69eq4ffu23FGoHNy9exdff/013nnnHQQEBAAAjh49isGDByM1NVWt7TvvvIOtW7fC2Fg/f6/HxsaiRYsW6Ny5M9auXasaP0pEZMhYaBKVQZ06dWBlZaVxoXZ6daSlpeHQoUOIjo6GhYUF2rRpg6ZNm+o9R0REBEaOHIl79+5hyJAh8PT0hKWlpdb2I0eO1GM6ovIXHBwMW1tbtUl62ly9ehWpqalcP9bAsNAkKoP//ve/+OabbxAZGYlatWrJHYdecZs3b8bMmTORkJBQonHAfGoRVXQKhQJt2rRBUFDQS9t26NABJ0+e5PqxBoZjNInKYPbs2Th27Bh69+6NjRs3wtfXV+5I9Iravn27qoeyRo0a8PHx4cQzei2Upj+MfWeGh4UmURmYm5sjKCgIgwYNQtOmTdGkSZNib2dKkoR169bpOSWVVlhYGH788UcEBQUhPj4e2dnZar0kq1evRkxMDD799FO19TV1aeHChZAkCQsXLsTHH38MhUKhl+sSVRTJycmwsLCQOwa9gIUmURnk5+dj0qRJOHDgAAoKCnDp0iVcunRJa3sWmobvp59+wkcffaRWWL7Ya5idnY1vvvkGDRo0wLBhw/SSKyIiAi4uLvjkk0/0cj0iOSiVyiIT77KzsxEXF6e1tzIzMxNBQUG4du1aicZykn6x0CQqg/nz52P9+vUwNTVF//790bhxY97OrMBOnDiBDz/8ENbW1liwYAF69eqFIUOG4J9//lFrN3DgQEydOhV79uzRW6FZpUoVODo66uVaRHJZunQpvvzyS7VtFy9ehJubW4mOHzt2rA5SUVmw0CQqgw0bNkChUODYsWNo06aN3HGojJYsWQIA2LJlC7p37w6gaG8mADg5OcHV1RU3btzQW7aePXvi119/RXJyMqpUqaK36xLpk52dHWrWrKn6ODY2FqampnByctLYXpIkWFhYwMPDA4MGDcLw4cP1FZVKiLPOicrA0tISbm5uei04SHeqVq0KMzMzxMfHq7a1adMGZ86cKTKDu1WrVggPD0daWppesj1+/BitWrVCrVq1sHnzZvZuVgAdO3Ys8zkkScLx48fLIU3FpFAo4O/vj+DgYLmj0L/EHk2iMnBzc+OkjFdIRkZGiZepysnJ0evyQStWrMDbb7+NlStXwtPTE926dXvpxLP//ve/estHRQUGBmrdV9hTrqmv5/l9r/swnF9//ZV/VFVw7NEkKoNvvvkGn332Ga5cuQIfHx+541AZubu7IyUlRW0ygqYezaysLFSuXBnu7u64fv26XrIpFApIkvTS5VsK20iSxHU0ZaZt7cdTp07hyy+/hL29PcaMGYP69evD0dERiYmJCA8Px/r165GSkoLPP/8cb775Jtq1a6fn5ETlhz2aRGUwY8YMXLx4ET169MCKFSvQs2dPuSNRGXTo0AG//fYb1q9fjzFjxmht98MPPyArK0v1mEp9+OKLL/R2LSofmgrES5cuYcGCBejfvz9+/fVXmJmZFWnzxRdf4N1338X8+fNx+vRpfUQ1WEqlEtHR0ahSpQpcXFzU9u3evRtr1qzB/fv30axZM3z55ZeoUaOGTElJG/ZoEpVB4Ris06dPIy8vD5UrV37p7czXebyVobt16xYaNWoEIyMjLF68GKNGjUK3bt1UPZqpqan48ccf8dVXX8Hc3Bw3btyAq6ur3LGpAunZsyeCg4ORkJBQ7ONDnz59CmdnZ7Rr1w779+/XY0LDMnfuXHz11VdYs2aN2h9/v/32G8aMGaPWw+/q6oqwsDC9rW1LJcNCk6gMSjs+k7czDd+OHTswatQo5OTkwMjICEZGRsjJyYGLiwsSEhJQUFAAU1NTbNu2Db1795Y7LlUwVatWhYeHB86fP//Sti1atMCdO3eQlJSkh2SG6c0338SFCxfw+PFjWFlZqba7u7sjNjYWM2fORKtWrfDDDz8gMDAQCxYswKeffipjYnoRC02iMijJ83dfxPFWhi8sLAxz587FoUOHkJWVpdpuYmKCgIAAfPXVV7IvDJ2ZmYnbt28jPT0d1tbW8PT05FNRKgArKyvY2dnh3r17L21bo0YNpKamIiMjQw/JDJOLiwtMTEwQHR2t2nb58mU0b94cHTt2xF9//QXg2VOBXFxc4OPjgwsXLsiUljThGE2iMmDR+Gry8fHBrl27kJubi4iICKSlpcHKygp16tSRvZg7cuQIFi5cWGSCkpGREfz9/fHpp5+ia9euMiak4vj6+uLcuXNYtWoV3n//fa3tVq9ejfv376NVq1Z6TGd4kpOT0bhxY7VtQUFBkCQJffr0UW2rUqUKvLy8EBMTo9+A9FJcl4WISAsTExM0aNAAb7zxBnx9fWUvMufOnYu3334bwcHByMvLg4mJCapXrw4TExPk5eUhMDAQ3bp1w9y5c2XNSdrNmDEDQghMnjwZQ4YMQVBQEBITEyGEQGJiIoKDgzF06FBMmjQJkiRhxowZckeWlampKR4/fqy2rXBNzbZt26ptt7CwwJMnT/SWjUqGt86Jysnp06cRFBSE+Ph4ZGVlqT3TPDo6Gjk5OfDy8pIxIZWWId2ePnz4MN5++20YGRlhwoQJmDJlCurUqaPaHxkZiR9++AG//PIL8vPzcfDgQb3OiqeSW7x4MWbPno2CggKN+4UQUCgUmD9//ms/3rBFixa4dOkSwsPD4eXlhZSUFLi6usLS0hKJiYlqbV1dXWFsbIy7d+/KlJY0EkRUJpGRkaJFixZCoVAIhUIhJEkSCoVCrc3EiROFQqEQwcHBMqWk0jh48KBo3769MDExUX1dFQqFMDY2Fu3btxd//vmn3jN169ZNKBQKsXHjxmLbbdq0SUiSJLp166anZPRvhISEiOHDhwtHR0chSZLq5ejoKIYPHy4uXbokd0SDsGzZMiFJkqhVq5aYPn26aNy4sVAoFOKjjz5SaxcdHS0kSRIBAQEyJSVtWGgSlcGDBw9E9erVhSRJokWLFuLLL78UderUKVJonjt3TkiSJKZMmSJPUCqxKVOmqP5gkCRJmJubC1dXV2Fubq7aplAoxH/+8x+95qpataqoWbNmidrWrFlTVKlSRceJqLykpqaKe/fuidTUVLmjGJy8vDzRv39/tWK8VatWRT5XX331lZAkSXz77bcyJSVtOEaTqAy+/vprJCQkYNKkSTh79iz++9//anxcWosWLWBtbY0zZ87IkJJK6tdff8WPP/4IY2NjTJs2DVFRUcjMzERsbKzqNvq0adNgYmKCn376CevXr9dbtvT09BI/is/R0ZFj1SoQW1tbuLi4wNbWVu4oBsfIyAg7d+7ExYsX8fvvv+PUqVM4c+ZMkc+Vh4cHli5diiFDhsiUlLThGE2iMvD09ERiYiKSkpJUT/jQ9MhCAGjSpAkePHiAhIQEOaJSCTRt2hShoaH4448/0K9fP63t9uzZg/79+6NJkya4dOmSXrJ5eHggKSkJCQkJqFSpktZ2T548gZOTExwcHHDnzh29ZKN/Jy4uDidPnkR8fDwyMzPx+eefq/bl5uZCCAFTU1MZExKVHXs0icogPj4ederU0fgYuReZmZkhJSVFD6no37p58yZq1apVbJEJAH379oWbmxvCw8P1lAwICAhARkYGxo8fj5ycHI1tcnJyMG7cODx9+hRvvfWW3rJR6SQlJWHQoEFwd3fHiBEj8Omnn2LevHlqbd59911YWFjo7Q8ZIl3hOppEZWBlZYVHjx6VqG1sbCyqVKmi40RUFtbW1iX+GlWpUgVPnz7VcaL/89lnn2H79u3Yvn07AgMDMX78eHh7e6NatWpITEzEjRs3sGbNGjx8+BC2traYNWuW3rJRyaWnp6Ndu3YIDw+Hq6srOnfujGPHjiE+Pl6t3bhx47B161bs3r0bzZo1kymt/DZu3FjqY0aOHKmDJPRvsdAkKoMmTZrg77//RlhYGHx8fLS2CwoKwoMHD9C3b189pqPSat++Pf73v//h8ePHqFy5stZ2ycnJuH79Onr16qW3bK6urjh06BAGDhyIuLg4zJ8/v0gbIQRq1qyJHTt28BnsBmrx4sUIDw9H//79sXHjRlhYWKBNmzZFCs22bdvCwsICJ06ckCmpYRg9ejQkSSpRWyEEJElioWlgWGgSlcHYsWNx/PhxjBkzBvv374ezs3ORNrdv38aYMWMgSRLGjx8vQ0oqqfnz5+Po0aMYNGgQtm7dCgcHhyJtHj16hKFDh8Lc3FxjsadLLVu2xM2bN7F161YcPXoUERERyMjIgJWVFby8vBAQEIAhQ4bIvrA8abdz506YmZlh7dq1xX6dFAoFateujdjYWD2mMzwjR47UWmg+efIEUVFRCA0NhYmJCd555x2YmJjoOSG9DCcDEZXRwIEDsXPnTtja2iIgIAD//PMP7t27h9mzZ+PatWs4ePAgcnJyMGLECPz2229yx6VibNy4EREREVi8eDGMjY3Rr18/1K9fH9WqVcOjR48QHh6OXbt2IT8/HzNmzNC6AD97VEgbCwsLeHl5ITQ0VLVN2wTC1q1bIyQkBFlZWfqOWaFcvHgRo0ePhoODA44eParXYjMlJQVRUVGwsLCAt7c3FIrip76EhoYiLS2tyFONXmUsNInKKC8vD//973+xbNkyZGdnq7ZLkqSaNTp16lQsWLAARkZGMiall1EoFKqvW6Hne1O0bX/RiwUDUSF7e3vY29urrQigrdB0c3NDZmYmHj58qO+YFU5kZCTq16+P2bNnF5lYpQspKSmYMGEC9uzZo3rCk729PaZNm4ZPPvkExsaabxi3adMG//zzD/Ly8nSe0VCw0CQqJ0lJSTh48CDCwsKQlpYGKysreHt7o3v37hpvqZPhKc14sOL8+uuv5ZCGXkX+/v44d+4coqKiUKtWLQCaC80rV66gadOmeOutt3Dw4EG54lYovr6+ePr0KaKionR6nZycHLRq1QqhoaF4sYSSJAlNmjTBnj17NI6T1vZHxauMYzSJyknVqlV5y7SC27Bhg9wRivX48WN89913OHToEO7cuYOMjAytbSVJ0luvSUFBASIjI/H48WPk5uZqbfc63S7UZvjw4Thz5gzee+897NmzB5aWlkXapKSkYOzYsZzYUkq5ublFJlXpws8//4wrV66gWrVq+PHHH9G1a1dkZWVh+/btmD9/Pi5fvow33ngDR48eRf369XWex9CxR5OIqAK4e/cu2rRpg4SEhCK9KNoU3tLTlUePHuHTTz/Fjh07XrrUkz4LX0OWn5+Pjh074uTJk3B3d8eAAQOwe/du3L59G2vWrMG1a9ewefNmJCUloWvXrjh8+LDckSuECxcuoHXr1nBxcUFMTIxOr/XGG2/g3Llz+Ouvv9ChQwe1fQkJCRg4cCBOnz6NqlWr4tChQ2rLU72OPZosNInKQVhYGH788UcEBQUhPj4e2dnZar9UV69ejZiYGHz66aewsbGRMSmVVkZGBtLT02FtbQ0rKyvZchROOvP19cX8+fPh5+eHatWqlcut/n8jOTkZfn5+iImJQY0aNZCWlob09HS88cYbiIuLQ3x8PPLz82FhYYEWLVoAwGu/VE+h9PR0vPfee9i+fbvamODn/3/gwIFYt25dsU+Beh0EBwdr3SeEwKNHj3DhwgWsWbMGaWlpmDZtGr799ludZrKzs4ONjY3WFQFyc3MxatQobNu2DTY2Nvjf//6HNm3aAHg9C03o8bnqRK+kFStWCBMTEyFJkuqlUCjU2vzwww9CoVCIzZs3y5SSSiMsLEyMGjVKODs7C4VCoXo5OzuLd999V4SFhek9U+XKlYWFhYV4+PCh3q+tySeffCIkSRIffvihEEIIf39/te/75ORkMWvWLGFqaipGjRolU0rDdvXqVTFv3jzxzjvviC5duoi+ffuK2bNni4sXL8odzWAU/jwt7lX4c7dLly7iyZMnOs9kamoqWrRo8dJ2kyZNEpIkCUtLS3Hw4EEhRNF/J68D9mgSlcGJEyfQuXNnWFtbY8GCBejVqxeGDBmCf/75R+0v1gcPHqB69ero168fdu7cKWNiepl169Zh0qRJqmdNa2JqaoqffvoJY8eO1VuuSpUqoW7durh8+bLerlkcb29vxMTEICEhATY2Nlp7ajZu3Ih3330Xy5cvxwcffCBTWqqo2rdvr7XXXpIkVKpUCR4eHujWrZveHrvq6uqK/Px83L9//6VtZ8+ejYULF8LMzAy//fYbli9fzh5NIiq57t27C4VCIQ4cOKDapu0v1po1a4r69evrMx6V0tmzZ4WRkZGQJEl0795dHD16VMTHx4u8vDwRHx8vjh49Krp37y4kSRLGxsbi3LlzesvWrFkz4e7urrfrvYylpaXw9vZWfdy2bVuhUChETk5OkbYuLi6iadOm+oxHpDM9e/YUCoVCXLt2rUTtFy9erPqZYWtr+9r1aBa/sigRFevs2bNwcnJC9+7dX9rW2dlZLzMi6d/79ttvIYTA119/jQMHDqBLly6oXr06jIyMUL16dXTp0gUHDhzAokWLkJ+fr/OxYM+bOnUqoqOjcfToUb1dszgmJiZqM6atra0BPOu9f5GzszMiIyP1lq0iSUlJQVxcHGJjY7W+yLB07NgRQgisW7euRO1nzJiBVatWQQiB9PR0HaczPCw0icogIyMDTk5OJWqbk5Pzet0uqYBOnToFBwcHfPrpp8W2mzFjBqpVq4aTJ0/qKdmzZXE+/fRTDBo0CD/88IPsv7Bq1KiBhIQE1ceFT0l68XPy5MkTREZGyjZpSZuHDx8iJCTkpbPldSEiIgJDhw5F5cqVUbVqVbi5ucHd3V3jy8PDQ+/5qHgDBgyAi4sL/vzzT6SmppbomPfeew+///671oXcX2Uco0kVyv379xEfH4/MzEyDWJPP3d0dKSkpaj9sNI1Vy8rKQuXKleHu7o7r16/LkJRKwszMDI0bN8a5c+de2rZly5YIDQ3V2+MBCwuOe/fuqb63qlatqnVWsiRJuH37ts7yjBkzBps2bcKjR49gZ2eHEydOoFOnTnB2dsZvv/2G1q1b4+HDh5g+fTr279+PLl266HWpnnPnzmH79u3o1KmT2h0HpVKJESNG4MCBAwCejX394Ycf8O677+ol15UrV9CuXTtkZGRACAFzc3M4ODgU++jCu3fv6iWbIdq4cWOJ2xoZGcHa2hpubm5o0KABn8RmKOS8b09UUj///LOoXbu2apahkZGR2v5p06aJ1q1bi5iYGL3mevfdd4VCoRDr1q1TbdM0RnPRokVCkiTx0Ucf6TUflU6NGjVE5cqVRW5ubrHtcnJyROXKlYWLi4uekgm1VQ1K8tL1OLC9e/cKSZLExo0bVdv69OlTZJawJEnC3NxcXLhwQad5XjRu3DihUChEUFCQ2vb33ntPSJIkjIyMROXKlVX/f/XqVb3k6tatm5AkSXTu3FmW1QsqmpLMOtf0qly5spgxY4ZeZqFT8VhokkErKCgQAwcOVP3w8PDwEDY2NkV+iW7fvl1IkiSWLl2q13w3b94UZmZmwtLSUqxYsUKkp6erFZopKSli3rx5wtjYWFhZWYnY2Fi95qPSGT58uFAoFGLatGnFtvvoo4+EQqEQI0aM0FMyIaKjo0v90qX8/Hxx7949kZaWptqWk5Mj5s2bJ+rWrSvMzMyEnZ2d6NGjh7h06ZJOs2hSv359YW1trbYtPT1dWFhYCBsbGxEeHi6EeLb0mCRJeluCydbWVlhbW4uMjAy9XK+iGzVqlBg6dKgwNTUVkiQJDw8P0bt3bzF8+HDRu3dv4eHhISRJEmZmZmLIkCFiwIABwsfHR1WgtmzZUmRmZsr9Nl5rLDTJoK1Zs0ZIkiQaNmyo6nHQ1GOYkZEhjI2NRefOnfWecfv27cLc3FwoFAphYmKi+n9XV1dhbGwsFAqFMDc3F3v37tV7Niqd69evq75+zZo1E+vXrxdnz54Vd+7cEWfPnhXr168XTZs2VX1Nr1+/Lndk0qJy5cpqs+KFEOLAgQNCkiQxfvx41bb8/Hzh4OAg6tWrp5dc1tbWonnz5nq51qvgyZMnokWLFsLNzU0EBgZqbBMUFCTc3d1FixYtVD2Y58+fF25ubkKhUIjvvvtOZ/mePn0qdu/eLT755BPRu3dv0bZtW+Hn5yc6dOgghgwZIr777jtx8+ZNnV2/ImChSQatVatWwsjISNX7IIT25YPq1q0rPDw89BlP5erVq6Jfv37CwsJC7falqamp6Nmzp7hy5Yosuaj09u3bJ2xtbbXespMkSdja2or9+/fLHZWKYWJiUqSg+/TTT4VCoRA7duxQ2+7n5ycqVaqkl1xt2rTR65CLim7mzJnCyMhI3Lhxo9h2169fFwqFQsyYMUO17dy5c0KSJOHn51fuufLz88X8+fNVyxVpW0S+8OOuXbuKyMjIcs9REXAyEBk0GxsbODk5ISIiQrVN28LQrVu3RmhoqCyzSAvl5uYiIiICaWlpsLKyQp06dWBhYSFbHvp3EhIS8NNPP+HYsWOIiIhARkYGrKys4OXlhYCAAEycOBHOzs6y5bt7964qW+HjMb28vNClSxe4u7vrPU9cXBxOnjypmqj3+eefq/YVLnxvamqq10zOzs7Iz8/Hw4cPVTPe/fz8cPnyZSQkJKBatWqqtk2aNEFMTAweP36s81yHDh1Cjx49sGHDBowYMULn16voPDw8YGVlhatXr760baNGjZCeno47d+6otrm7u+Px48dIS0srt0xCCPTp0wcHDhyAEAIuLi5wcXFBfHw84uPjIUkSBg4ciNq1a+PixYsIDAxEdnY2rKyscODAAYOYyKpXspa5RC9RqVIl0bBhQ7Vt2no0vb29hZ2dnb6iCSGEcHNzE76+viI7O1uv131VxMfHi/PnzxeZsCGXmJgYERMTI/Lz8+WOotHjx4/F4MGDhZGRkVrPyfOT5IYOHSoeP36slzyPHj0SAwcOVMvz4r/NYcOGCYVCoffHKvbp00coFAqxevVqIYQQx44dE5IkiSZNmqi1KygoEFZWVkVus+vSqlWrhJWVlZg6daoICwsTT58+1du1Kxpzc3PRqFGjErVt1KiRMDc3V9vWokWLItvK6ueff1YN6Tp79qzavnPnzon69esLc3NzERoaKoR49u928uTJQpIkUaVKFYN5jKy+sNAkg9awYUNhbm4u0tPTVds0FZoJCQnCyMhItG7dWq/5LC0tdXJb5lVnqKsISJIknJycDLLQfPr0qWjSpImquHzjjTfE+PHjxZw5c8T48ePFG2+8oSo6mzZtqvMJEEqlUnh7ewtJkkTNmjXFmDFjhKura5F/mydOnBCSJInPPvtMp3ledPLkSVUBXKVKFdX/b968Wa1dYGCgXicDlXb29Iv/Nl43bm5uwtjYWNy6davYdrdu3RJGRkbCzc1Nbbuzs7OoXr16uWby8/MTJiYmWn8+FWbp27ev2vaZM2cKSZLEJ598Uq55DB0LTTJon332WZFlgTQVmkOHDhUKhUIsXrxYr/l8fHxEnTp19HrNiszQVxGws7MTLVu21Os1S2rBggVCkiRRv359rUsFXbhwQXh7ewuFQiEWLlyo0zxz5swRkiSJd955R9Ujp+nfZn5+vrC0tNT7H4FCPFuCqfCP1Tp16oiff/65SJvBgwcLSZLEli1b9JKptMtUSZKkl1yGasaMGUKSJFGnTh3xzz//aGxz9uxZUadOHaFQKNSKuPj4eCFJkujQoUO5ZrK2ti7SM/6iunXrisqVK6tty8jIEBYWFq/do4hZaJJBe/z4sXBxcREKhUK888474tChQ6J58+ZCoVCIO3fuiH379olOnToJSZKEp6en3pcM+eabb4RCodDrM68rMkNfRaBNmzbC2dlZr9csqUaNGgljY2Nx+/btYttFRUUJY2PjEt9u/Lfq1asnzM3NRWpqqmqbtmEtvr6+BjsBRqlUitTUVIPsxaZnS1I1adJE1Vtft25dMWDAADFmzBgxcOBAUa9ePVUvf9OmTdV+BxR2VHz77bflmsnKyqrIkK4Xubu7C0tLyyLbmzRporeJZ4aChSYZvGvXrglPT89iZwF7enrKsoREXl6e6NGjh3BychJ79+4VBQUFes9QkRj6KgI7d+4UkiSpLcBvKCpVqiQaN25coraNGzfW+S8zc3Nz4evrq7ZN29eyVatWwszMTKd56NWlVCrFpEmTiqzqUfiysLAQkydPFkqlUi95mjVrJoyMjLQuuF84271BgwZF9jVo0EDY2trqOKFhef0eukkVToMGDXD16lWsW7cOe/bsQVhYmGpWt7e3N/r164cJEyZofRSfLnXp0gVCCCQlJaFfv36wtbVFnTp1in0s4PHjx/Wc0nBcv34dHh4eqFev3kvb2tvbIzQ0VA+p/k///v2xaNEiTJo0CWFhYRgxYgTq169vECsHGBkZITc3t0Rtc3Nzi32kYXkwNzcv8fPWExISYGtrq9M89OqytrbGihUr8PXXX+PkyZOIjIzEkydPUKlSJXh5ecHf3x82NjZ6yzNkyBBcvnwZ3bp1w9KlS9GnTx8YGxsjLy8P+/btw9SpUyFJEvr37692XH5+Pu7evfvaPb+eyxsRlUFpf5lLklRkWabXiZWVFdzd3REWFqbapm25qgYNGuD+/ftISUnRW77SPhtZkiTk5eXpKI26Fi1a4NKlS7h8+TIaNWqktd2VK1fQtGlT+Pn5leiZ7f+Wv78/zp07h6ioKNSqVQuA5q9lYZ633noLBw8e1Fkebe7evYvt27cjNDQUjx8/1lqsv+5/BFLJ5eTkoG3btjh//jwkSYJCoYCDgwOSkpKQn58PIQTq1auHc+fOwdraWnXcvn370LdvX7z//vv4+eefZXwH+sUeTaIyOHHihNwRKhR3d3dERUWp1qXU5sGDB7h16xZatGihx3TP1sfTZfuyGDFiBC5evIgePXrg559/Rs+ePYu02b9/PyZPngxJknS+RuPw4cNx5swZvPfee9izZw8sLS2LtElJScHYsWMhSRJGjhyp0zyafPvtt5g9ezby8vJUa2k+/zV7flvh/+taaf6YUSgUsLa2hpubG/z9/TFu3Dj4+vrqMB2VhKmpKf766y98+OGH2LRpE/Lz8/HgwQMAz76n+vXrh5UrV6oVmQDg6emJPXv2FPuH4quIPZpEpDezZ8/GwoULMXXqVHz//fcANPeCDRs2DNu2bcOiRYswY8YMueIalLy8PAQEBODEiROQJAk1a9ZEvXr1UK1aNSQmJiI8PBxxcXEQQqBjx444cuRIqXtoSyM/Px8dO3bEyZMn4e7ujgEDBmD37t24ffs21qxZg2vXrmHz5s1ISkpC165dcfjwYZ1l0eTgwYPo0aMHnJ2d8dVXX2HZsmW4fv06jh49iri4OISGhmLdunXIz8/HokWL4Ovri3bt2uk8V1mGNBgZGeHrr79+rf5NBAcHl/oYfS6I/vjxY5w7dw4pKSmwtbVF8+bN4ejoqLfrVwgyjQ0lKrGcnByxevVq0adPH+Hr6ys8PDyEu7u7xpdcj6CkkjH0VQQMXWZmppg+fbqoVKmSxkkRlSpVEh9//LHO19AspFQqVcsDvfjYvcL/HzRokCxfx7feeksoFApx6tQpIYTmiUpJSUmiXbt2wtbWVkREROgt29KlS4W5ubkYPXq0CAoKEikpKSIvL0+kpKSI4OBg8e677wpzc3OxdOlS8eTJE3Hp0iUxadIk1Vqgf/31l96yyk3bJFCuO1pxsEeTDFpSUhI6duyI69evl+g25es+BrIiuH79Onr37o07d+5ovF0phICHhwf+/PNP1K1bV4aEhi89PR2nTp0q8nhMf3//Irfr9CEsLEzjRL2+ffuiWbNmes8DANWqVYOpqSnu3bsHQPtY4ISEBNSqVQsDBw7E5s2bdZ5r165dGDhwIFasWIGJEydqbbdy5UpMnjwZ27Ztw4ABAwAA33//PT7++GP07t0be/bs0XlWQ9C+fXutwxqePHmC27dvIyUlBaampmjdujUADmkyNCw0yaCNGzcO69evR40aNfDJJ5/Az88P1apVK/b2U+HEBH3o2LFjidsaGRmpxlu9+eab6Nmzp86f/3z06FEcOnQId+7cQUZGhtZiXd8TIZ4+fWqQqwhQyW3cuBEAMGjQIJiZmcmcpihzc3M0atRINSGqS5cu+Pvvv5GSklJkhnLTpk3x8OFDxMfH6zxX69atERcXpyqAi1OjRg3UqFEDZ8+eBfBs+ETVqlVhYWGBhIQEXUetMHbt2oUpU6agXbt22LJli96vHxISgrNnz+LWrVtISUlBZmYmrKys4OTkhGbNmqFz586v96oLMvamEr2Uo6OjMDU1FZGRkXJH0ej5W4XP3y58/qVpn0KhEK6uriIwMFAnuZ48eSICAgKKzfViHuLjAUtDoVCI2rVryx1DK1dXV1GvXj3Vx4VPDzt//nyRtnXq1NHbOp9WVlaiRYsWJWrbokULYW1trbbNz89PmJqa6iJahXb+/Hm9P01s8+bNao/SfXF958L/t7S0FO+9955ITk7WWzZDotuF1ojKKC0tDXXr1kXt2rXljqLRiRMnsGDBAhgbG8PT0xNz587F7t27cezYMezevRvz5s1DnTp1YGJigq+//hoHDhzAt99+i4YNG+LevXvo0aMHIiMjyz3Xf//7Xxw9ehTW1tb46KOP8Pvvv+P48eM4ceKExtfff/9d7hkqIvHsIRYlfhUUFOgt2/79++Hh4YElS5YU227JkiXw8PDQ+VJCDg4OsLe31+k1ysLT0xP3799XfdyyZUsIIbBy5Uq1dsePH0dUVBRcXFz0ksvExAQRERHIzs4utl12djYiIiJgbKy+OIxSqZRleISh8/Pzg5eXF9asWaOX602aNAkjR47E7du3IUkSHBwcIEmS6q5Rr169MHToUHh5eSEzMxNr166Fj48PwsPD9ZLPoMhR3RKVVMOGDUXdunXljqFVSEiIsLS0FKNGjRK5ubka2+Tl5YnRo0cLCwsLcenSJSHEs+c/Dx8+XEiSJMaPH1/uuVxdXYWxsTEfjVmOnjx5IkJDQ8XkyZNFpUqVxOrVq/V6/f79+wuFQvHSR1BGRkaqJuHoOo+NjY3eJh6V1qJFi9R6MB89eiTs7OyEQqEQb775pvj444/FiBEjhJmZmVAoFOKLL77QS64ePXoIhUIhxo0bp/WxlwUFBWL8+PFCkiTRs2dP1fbs7GxhamoqfHx89JK1ovHx8REWFhY6v87vv/8uJEkSLi4uYvv27SInJ0cI8Wzi6vbt24WLi4uoXLmyiI6OFkI8e7pdz549hSRJombNmiI9PV3nGQ0JC00yaMuWLRMKhUJcvHhR7iga9erVS9ja2r70l21mZqawtbVV+6Xx+PFjYWpqKtzd3cs9l7m5uahfv365n7c8vAqrCGzYsEEoFApx8OBBvV3Tw8NDODk5laitk5OT8PT01GmeK1euCHNzczFx4kSdXuffunPnjhg3bpw4fPiwatuRI0eEvb19kaEjAwYM0PqHYnm7fPmyMDc3FwqFQnh7e4uFCxeKP//8UwQHB4uDBw+KRYsWiYYNGwqFQiHMzc1FSEiI6tht27YJSZLEhx9+qJesFUlkZKQwMTEp8b+Rsmjbtq0wMjISV69e1bj/7NmzQpIkMXLkSLXto0aNEgqFQnz11Vc6z2hIOBmIDJoQAiNGjEBQUBBWrFiB3r17yx1JjYODAzw8PEr0BJaWLVvi9u3bSEpKUm1r3Lgxbt26hczMzHLNVadOHVhYWODq1avlet6yepVWEXBxcYGnp+e/Wufv37CwsICvr2+JvtdatGiB69ev48mTJzrLExwcjKCgIHz55Zfw9fXFsGHDUL9+/WIncelzfUNt0tLScOjQIURHR8PCwgJt2rRB06ZN9Zrhr7/+wogRI/Dw4UOtKy84OTlh06ZN6NSpk2p7YGAgYmJi0KZNm9fmMYaxsbFa9wkh8OjRI1y4cAGLFy9GbGwsxo4di19++UWnmezt7VGjRg21J5y9yMPDA0+fPlUt5A48W3PTyckJDRo0QEhIiE4zGhIWmmQwipvBffr0aeTl5cHe3h6enp4G8yxxKysr2NvbIy4u7qVtXV1dkZKSgoyMDNW2Jk2a4M6dO0hLSyvXXHPnzsX8+fNx69YteHp6luu5y8LQVxEojebNmyMiIgJKpVIv16tatSpsbGxw586dl7b18PBAamoqHj9+rLM8CoVCbUzay56so8/HdQLPCpSaNWvq7XqllZGRga1bt+LYsWNFnt3dpUsXDBkypNinZ70uCr/PXkYIgQYNGuDEiROoWrWqTjNVqlQJdevWxeXLl7W2qVevHmJiYop0IjRq1AjR0dHl/jPfkLHQJINRlidmFNJ3D1irVq1w4cIFrFq1CuPHj9fabu3atXjvvffQsmVL/PPPP6rtNjY2cHR0LPcJQTk5OejatSuSk5OxceNGNGnSpFzP/285OTkhJSUF169fN9gJXiXx5MkTODk5wcjICKmpqXq5ZocOHRAcHIxz586hefPmWttdvHgRLVq0gL+/v057W4tb31Abfa5vaGRkhFq1aqFt27Zo164d2rZta1B/dFHJuLm5af0+kyQJlSpVgoeHB7p164Z3331XL0tt+fj44NatW7h9+zZcXV2L7I+MjET9+vXh7u5e5Gd748aNcefOHb39gWoI+KxzMhgVcZHdadOmYfDgwZg4cSIuXbqE0aNHw9fXF5aWlsjMzMTVq1exYcMGrF27FpIkYfr06apjT506hYyMDJ0MB3j//ffh6uqKf/75B35+fmjcuPFLe4LXrVtX7jleZOirCJREeHg4pk2bhqdPn+Ktt97S23WHDh2KoKAgDBs2DIcOHdJ46/Tu3bsYNmwYJEnC0KFDdZonMDBQp+cvK1dXV0RHRyM6OhqbNm0CADg7O6Ndu3aqwrNevXoypzRcQghcvXr1pWvwAtDpc+yjo6N1du5/q2/fvpg/fz569OiBtWvXws/PT7Xv4sWLGD16NIQQ6NmzZ5Fj7969i+rVq+szruzYo0lURl999RXmzZun9oP4+VuKQghIkoR58+Zhzpw5qjbr16/HyZMnMXr06HJ/xvKLtzVfRl89wT4+PsjNzcXNmzd1fq1/o7hxb4XjwTIzMyGEgJWVFU6ePIlGjRrpJVt+fj7atWuHM2fOwNzcHP369UPLli1hZ2eH1NRUnD17Fnv37kVmZibeeOMNBAUF6fRZ5yUVHx+PLVu2YPPmzXofMxwXF4egoCAEBQUhODhY1btU2EPm4OCg1uPp4+Oj13yGauvWrZg5c6ba8lDFMYRx1Pr8PlMqlWjevDmioqIgSRJq1KgBFxcXxMfH4969exBCwNnZGSEhIahWrZrquL///hudO3fGqFGj8Ouvv+o0oyFhoUkGpWPHjvD19cWyZcvkjlIqFy5cwPfff4/jx4+rTfapWrUqunTpgqlTp6r91atrv/32W6mPGTVqlA6SqPvhhx8wbdo0nD9/XrZHExanJMM3bG1tERAQgHnz5un9EZmpqal49913sW/fPgDq4yILf5T37dsX69atg52dnV6zPS8jIwO7du3Cpk2bEBgYqMomd0Hy4MEDBAYGIjg4GMHBwWprGupiDGnhuPNatWqpCovSPE2sMJc+x53/8ccfGDRoEIBnQ10aNWr00nHUchVNcn6f3b9/H8OHD9fYs9+sWTNs2bIFXl5eattPnz6Nixcvon379nr7A9UQsNAkg6JQKHQ+tkzXUlNTVQP75fxlb4gMfRWBmJgYrfsKx4NVqVJFj4k0u3jxIvbt24fw8HDVAt4NGjRAnz599D6DulBBQQGOHj2KTZs2Yd++faqeX+DZpLfhw4fjo48+kiXbi+7fv48TJ05g586d2L9/v+quQ3kXKIXFWb169XDjxg21bSWl73HnzZs3R0hICGbOnIkvv/yyyILxcjO077OrV6/i9OnTSElJga2tLfz8/NCiRQu9Xb8iYKFJBuVVKDQNUUREBCIiIpCeng5ra2t4eXkV+Wu7vFXEVQSo9EJCQrBp0yb8/vvvSExMVP3SNzMzw7Rp0zB8+HDUr19f1oyxsbGqW+hBQUGqmftCCDg6Oqpun3/wwQflet2goCAAgKWlpeqORuG20ijvoTXFsbS0hI2NjdqyPIagInyfkWYsNMmgVNRC88mTJ/jf//6H0NBQPH78GLm5uRrb6WvSTaHVq1fjm2++0dhTV6tWLcyaNavY2fJlURFWEdi4cSMcHR0REBBQZJ9SqYSxsTEsLS01HrtixQrcuXMH33///WuX7d69e9iyZQs2bdqkuv0shIC9vT0GDBiAX375BU5OTiUe46cLv/76q6qwjI2NVRUmNWrUUBuXqe/hD4bOwcEB7u7uOH/+vNxRKsT3WWnFx8cjPz/foJfeKm8sNMmgVMRCc9u2bZg4caLachWa1hbU1e05bd59911s3LgRQgiYmZnB1dUVjo6OePjwIeLi4pCdnQ1JkjBy5EidjLH6Nz03muiyN0ehUKBNmzYasxa3DwDatGmDM2fO6OzraWjZMjIysHPnTmzatAlBQUGq571bWFigR48eGDZsGLp16wYTExMoFArZC4DCCXFOTk7o2rWrara5u7u7bJleRqlU4tChQ7h//z6aNm2q157MQgMGDMCxY8eQmJgIU1NTvV+/on2flZaDgwNSUlL0uqas3Axr8AVRBfPPP/9gxIgRsLCwwOzZs7F9+3ZERUVhzZo1iIuLQ2hoKP73v//BzMwMc+bM0duyFlu3bsVvv/2GSpUq4YsvvsD777+vtvhzRkYGVq1ahS+//BIbN25E165dMWTIkHLNoOmXZHBwMGxtbUs0EP7q1at6WaOyuL+15f473JCyOTo6IisrC0IIGBkZoVOnThg2bBj69etnsAuLCyGQmJiIsLAwVK5cGVWqVIGdnR3s7e1ly7R9+3Z88803+OCDDzBu3DjV9ps3b6Jr166Ij49XbRsxYgQ2bNig13zz58/HkSNH8Mknn8gyKbMifp+Vltw/V/SuPJ5jSVReJEkSCoXiX7+MjIz0mrdfv35CoVCI/fv3CyGE8Pf3FwqFQq1NeHi4aNiwoXBxcREPHjzQS6727dsLhUIhjhw5Umy7I0eOCEmSRIcOHfSSS5Ik0bZt2xK1bd++vc6/npIkiTZt2pR6nxCav9blydCyFf7brFy5sti6dasoKCgotq2zs3O5Xr+0zp49KxYvXiy6d+8u7OzsVPmNjIxEo0aNxIcffih27dolHj16pNdcffv2FQqFQty6dUtte0BAgJAkSdSuXVv07dtX2NjYCIVCIf7880+95gsKChJLliwRpqamomnTpmL58uXi4MGDIigoSOurPFW077PSqlq1qk5/bhgi3jong1LWcX36nqFZvXp15Ofn4+HDhwC037KMiIhA/fr1MX78eKxatUrnuQp7b0ryxCEvLy88evQIKSkpOs9VmqERhU/C0eXXs7g8L8uqj1vnhpTNx8cH169fB/Ds35mzszMGDRqEoUOHFlmqytBuaQohcOXKFQQGBiIoKAinTp3C48ePVUNb6tevj/bt22PFihU6z+Lp6QmlUolHjx6ptiUkJKBGjRpwdXXFzZs3YW5ujuDgYLRv3x7dunXDn3/+qfNcheR+tGhF+D77+uuvy3RsZmam7Et96RNvnZPB8fHxwY8//ih3jBJJTk6Gr6+v6uPCMU2FyxsV8vLyQoMGDXDo0CG95MrKyirx0ko2Nja4d++ebgP9C8nJybCwsJA7Bv1/YWFhCA0NxcaNG7Ft2zbcv38fy5Ytw7Jly1C7dm0MGzYMQ4cONcinPkmShCZNmqBJkyaqpW/Onz+PhQsXYv/+/bhx4wbCw8P1Umg+evQIderUUdt24sQJCCEwdOhQmJubAwDatm2LWrVqqa31qQ9t27Yt9aNFy1NF+D6bM2fOv/4cif8/Vv91wkKTDI6tra0sg+D/jSpVqiAzM1P1cdWqVQEAt2/fVitAAaj1fOpazZo1ce3aNSQlJakyafLo0SNcv34dtWrV0kkOpVJZZJxldnY24uLitI5TyszMRFBQEK5du/ZaLWpcETRq1AhLlizBt99+i7/++gu//fYb9u3bh8jISMybNw/z5s2TbR3Pl3ny5AlOnz6tmol+8eJF5Obmqr4P9TXxJScnp0hv1smTJyFJEjp06KC23dHREaGhoXrJVcgQHi1q6N9nRkZGKCgo+FfjRrdt24acnBwdJTNQ8t21JyrqZWPPDE3r1q1F5cqVVR8vWrRISJIkPvnkE7V2V65cEcbGxqJGjRp6yTVjxgwhSZLo2LGjSExM1Njm4cOHokOHDkKhUBTJW17mzp2rNoa2NGNwJUkSy5cv10muQoY2DrKk15c72/MyMjLEhg0bRKdOnYSRkZGQJElIkiSMjIxEx44dxa+//iqUSqVesjwvNTVVHDhwQMyYMUO0aNFCmJiYqL6vJEkSlpaWokOHDmLu3Lni77//FpmZmXrJ5eHhIaysrMSTJ09U29zc3ISpqanaNiGE8Pb2Fo6OjnrJZegM6fusUaNGJRoDr8nrOEaTPZpEZdClSxecO3cO169fR4MGDTB06FDMmzcP3333HeLj49G6dWs8fPgQP//8MwoKCtC/f3+95Pr000+xbds2BAYGolatWhgwYAC8vb1RrVo1JCYm4saNG/jjjz+QlZUFV1dXzJw5Uyc57Ozs1NaLi42NhampKZycnDS2lyQJFhYW8PDwwKBBgzB8+HCd5HpeYmIiNm7c+K/26ZohZytUqVIljBo1CqNGjcL9+/exefNmbN68GdeuXcOJEycQGBiISZMmoVevXvj999/1lqtq1aooKChQ9VhaWVnhjTfeUK2f2aJFC5iYmOgtT6HOnTtj7dq1+M9//oOPPvoIO3fuRExMDAICAtTWRc3MzERkZKROe/VjY2MBACYmJnB2dlbbVhr6WBPSkL7PWrRogbCwMFy8eBFdu3bV6bX05ZNPPkFSUpJu1nqWu9Ilel5F69G8du2a6Ny5s9i1a5dq24YNG4Spqala750kSaJ169YiPT1db9kiIyNF8+bNVX/5v9hbKEmSaNGihYiKitJbJkP7+pZllYPCY1/HbCUREhIiPvroI+Hs7CxLHnt7e9GzZ0/x3XffifPnz4u8vDy9Xl+bmJgYVa9W4dfK1NRUnDt3Tq3d9u3bhSRJ4qOPPtJZlsKvi7e3d5FthrrSx4vk+D5bs2aNkCRJ9OrVq9THVqlSRfZ/m5q4ubnp7PPHHk2iMmjQoAGOHTumtm3UqFFo06YNduzYgejoaFhYWMDf3x99+vSBkZGR3rLVrl0bFy5cwPHjx3H06FFEREQgIyMDVlZW8PLyQkBAQLGPidSFX3/9FY6Ojnq9ZnFq1qxpsAPzDTlbSTRu3BiNGzfGd999h6NHj2Lz5s16vX5ycrJBfv5q1qyJixcv4rvvvkNUVBRcXV0xadKkIj2XgYGBaNSoEXr37q3TLIUzu1/cVlHI8X3WuXNnTJkypdjx79rs379f65Pj5PTOO+8gKSlJJ+fm8kZEREREpBNlfxgxEREREZEGLDSJiIiISCdYaFKFlZ2djblz5yI7O1vuKGoMNRdguNkMNRdguNkMNRdguNkMNRdguNkMNRdguNkMNRcgTzaO0aQKS6lUwtbWFmlpabCxsZE7joqh5gIMN5uh5gIMN5uh5gIMN5uh5gIMN5uh5gIMN5uh5gLkycYeTSIiIiLSCRaaRERERKQTXEeT9KagoAD379+HtbV1uazTplQq1f5rKAw1F2C42Qw1F2C42Qw1F2C42Qw1F2C42Qw1F2C42Qw1F1C+2YQQSE9PR/Xq1aFQaO+35BhN0pt79+7B1dVV7hhERERUTuLi4lCjRg2t+9mjSXpjbW0NAPjmm29gbm4ucxp1a9askTuCVvXq1ZM7glZt2rSRO4JGf/zxh9wRtMrKypI7gkZ5eXlyR9CqcuXKckfQKCYmRu4IWjVp0kTuCBqFhobKHUErQ+13M9QOmry8PJw8eVL1u10bFpqkN4W3y83NzWFhYSFzGnX6fDRkaZmYmMgdQStD+zoWMjY23B9thprNUH/JAob7OePPjdIz5M+Zof4bMNTv/0IvGwrHyUBEREREpBMsNImIiIhIJ1hoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmkRERESkEyw06aXc3NwgSRI2bNggdxQiIiKqQFhoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREemEsdwB6NWVnZ2N7Oxs1cdKpVLGNERERKRv7NEknVm4cCFsbW1VL1dXV7kjERERkR6x0CSdmTVrFtLS0lSvuLg4uSMRERGRHvHWOemMmZkZzMzM5I5BREREMmGPJhERERHpBAtNIiIiItIJFppEREREpBMsNImIiIhIJ1hoEhEREZFOsNCkEvvPf/6DqlWran1du3ZN7ohERERkQLi8EZVYRkYGMjIytO7Py8vTYxoiIiIydCw06aWio6PljkBEREQVEG+dExEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnJCGEkDsEvR6USiVsbW3RpEkTGBkZyR1HzZEjR+SOoNW4cePkjqBVXFyc3BE0sre3lzuCVvfv35c7gkb37t2TO4JW9evXlzuCRkOHDpU7gla//PKL3BE0cnd3lzuCVnfv3pU7QoWSn5+P8PBwpKWlwcbGRms79mgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSYRERER6QQLTSIiIiLSCRaaRERERKQTLDSJiIiISCdYaBIRERGRTrDQJCIiIiKdYKFJpRYYGAhJktC+fXu5oxAREZEBY6FJRERERDrBQpOIiIiIdIKFJhERERHpBAtNIiIiItIJFpoldO3aNXzxxRdo3bo1nJ2dYWpqCmdnZ/Tr1w9nzpzRelx8fDymTZsGb29vVKpUCba2tvDx8cHHH3+MyMjIIu2fPn2K7777Dq1atYKdnR0sLS1Rp04djBgxAkFBQUXaP3nyBPPnz4evry8qVaoEGxsbtGzZEj/99BPy8vKKtH9+Ik9eXh4WL14MHx8fWFpaws3NTa3tnj178MYbb6BSpUqoUqUKevTogYsXL5b+k0dERESvJWO5A1QUU6dOxfHjx2FnZwdnZ2dUr14dsbGx2LNnD/bv34+NGzdi6NChasccP34c/fr1g1KphImJCerXr4+CggLcuXMHS5YsgZWVFebOnatqHxsbi7feegvh4eEAgDp16sDa2hrR0dHYvHkz4uLiEBgYqGr/6NEjdOrUCWFhYVAoFGjYsCFyc3Nx/vx5nD9/Hvv27cP+/fthbm5e5P0IIdCnTx/8+eef8PT0hLe3N7KyslT7Fy9ejJkzZwKA6v0GBQXB398fc+bMKcfPLBEREb2q2KNZQu+//z6uXr2KlJQU3LhxA5cuXUJiYiL27t0LCwsLTJw4Eenp6ar2sbGx6N+/P5RKJUaOHIkHDx4gNDQUYWFhSE9Px4EDB9CsWTNV+/z8fPTr1w/h4eFo3rw5bty4gYiICFy6dAnJyckICQnBoEGD1DJNnDgRYWFhaNCgASIiIhAaGoobN27gwoULcHR0xLFjx/DFF19ofD+nT5/GhQsXcObMGURFReHixYuq3sqQkBB89tlnkCQJK1asQHx8PC5evIiEhAT06dMHX375pQ4+w0RERPSqYaFZQu+88w58fHzUtkmShN69e2Pq1KlQKpX43//+p9r3zTffIC0tDZ06dcKGDRtQuXJl1T6FQoHu3bujZ8+eqm27d+/GpUuXUK1aNRw+fBj169dXu1bjxo0xceJE1ceRkZHYvXs3AGDTpk3w9PRU7WvevDmWL18OAPjpp5/UCuBC+fn5WLlyJVq3bq3aVtjz+f333yM/Px/vvPMOJk2aBEmSAABWVlbYsGED7O3tS/Q5y87OhlKpVHsRERHR64OFZinExsZi0aJFGDhwIDp27Ah/f3/4+/tj+/btAIDQ0FBV23379gEAZsyYoSrUilPYfsyYMahSpcpL2x87dgxCCPj7+6NJkyZF9vfv3x81atTAkydPcPr06SL7bW1t0bt3b43nPnr0KACoFbaFzM3NMWbMmJfmA4CFCxfC1tZW9XJ1dS3RcURERPRq4BjNEvrtt9/w/vvvq41jfNHjx48BAOnp6YiPjwcAtGrVqkTnLxyXWdL2ERERAABvb2+N+xUKBerVq4d79+4hIiICb731ltr+OnXqwMjIqMhxqampSExMBIAivaqFtG1/0axZszBt2jTVx0qlksUmERHRa4Q9miVw+/ZtjB8/HllZWZg+fTpCQkKgVCpRUFAAIQTWrFkDAMjNzQUAtVvEtra2JbpG4TF2dnYlap+RkQEAqFatmtY2jo6OAKDx1nmlSpWKPS8AODg4FHvelzEzM4ONjY3ai4iIiF4f7NEsgR07diA3NxeDBw/Gd999V2R/XFyc2sfW1taq/09LSytRsVl4TGpqaokyWVlZAYCq91GThw8fFslT0vMCz2a1Ozk5FWlT3DWJiIiICrFHswSio6MBAG+88YbG/c+PzQQAGxsb1KhRAwBw9uzZEl2jQYMGpWrv5eUFALhx44bG/QUFBbh586Za25Kws7NT9ZIWHv+iwtv8RERERMVhoVkCFhYWAP6vh/B5N2/eVJttXqhPnz4AgCVLlpToGoXt169frxrrWZyuXbtCkiScOnUKISEhRfbv3r0b9+7dQ6VKlfDmm2+WKEOhLl26AABWrVpVZF92djbWr19fqvMRERHR64mFZgn4+/sDAH7++WdcuXJFtT0iIgIDBgyAqalpkWNmzJgBW1tbHDt2DGPHjkVKSopqX0FBAQ4ePIgDBw6otvXp0wfNmzdHYmIi3n77bdy6dUvtfKGhoVi5cqXq49q1a6Nfv34AgJEjR+LOnTuqfZcvX8aHH34IAJg8eXKpbp0DwEcffQSFQoEdO3Zg1apVEEIAePYUojFjxpSoECYiIiJioVkCffr0QatWrZCSkoLmzZvD29sbPj4+qFevHpKTkzU+KadmzZrYuXMnrK2tsX79ejg6OqJx48bw9fWFjY0NunfvrvY4RyMjI+zatQt169bFuXPnUK9ePdStWxfNmzdH1apV0bhxY9UySoVWrlwJHx8fXLt2DV5eXmjcuDEaNGiAZs2aISEhAZ07d1Z78lBJNWvWDPPnz4cQAhMnTkSNGjXg5+cHZ2dn7Nq1C59//nmpz0lERESvHxaaJWBsbIwjR47gP//5DxwdHREVFYXU1FSMHTsWly5dgouLi8bjOnfujGvXrmHy5MmoVasWbt68ibi4OHh6emLGjBkYMWKEWvuaNWvi0qVLWLhwIZo2bYr79+8jPDwclStXxqhRo/DVV1+ptXdwcMA///yDL7/8EvXr10dERARiYmLg5+eH5cuX4+DBgxofP1kSs2bNws6dO9GyZUukpKTg9u3baNOmDU6dOqXq4SUiIiIqjiQK74sS6ZhSqYStrS2aNGmicQ1POR05ckTuCFqNGzdO7ghavbjigqEo6dOr5HD//n25I2h07949uSNoVdK1e/Vt6NChckfQ6pdffpE7gkbu7u5yR9Dq7t27ckeoUPLz8xEeHo60tLRily9kjyYRERER6QQLTSIiIiLSCRaaRERERKQTLDSJiIiISCdYaBIRERGRTrDQJCIiIiKdYKFJRERERDrBQpOIiIiIdIKFJhERERHphLHcAej14+PjA1NTU7ljqGncuLHcEbQ6ceKE3BG0WrRokdwRNLpx44bcEbTKy8uTO4JGhvxvICEhQe4IGq1du1buCFq1bt1a7gganT9/Xu4IWhUUFMgdQaM6derIHUGj3NxchIeHv7QdezSJiIiISCdYaBIRERGRTrDQJCIiIiKdYKFJRERERDrBQpOIiIiIdIKFJhERERHpBAtNIiIiItIJFppEREREpBMsNImIiIhIJ1hoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmq+ImJgYTJgwAR4eHjAzM4O1tTU8PDzQt29fbNu2TdVu7ty5kCQJc+fO1XieDRs2QJIkjB49Wuv2J0+e4LPPPoOXlxfMzc3Rvn173b0xIiIiqrCM5Q5AZRcdHQ0/Pz8kJSXB0tISdevWhZGREWJjY7F3717cvXsXgwcPLpdrZWZmom3btggJCUG9evXg7e0NMzOzcjk3ERERvVpYaL4ClixZgqSkJIwaNQorVqyAlZWVat/NmzcRHBxcbtfatWsXPD09cf36ddSvXx8AkJWVpbFtdnY2srOzVR8rlcpyy0FERESGj4XmKyAyMhIAMG3aNLUiEwDq1auHevXqldu18vPz8fvvv6uKTAAwNzfX2HbhwoWYN29euV2biIiIKhaO0XwFuLq6AgB27twJIYROr9WgQQM0bdq0RG1nzZqFtLQ01SsuLk6n2YiIiMiwsNB8BUyaNAkmJib46quv4O7ujvfffx9btmzB/fv3y/1az/dkvoyZmRlsbGzUXkRERPT6YKH5CmjcuDGCg4PRtWtXxMfHY/Xq1Rg+fDhq1KiBgIAAhIeHl9u1KlWqVG7nIiIiolcbC81XRKtWrXDkyBGkpKTg8OHDmDlzJmrUqIGjR4+iS5cuSE1NBQBIkgQAWm+xP3nyRF+RiYiI6BXHQvMVY2VlhYCAACxatAg3b96Ep6cn4uPjcejQIQD/1yP56NEjjcdHRUXpLSsRERG92lhovsIsLS3h4+MDAKrxmh4eHgCACxcuFGn/5MkTtcXdiYiIiMqCheYrYOLEidi+fTuePn2qtj04OBjHjx8HANVM8Q4dOsDc3BwXL17EL7/8omqbmpqK0aNHIzk5WX/BiYiI6JXGQvMV8M8//2Dw4MGwtbWFt7c3WrZsCTc3N7Rr1w7p6ekYPnw4OnToAACwt7fH7NmzAQATJkxAjRo10Lx5c1SvXh0nT55U7SMiIiIqKxaar4ClS5diypQp8PX1RVJSEq5cuQIACAgIwP79+7Fx40a19nPmzMFPP/0Eb29vPHr0CHFxcXjnnXdw8eJF1KpVS4Z3QERERK8iPhnoFdChQwdVj2VJffDBB/jggw+KbB89ejRGjx5d4u1ERERE2rBHk4iIiIh0goUmEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wQXbSe8CAgJgaWkpdww1x44dkzuCVkuWLJE7glbdu3eXO4JG165dkzuCVo8fP5Y7gkbW1tZyR9CqWrVqckfQ6MaNG3JH0MpQH7ARFBQkdwStkpOT5Y6gkaurq9wRNMrLyytRO/ZoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSXBzc4MkSYiOjpY7ChEREb1CWGgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSZp9ODBAyxfvhwBAQFwc3ODubk57O3t0a5dO2zatEnueERERFQBsNAkjdauXYsPP/wQJ0+ehLGxMXx8fGBjY4Pg4GCMHDkSEydOlDsiERERGTgWmqRR+/bt8ffffyM9PR1RUVG4cOECYmJiEBoaivr162PVqlUICgqSOyYREREZMBaapJG/vz86dOgAIyMjte2+vr5Yvnw5AGDLli3FniM7OxtKpVLtRURERK8PY7kDkOFKT0/Htm3bcOrUKSQkJCAzMxNCCGRnZwMAQkNDiz1+4cKFmDdvnj6iEhERkQFioUkahYSEoEePHrh//77WNo8fPy72HLNmzcK0adNUHyuVSri6upZbRiIiIjJsvHVOReTn52PgwIG4f/8+3n77bQQFBSEpKQl5eXkQQiAyMhIAkJubW+x5zMzMYGNjo/YiIiKi1wd7NKmI8+fPIyoqCrVq1cLu3bthZmamtj8uLk6mZERERFSRsEeTioiOjgYANGvWrEiRCbx8bCYRERERwEKTNLCwsAAAPHz4sMi+3NxcLFu2TM+JiIiIqCJioUlFtGrVCsbGxjh9+jQ2btyo2p6WloZhw4ZpLECJiIiIXsRCk4pwcnLC1KlTAQCjRo1CrVq10Lx5czg7O2Pv3r1YunSpvAGJiIioQuBkINJo8eLFqFGjBlatWoU7d+7g6dOn6Ny5M2bPng1HR0e54xEREVEFwEKTVJN/nidJEqZMmYIpU6ZoPEYIoeNUREREVNHx1jkRERER6QQLTSIiIiLSCRaaRERERKQTLDSJiIiISCdYaBIRERGRTrDQJCIiIiKdYKFJRERERDrBQpOIiIiIdIKFJhERERHphCT4iBfSE6VSCVtbWwwcOBAmJiZyx1Hz8OFDuSNopVAY7t+DDg4OckfQKD4+Xu4IWtnb28sdQaNJkybJHUGrmTNnyh1Bo8zMTLkjaFW7dm25I2gUExMjdwSt8vPz5Y5QoeTn5yM8PBxpaWmwsbHR2s5wf4MRERERUYXGQpOIiIiIdIKFJhERERHpBAtNIiIiItIJFppEREREpBMsNImIiIhIJ1hoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0JRR+/btIUlSsa/27dsXOe769esYMWIEatSoAVNTUzg6OqJ///44e/asxuuMHj0akiRhw4YNuH//PsaMGQNnZ2eYm5ujQYMG+Omnn4rNef78eQwePBguLi6q6w0YMAAhISHl8WkgIiKiV5Sx3AFeZz4+PsjLy9O47/bt23jw4EGR7fv378fAgQORnZ0NOzs7NGrUCDExMdi9ezf27t2LVatWYfz48RrPGRMTg2bNmiE1NRXe3t5QKBS4ceMGJk+ejNTUVMyePbvIMUuXLsX06dMhhEDlypXRsGFDxMbGYufOndi3bx+2bduGfv36le0TQURERK8k9mjKaPny5Th16lSR17p165CZmQkAeP/991Xt79+/jxEjRiA7OxtTpkzBw4cPceHCBTx48AALFixAQUEBJk2ahKtXr2q83oIFC+Dv74+EhARcunQJ8fHx+PnnnwEA8+fPR2pqqlr7w4cPY/r06ahSpQp27dqF5ORkXL58GUlJSVi7di2EEBg9ejQSEhI0Xi87OxtKpVLtRURERK8PFpoGJi0tDb169UJaWho+/fRTDB48WLXv559/hlKpROPGjbFs2TKYmpoCABQKBT777DO8/fbbyM3NxXfffafx3FWqVMGGDRtgZ2en2jZx4kQ0bdoUWVlZOHHihFr72bNnQwiBdevWFem1HDt2LKZMmYL09HSsXbtW4/UWLlwIW1tb1cvV1fXffEqIiIiogmKhaUAKCgowdOhQREREoHv37liwYIHa/qNHjwIAJk+erPH4KVOmqLV70ZAhQ1CpUqUi2/38/AAAd+7cUW2LiYnB5cuXUa1aNfTq1Uvj+Qq3BwUFadw/a9YspKWlqV5xcXEa2xEREdGriWM0DcisWbNw8OBB1KtXD1u3boVCof53QEREBADA29tb4/ENGjQAADx8+BBKpRI2NjZq+z09PTUeV61aNQBARkaGaltYWBgAICsrC/7+/hqPy8rKAgDEx8dr3G9mZgYzMzON+4iIiOjVx0LTQGzbtg2LFy+GnZ0d9u3bV6RIBP6vECwsDF/k6Oio+v/09PQi59DUmwlAVdAKIVTb0tLSAABKpRKnT58uNnvheFIiIiKi5/HWuQG4fPkyxowZA4VCga1bt8LLy0tjOysrKwBAYmKixv0PHz5U/b+1tXWZMhVe680334QQothXdHR0ma5FREREryYWmjJLTExEnz59kJmZiUWLFqFbt25a2xYWoDdu3NC4//r16wCe9Wxq6hEtjcLb8+Hh4SgoKCjTuYiIiOj1xEJTRrm5uXjnnXcQFxeHYcOGYcaMGcW2DwgIAACsWLFC4/4ff/xRrV1Z1KlTBw0bNsTjx4+xcePGMp+PiIiIXj8sNGX0n//8BydPnkTz5s2xZs2al7afOHEibGxscOXKFXz00UfIyckB8Gy2+uLFi/Hnn3/CxMQE06dPL5d833zzDSRJwqRJk7B27doii8vfuXMHCxYswO7du8vlekRERPRq4WQgGa1evRrAs0k+Xbp00dimSZMmWL58OQCgevXq2LRpEwYMGIBly5bht99+Q+3atRETE4PExEQoFAqsWLECvr6+5ZLv7bffxvLlyzFlyhSMHz8e06ZNg5eXFyRJQlxcnGpM6MqVK8vlekRERPRqYaFpAG7evKl1n7Gx+peoV69euHTpEhYtWoS///4bV65cgZ2dHfr27YsZM2agdevW5Zpt0qRJaNeuHX744Qf8/fffuH79OszMzFCjRg107NgR/fr1w9tvv12u1yQiIqJXgySeX9OGSIeUSiVsbW0xcOBAmJiYyB1HzfMz9g3Ni+upGhIHBwe5I2ikbW1XQ2Bvby93BI0mTZokdwStZs6cKXcEjQx5abfatWvLHUGjmJgYuSNolZ+fL3eECiU/Px/h4eFIS0srdgKy4f4GIyIiIqIKjYUmEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wScDkd49fvy4yBOP5Jabmyt3BK1MTU3ljqBVcnKy3BE08vT0lDuCVsuWLZM7gkbdunWTO4JWkiTJHUGj1NRUuSNoZag/09LS0uSOoJWjo6PcETRKT0+XO4JGBQUFJWrHHk0iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSYRERER6QQLTSIiIiLSCRaaRERERKQT5VJoCiHQrl07SJKELVu2lMcpiYiIiKiCK5dC84cffkBwcDA+/PBDDBs2rDxO+VqKjo6GJElwc3OTOwoRERFRmZW50Lx16xY+++wztGnTBkuWLCmPTERERET0CjAuy8H5+fkYPXo07O3tsWPHDhgbl+l0rz0TExPUrVsXLi4uckchIiIiKrMyVYa3b99GQEAAevbsCScnp/LK9NpycXHBzZs35Y5BREREVC7KVGh6eXlh7ty55RSFiIiIiF4l/2qMZl5eHlatWgV/f3/Y2dnB3Nwc9erVw5w5c6BUKjUeEx8fj2nTpsHb2xuVKlWCra0tfHx88PHHHyMyMrJI+9jYWEycOBHu7u4wMzND1apV0a1bNxw6dEjj+efOnQtJkjB37lykpaVh6tSpqFmzJszMzFC7dm189dVXyMvL0/qe/vzzT7z11luoWrUqzMzM4O7ujg8++ABxcXEa27u5uUGSJERHRyMoKAidO3eGnZ0dKleujL59+6q9p/3796NNmzawsbGBvb09hgwZgvv37xc558smA8XExGD48OGoVq0aLC0t4evri59++glCCLU8z5MkCZIkaX3f2o4Dnq0msG3bNnTp0gVVqlSBmZkZPDw88OGHH+LBgwdaz0lEREQE/ItCU6lUolOnTpg4cSL++ecf2NnZoU6dOrh79y4WLFiAVq1aITExUe2Y48ePw9vbG0uXLkVUVBRq166NmjVr4s6dO1iyZEmRJZHOnTuHRo0aYdWqVXj06BF8fHxgYWGBw4cP4+2338bnn3+uNV9aWhpat26Nn376CVWqVEH16tVx+/ZtfP7555g4caLGY2bNmoUePXrgyJEjsLCwgI+PDxITE7Fy5Uo0atQIFy9e1Hq9PXv2oFOnTggLC4OnpydycnKwd+9etGvXDg8ePMDSpUvRu3dvREdHw8PDA5mZmdi2bRs6duyIrKysEn/ew8PD0axZM2zZsgXp6enw9vZGWloaJk+ejMmTJ5f4PCWVm5uLQYMGYciQIfjrr79gbm6O+vXr4+HDh1i+fDmaNm2KiIiIcr8uERERvTpKXWhOmDABwcHB6NSpEyIjIxEdHY2wsDA8ePAA/fr1Q3h4OCZNmqRqHxsbi/79+0OpVGLkyJF48OABQkNDERYWhvT0dBw4cADNmjVTtX/69CkGDhyI1NRUDBw4EAkJCbh48SLi4uKwYcMGGBkZ4auvvtLas/nTTz/BwcEBMTExCAkJwd27d7F//34YGRlh7dq1RcZAHjhwAIsWLYKxsTE2b96MuLg4XLx4EQkJCejbty9SUlIwYMAAZGZmarzezJkzsXjxYiQkJODSpUu4d+8eWrVqhYSEBIwbNw5z5szBli1bEBcXhytXriAyMhIeHh64desWfv311xJ9zoUQGD58OJKTkxEQEID4+HhcvHgRMTEx+P3337FmzRrEx8eX6Fwl9fnnn+OPP/5AkyZNEBISgvj4eFy5cgVJSUn44IMPkJCQ8NKlrLKzs6FUKtVeRERE9PooVaF59epVbNu2DbVq1cKePXvg4eGh2mdvb49NmzbB1dUVu3btQkxMDADgm2++QVpaGjp16oQNGzagcuXK/3dxhQLdu3dHz549Vdu2bt2K2NhYODo64rfffoO1tbVq36hRozBhwgQAwMKFCzVmNDY2xpYtW1C9enXVtp49e6J3794AUKRAXbRoEQBg0qRJaoWTjY0NNm/ejKpVqyI6Ohq///67xuu9/fbbmDZtGhSKZ59KOzs7zJs3D8Cz2/Hjx4/H0KFDVe1dXV3xySefAAAOHz6s8Zwv+vvvv3H58mVYWFhg8+bNap/DwYMHY+LEicUOCyitR48eYenSpbCxscH+/fvRuHFj1T4LCwssX74cfn5+uHjxIk6ePKn1PAsXLoStra3q5erqWm4ZiYiIyPCVqtDcs2cPAGDgwIFqBWAhS0tLdO7cGUIIVQGyb98+AMCMGTOKHStY6OjRowCA8ePHw9zcvMj+KVOmAADOnDmDJ0+eFNn/1ltvoUaNGkW2+/n5AQDu3Lmj2paRkYF//vkHAPCf//xH4/sZP368Wq4XjR07tsi25wszTfubNGlSJEtxjhw5AgAYMGAAqlatWmT/Bx98UKLzlNTBgweRnZ2NgIAAjZ9LhUKBHj16AACCgoK0nmfWrFlIS0tTvbSNdyUiIqJXU6lmnYeFhQF4VnCeOXNGY5vCnsz4+Hikp6erbum2atWqRNcoHPfn7e2tcX+dOnVgamqKnJwc3L59G76+vmr7PT09NR5XrVo1AM+Ky0JRUVEoKChQTXLRpEGDBmq5XqTpeg4ODiXa/3yW4hReu379+hr316lTB8bGxuXWq1n4dT579iz8/f01tnn48CEAFHvL3szMDGZmZuWSiYiIiCqeUhWaaWlpAJ4VaFFRUcW2zczMVBuTZ2trW6JrFBZfhYXhiyRJgoODg6qQfVGlSpU0Hld4a1sIUeRaDg4OWntbHR0dAUDjtYBnvZ6aMpZk//NZivN8Tk0UCgWqVq1abjPBC7/OcXFxL+2F1DZ2lYiIiKhUt86trKwAAGvWrIEQotjX3Llz1W6vFxYvJb3GizPXCwkh8OjRIwDQePu+NAqv9ejRI61FX2HPXVmvVRbP59SkoKAAycnJxZ5D2/vTNPyg8HqzZ89+6dd5w4YNpXgnRERE9DopVaFZeDv72rVrJWpvY2OjGuN39uzZEh3j5eUFALhx44bG/ZGRkcjJyYGRkZHW2+QlVbt2bSgUCmRnZ2sdL3n9+nW1XHIovLa2pwZFRUUhNzdX477CHl5NRWpaWhqSkpKKbC/t15mIiIhIk1IVmn379gUAbN68+aU9aIX69OkDAFiyZEmJ2gcEBAB41muqaZ3JH3/8EQDw5ptvar1NXlJWVlZ44403AADLly8vsj8zMxNr165VyyWHrl27AgD++OMPjZ/3n3/+WeuxhWNPL1y4UGRf4Xt7Uffu3WFqaoqDBw9qXEyfiIiIqCRKVWg2b94cAwcORHJyMrp06YKQkBC1/fn5+QgMDMSwYcOQnZ0N4Nlsc1tbWxw7dgxjx45FSkqKqn1BQQEOHjyIAwcOqLYNGTIENWvWxMOHDzF69Gi1CTObN2/G6tWrAQCffvpp6d+tBjNnzgTwrFjbunWrant6ejpGjhyJR48ewc3NDYMHDy6X6/0bnTp1QpMmTfD06VOMGDFC7XO4Y8cOrFy5EsbGmofbduvWDQAwZ84c1TAA4NnSSl9++aXG46pXr46pU6ciNzcXAQEBCAwMVNsvhMD58+cxceLEEs+cJyIiotdPqRdsX7dunarIbNq0KWrVqoVWrVrB19cX1tbW6NChA7Zu3aoaE1izZk3s3LkT1tbWWL9+PRwdHdG4cWP4+vrCxsYG3bt3V3vyjqWlJXbs2AFbW1ts374dTk5O8PPzQ82aNTFixAjk5eVhzpw5qgKqrHr06IFPP/0Uubm5GDZsGGrWrAk/Pz84Oztj586dsLe3x44dO2BhYVEu1/s3JEnCpk2bULlyZRw6dAguLi7w8/ODm5sbBg0ahHHjxsHFxUXjsR9//DGcnJxw5coV1KpVC02aNIG7uzu6deuGDz74QOtxCxYswPDhw3H37l106NABzs7OaNmyJRo3bgxbW1u0bNkSq1atQk5Oji7fOhEREVVgpS40rayscPjwYWzZsgUBAQF4+vQpLl++jKSkJPj6+mLmzJk4f/682hqYnTt3xrVr1zB58mTUqlULN2/eRFxcHDw9PTFjxgyMGDFC7RotW7ZEaGgoJkyYgKpVq+Lq1avIyMhA165d8eeff+Krr74q+zt/zsKFC/G///0PXbp0QUZGBq5evYqqVavi/fffR2hoqGoNTjk1aNAAFy9exNChQ2FpaYlr167BxsYGy5cvx4oVK7Qe5+DggNOnT2PAgAGwtLTErVu3YG9vj19//VXrovfAs4XvN23ahD///FM1/CEkJAQJCQnw8vLC5MmTERgYKOvYVSIiIjJskijpGjtk0Nzc3BATE4O7d+/Czc1N7jgaKZVK2NraonPnzlpv9cvFkJdpMjU1lTuCVkZGRnJH0EhbT70hWLZsmdwRNCqvu0S6oGm8viG4f/++3BG0atSokdwRNAoPD5c7glaFyxkaGm3LK8otPz8ft27dQlpaGmxsbLS2K3WPJhERERFRSbDQJCIiIiKdYKFJRERERDrBQpOIiIiIdMKwZmTQvxYdHS13BCIiIiI17NEkIiIiIp1goUlEREREOsFCk4iIiIh0gmM0Se8yMjIMbsH2WrVqyR1BK0NeFNpQF7o31EXRAWDatGlyR9AoMTFR7ghaGeqjbg11UXQAyMvLkzuCRvn5+XJH0Co3N1fuCBoZ6kM7Svq1ZI8mEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSYRERER6QQLTR0YMWIEJEnC119/LXcUIiIiItmw0Cxne/fuxebNm9G7d2/MmjVL7jhEREREsmGhWY6SkpIwYcIE1K1bFxs3boQkSXJHIiIiIpKNsdwBXiUffPABnj59ihMnTsDGxkbuOERERESyYo9mOUlMTIS3tzf27t0Lb29vueMQERERyY49muWkWrVqmDt3rtwxiIiIiAxGufRo5uXlYdWqVfD394ednR3Mzc1Rr149zJkzB0qlUq3t3LlzIUmS1qJsw4YNkCQJo0eP1rr9yZMn+Oyzz+Dl5QVzc3O0b99e1U4Igc2bN6Ndu3aws7ODhYUF6tWrh5kzZ+Lx48carylJkmo85datW9GiRQtYWVmhcuXK6NOnD65du6b1vQshsG3bNnTp0gVVqlSBmZkZPDw88OGHH+LBgwdaj3v8+DFmz56Nhg0bolKlSrC2tkarVq2wZs0aFBQUFGk/evRoSJKEDRs24P79+xgzZgycnZ1hbm6OBg0a4KefftJ4nX97XKHz589j8ODBcHFxgampKRwdHTFgwACEhIQUexwRERFRmQtNpVKJTp06YeLEifjnn39gZ2eHOnXq4O7du1iwYAFatWqFxMTE8sgKAMjMzETbtm2xaNEiGBsbw9vbG2ZmZgCeFX3Dhw/HiBEjEBwcjCpVqsDb2xt3797F4sWL0bRpU9y5c0fruRcvXoxhw4YhLi4O9evXR15eHvbt24cWLVrg1KlTRdrn5uZi0KBBGDJkCP766y+Ym5ujfv36ePjwIZYvX46mTZsiIiKiyHHXr1+Hr68vvv76a0RGRsLNzQ2Ojo44f/483nvvPQwaNAhCCI0ZY2Ji0KxZM/z++++oXr06qlSpghs3bmDy5MlYsGCB1vf2b45bunQpWrVqhe3btyMrKwsNGzZEfn4+du7ciZYtW2L37t1ar0dERERU5kJzwoQJCA4ORqdOnRAZGYno6GiEhYXhwYMH6NevH8LDwzFp0qTyyAoA2LVrFzIyMnD9+nXcuHEDly9fxr59+wAAP/30E7Zu3Qpra2scPXoUt2/fxqVLlxATE4M333wTMTExGDp0qNZzz5kzB0uWLEF8fDwuXLiABw8eYNiwYcjMzMTw4cORmZmp1v7zzz/HH3/8gSZNmiAkJATx8fG4cuUKkpKS8MEHHyAhIQHDhg1TO+bJkyfo3bs34uPj8eGHH+LRo0e4fv06oqKicO3aNTRo0AA7d+7Ezz//rDHjggUL4O/vj4SEBFy6dAnx8fGqtvPnz0dqamq5HHf48GFMnz4dVapUwa5du5CcnIzLly8jKSkJa9euhRACo0ePRkJCgtbPZ3Z2NpRKpdqLiIiIXh9lKjSvXr2Kbdu2oVatWtizZw88PDxU++zt7bFp0ya4urpi165diImJKXNYAMjPz8fvv/+O+vXrq7aZm5tDCIHFixcDAL788kt06dJFtd/JyQnbt2+Hqakpzp07h7///lvjubt164Zp06ZBoXj2abG0tMT69evh5OSEmJgYbNu2TdX20aNHWLp0KWxsbLB//340btxYtc/CwgLLly+Hn58fLl68iJMnT6r2rV+/Hrdv30bfvn3xww8/qM1O9/b2xtatWyFJEr7//nuNGatUqYINGzbAzs5OtW3ixIlo2rQpsrKycOLEiXI5bvbs2RBCYN26dejXr5/avrFjx2LKlClIT0/H2rVrNV4PABYuXAhbW1vVy9XVVWtbIiIievWUqdDcs2cPAGDgwIGwtrYust/S0hKdO3eGEEKt2CqLBg0aoGnTpkW2h4eHIy4uDubm5hg/fnyR/S4uLujfvz8A4OjRoxrPrann1dTUFOPGjQMAHDlyRLX94MGDyM7ORkBAAGrUqFHkOIVCgR49egAAgoKCVNsLbzcXnvNFvr6+cHNzw507d3Dv3r0i+4cMGYJKlSoV2e7n5wcAWocGlOa4mJgYXL58GdWqVUOvXr00nq9w+/Pv7UWzZs1CWlqa6hUXF6e1LREREb16yjTrPCwsDMCzgvPMmTMa2xT2ZMbHx5flUirP92Q+r3AsZM2aNTUWVMCzIvX5tiU9d+H2548rfO9nz56Fv7+/xuMePnwIQP29Fx73+eefa31EZVJSkuq4F4tYT09PjcdUq1YNAJCRkaFxf2mOK8yYlZWl9b1lZWWpMmpjZmamGj9LREREr58yFZppaWkAgKioKERFRRXb9sXxjf+WtiKysFAqLJw0cXR0BACkp6dr3K/tWE3HFb73uLi4l/bUPf/eC4+7dOlSsce8eFwhbe+/8Ha/tklEpTmuMKNSqcTp06dLnZGIiIgIKOOtcysrKwDAmjVrIIQo9lW4nFHhMkLaCqInT56UKUtxM9wLexg13eYHno271KTwnM8fV3i9wrGMxb02bNhQ5LjIyMiXHvf8sk36VJjxzTfffGnG6OhoWTISERGR4StToVn4BJzi1pl8UWHPmrai7mU9o9p4eXkBAGJjY7XePr5+/bpa2xeFh4cXu/354/7Ney/LcfpUmDE8PFzjmp5EREREJVGmQrNv374AgM2bNyM5OblExxTOTL9w4UKRfU+ePFGb2V0a9evXR82aNZGVlaVxJvT9+/exa9cuAEBAQIDGc2haUignJwfr1q0DAHTt2lW1vXv37jA1NcXBgwcRGRlZ4pyFM7h//PFHrb26cqtTpw4aNmyIx48fY+PGjXLHISIiogqqTIVm8+bNMXDgQCQnJ6NLly5FnhaTn5+PwMBADBs2DNnZ2QCADh06wNzcHBcvXsQvv/yiapuamorRo0eXuGB9kSRJmDFjBgDgiy++wPHjx1X7Hj58iMGDByMnJwetWrVChw4dNJ7jzz//xA8//KAqADMzMzF+/Hjcv38frq6uGDx4sKpt9erVMXXqVOTm5iIgIACBgYFq5xJC4Pz585g4caLajO4JEybAw8MDJ06cwLBhw4qsQ5mRkYEdO3Zg2rRp/+rzUF6++eYbSJKESZMmYe3atcjLy1Pbf+fOHSxYsICLthMREZFWZV6wfd26daois2nTpqhVqxZatWoFX19fWFtbo0OHDti6dauqeLO3t8fs2bMBPCu6atSogebNm6N69eo4efKkat+/MWnSJAwdOhRKpRKdO3dGnTp10KxZM9SsWRMnT55EzZo1sWXLFq3Hz58/H1OnTkX16tXRokULODk5YePGjTA3N8fmzZthaWmp1n7BggUYPnw47t69iw4dOsDZ2RktW7ZE48aNYWtri5YtW2LVqlXIyclRHWNlZYU///wT7u7u+P3331GjRg14e3ujVatWqFu3Luzs7DBo0CCts/j15e2338by5cuRnZ2N8ePHo3LlymjevDn8/Pzg5OQET09PzJkzp1yf+kRERESvljIXmlZWVjh8+DC2bNmCgIAAPH36VPUEGV9fX8ycORPnz5+Hubm56pg5c+bgp59+gre3Nx49eoS4uDi88847uHjxImrVqvWvs0iShM2bN2Pjxo1o06YNEhMTcf36ddSqVQszZszA5cuX1RaVf9Enn3yCLVu2wNXVFdevX4ckSejVqxfOnTuHtm3bFmlvbGyMTZs24c8//0SfPn0AACEhIUhISICXlxcmT56MwMDAImNC69Wrh9DQUCxatAh+fn6qJwrl5OSgXbt2+O677/71EILyNGnSJFy5cgXjxo2Dg4MDrl+/jsjISFStWhVDhgzBH3/8gZEjR8odk4iIiAyUJAx1oKAevWwmPJUPpVIJW1tbtGrVCsbGZVpZq9yV5Q8cXbt//77cEbTKzc2VO4JGhw4dkjuCVnIPi9GmuIcvyO35u0KGRNvay4bgxeFOhuLmzZtyR9DKwcFB7ggaGeqk3Pz8fISFhSEtLU3tKYcvKnOPJhERERGRJiw0iYiIiEgnWGgSERERkU6w0CQiIiIinTCsGRky4SQgIiIiovLHHk0iIiIi0gkWmkRERESkEyw0iYiIiEgnOEaT9M7R0REmJiZyx1Bz9uxZuSNo1aZNG7kjaJWRkSF3BI3eeustuSNo9eDBA7kjaGRvby93BK1GjRoldwSNli1bJncErbp37y53BI0KH5BiiAz159mLTxc0FDk5OQgLC3tpO/ZoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSYRERER6QQLTSIiIiLSCRaaRERERKQTLDRJq2vXruGLL75A69at4ezsDFNTUzg7O6Nfv344c+aM3PGIiIjIwLHQJK2mTp2KL7/8Ejdv3oS9vT18fHyQl5eHPXv2oG3btti6davcEYmIiMiAsdAkrd5//31cvXoVKSkpuHHjBi5duoTExETs3bsXFhYWmDhxItLT0+WOSURERAaKhSZp9c4778DHx0dtmyRJ6N27N6ZOnQqlUon//e9/Wo/Pzs6GUqlUexEREdHrw1juAGTYYmNjsXXrVly+fBlJSUnIyckBACQmJgIAQkNDMXToUI3HLly4EPPmzdNbViIiIjIsLDRJq99++w3vv/8+srKytLZ5/Pix1n2zZs3CtGnTVB8rlUq4urqWa0YiIiIyXLx1Thrdvn0b48ePR1ZWFqZPn46QkBAolUoUFBRACIE1a9YAAHJzc7Wew8zMDDY2NmovIiIien2wR5M02rFjB3JzczF48GB89913RfbHxcXJkIqIiIgqEvZokkbR0dEAgDfeeEPj/tDQUD2mISIiooqIhSZpZGFhAQB4+PBhkX03b94sdrY5EREREcBCk7Tw9/cHAPz888+4cuWKantERAQGDBgAU1NTmZIRERFRRcFCkzTq06cPWrVqhZSUFDRv3hze3t7w8fFBvXr1kJycjDlz5sgdkYiIiAwcC03SyNjYGEeOHMF//vMfODo6IioqCqmpqRg7diwuXboEFxcXuSMSERGRgeOsc9LKxsYGP/74I3788cci+0aPHo3Ro0frPxQRERFVGOzRJCIiIiKdYKFJRERERDrBQpOIiIiIdIKFJhERERHpBAtNIiIiItIJFppEREREpBMsNImIiIhIJ1hoEhEREZFOsNAkIiIiIp3gk4FI7xQKBRQKw/obx9zcXO4IWmVmZsodQStD+zoWMjExkTuCVnfu3JE7gkZNmzaVO4JWjRs3ljuCRk5OTnJH0Co3N1fuCBrZ2dnJHUGrxMREuSNodPXqVbkjaJSfn1+idob5W4KIiIiIKjwWmkRERESkEyw0iYiIiEgnWGgSERERkU6w0CQiIiIinWChSUREREQ6wUKTiIiIiHSChSYRERER6QQLTSIiIiLSCRaaryghBNq1awdJkrBly5ZyO++6desgSRLeeustFBQUlNt5iYiI6NXDQvMV9cMPPyA4OBgffvghhg0bVi7njI2NxbRp0+Du7o6tW7ca7OMHiYiIyDCwUngF3bp1C5999hnatGmDJUuWFNkfHR2NuXPnYsOGDSU+pxACY8aMQW5uLnbv3o3KlSuXY2IiIiJ6FbHQfMXk5+dj9OjRsLe3x44dO2BsbFykTXR0NObNm1eqQvPnn3/G8ePHsXr1ajRu3Lj8AhMREdErq2gVQhXa7du3ERAQgJ49e8LJyalczimEQGZmJtauXYsRI0aUyzmJiIjo1cdC8xXj5eWFuXPnlus5JUnCxx9/XK7nJCIiolcfb52/IvLy8rBq1Sr4+/vDzs4O5ubmqFevHubMmQOlUqlq1759e3To0AEAEBQUBEmSVC83N7ci5z1//jwGDx4MFxcXmJqawtHREQMGDEBISIi+3hoRERFVUOzRfAUolUr07NkTwcHBUCgUcHV1hbW1NSIiIrBgwQLs3r0bgYGBqFatGnx8fJCcnIxr167BxsYGPj4+qvM4OzurnXfp0qWYPn06hBCoXLkyGjZsiNjYWOzcuRP79u3Dtm3b0K9fP32/XSIiIqog2KP5CpgwYQKCg4PRqVMnREZGIjo6GmFhYXjw4AH69euH8PBwTJo0CQCwfPlyLF++HADQpEkTnDp1SvX6448/VOc8fPgwpk+fjipVqmDXrl1ITk7G5cuXkZSUhLVr10IIgdGjRyMhIUFrruzsbCiVSrUXERERvT5YaFZwV69exbZt21CrVi3s2bMHHh4eqn329vbYtGkTXF1dsWvXLsTExJT4vLNnz4YQAuvWrSvSazl27FhMmTIF6enpWLt2rdZzLFy4ELa2tqqXq6tr6d8gERERVVgsNCu4PXv2AAAGDhwIa2vrIvstLS3RuXNnCCFw8uTJEp0zJiYGly9fRrVq1dCrVy+NbQq3BwUFaT3PrFmzkJaWpnrFxcWV6PpERET0auAYzQouLCwMwLOC88yZMxrbFPZkxsfHl+qcWVlZ8Pf319gmKyvrpec0MzODmZlZia5JRERErx4WmhVcWloaACAqKgpRUVHFts3MzCzVOZVKJU6fPl0u5yQiIqLXDwvNCs7KygoAsGbNGowbN65cz/nmm2/i1KlT5XJOIiIiev1wjGYF5+3tDQC4du1aiY+RJKlE5wwPD0dBQcG/D0dERESvNRaaFVzfvn0BAJs3b0ZycnKJjrGwsACg/bZ3nTp10LBhQzx+/BgbN24sn6BERET02mGhWcE1b94cAwcORHJyMrp06VLkiT35+fkIDAzEsGHDkJ2dDQBwd3cHANy4cQOPHj3SeN5vvvkGkiRh0qRJWLt2LfLy8tT237lzR7UYPBEREZEmHKP5Cli3bh1SUlJw7NgxNG3aFDVr1oSzszOePn2KqKgoVc/lunXrAAAODg7o2LEj/v77b3h6esLb2xvm5uZwcnLCtm3bAABvv/02li9fjilTpmD8+PGYNm0avLy8IEkS4uLi8PDhQwDAypUr5XnTREREZPDYo/kKsLKywuHDh7FlyxYEBATg6dOnqqf4+Pr6YubMmTh//jzMzc1Vx2zduhWjR4+GjY0NLl26hKCgIJw9e1btvJMmTcKVK1cwbtw4ODg44Pr164iMjETVqlUxZMgQ/PHHHxg5cqS+3y4RERFVEOzRfEUoFAoMHToUQ4cOLVF7R0dH/Prrry9t17BhQ6xZs6as8YiIiOg1xB5NIiIiItIJFppEREREpBMsNImIiIhIJ1hoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEF2wnvYuNjYWRkZHcMdR8++23ckfQaunSpXJH0CotLU3uCBrl5eXJHUErMzMzuSNodPXqVbkjaPX999/LHUGj3bt3yx1Bq759+8odQaOqVavKHUGrrKwsuSNoZGlpKXeEMmGPJhERERHpBAtNIiIiItIJFppEREREpBMsNImIiIhIJ1hoEhEREZFOsNAkIiIiIp1goUlEREREOsFCk4iIiIh0goUmEREREekEC00iIiIi0gkWmlRiQgi0a9cOkiRhy5YtcschIiIiA8dC04AEBgZCkiS0b99e7iga/fDDDwgODsaHH36IYcOGyR2HiIiIDBwLTSqRW7du4bPPPkObNm2wZMkSueMQERFRBcBC04BYWlqibt26qFmzptxR1OTn52P06NGwt7fHjh07YGxsLHckIiIiqgBYMRiQFi1a4ObNm3LHKOL27dsICAhAz5494eTkJHccIiIiqiBYaNJLeXl5Ye7cuXLHICIiogqGt871ICYmBhMmTICHhwfMzMxgbW0NDw8P9O3bF9u2bVO1e9lkoJCQEPTs2RP29vawsrJCq1atsHPnTgCAJEmQJKnIMc9vP3ToENq2bQtra2vY2tqiW7duCAkJ0Zo7Ly8Pq1atgr+/P+zs7GBubo569ephzpw5UCqVZfiMEBER0euAPZo6Fh0dDT8/PyQlJanGYBoZGSE2NhZ79+7F3bt3MXjw4Jee56+//kKPHj2QnZ0NGxsb1K9fH7GxsRgwYAC+//77lx6/atUqfPDBB3BycoKXlxdu3bqFw4cP49SpU7hw4QLq1aun1l6pVKJnz54IDg6GQqGAq6srrK2tERERgQULFmD37t0IDAxEtWrV/vXnhoiIiF5t7NHUsSVLliApKQmjRo3Cw4cPcfXqVYSEhCA5ORnh4eH44IMPXnqO9PR0jBgxAtnZ2Xj33Xfx4MEDXLhwAfHx8VixYgVmzZr10nNMnz4d69evx/3793Hp0iUkJCSgU6dOyMjI0HhbfMKECQgODkanTp0QGRmJ6OhohIWF4cGDB+jXrx/Cw8MxadKkYq+ZnZ0NpVKp9iIiIqLXBwtNHYuMjAQATJs2DVZWVmr76tWrh/fee++l59i6dSsePHiAevXq4ZdffoGFhQWAZ7fFJ02aVKIe0bFjx2L06NGqj62trbF06VIAwOHDh9XaXr16Fdu2bUOtWrWwZ88eeHh4qPbZ29tj06ZNcHV1xa5duxATE6P1mgsXLoStra3q5erq+tKcRERE9OpgoaljhcXVzp07IYT4V+c4duwYAGDEiBEalxZ69913X3qOcePGFdnm4+MDc3NzpKWlITk5WbV9z549AICBAwfC2tq6yHGWlpbo3LkzhBA4efKk1mvOmjULaWlpqldcXNxLcxIREdGrg2M0dWzSpEn47bff8NVXX2Hjxo1466230KZNG3To0AHVq1cv0TkKe0V9fX017te2/Xmenp4atzs4OCAuLg4ZGRmoUqUKACAsLAzAs4LzzJkzGo8r7MmMj4/Xek0zMzOYmZm9NBsRERG9mlho6ljjxo0RHByML774An///TdWr16N1atXQ5IkdOnSBcuWLUP9+vWLPceTJ08AQGPvYnHbn1epUiWN2xWKZ53az/e2pqWlAQCioqIQFRVV7HkzMzNfem0iIiJ6PbHQ1INWrVrhyJEjyMjIwOnTp3HixAls3boVR48eRZcuXXDt2jXY2dlpPb6wSMzIyNC4Pz09vVzzFo4lXbNmjcZb7kREREQlwTGaemRlZYWAgAAsWrQIN2/ehKenJ+Lj43Ho0KFij/Py8gLwbJKOJoW3usuLt7c3AODatWvlel4iIiJ6vbDQlImlpSV8fHwAAPfv3y+2bZcuXQAAmzdvRn5+fpH9GzZsKNdsffv2VV3v+UlCRERERKXBQlPHJk6ciO3bt+Pp06dq24ODg3H8+HEAQNOmTYs9x5AhQ+Dk5IQbN27g/fffR1ZWFoBn4ypXrlyJrVu3lmvm5s2bY+DAgUhOTkaXLl2KPD0oPz8fgYGBGDZsGLKzs8v12kRERPTq4BhNHfvnn3+watUqGBsbo06dOrC2tsbDhw9Vs7aHDx+ODh06FHsOa2trbNq0Cd27d8fatWvxxx9/wMvLC/Hx8bh//z6WLFmC6dOnqyb2lId169YhJSUFx44dQ9OmTVGzZk04Ozvj6dOniIqKUk0CWrduXbldk4iIiP5fe/ceFNV5uHH82eW2KrBqEsELEbRqFG8xarEai9XEyURrtK1NtU2xiTW29ZbYqKkTrbYlk5pqahIydTLxEqqJYyHVVpLQ1ipe0Ch4v0GiYtQQEFkUWRbY3x8O+3NlV6nJ4Szw/cycGfac87LPAWWeefdcmhZmNA22fPlyzZo1S3379lVRUZFyc3MlSaNHj9bf//53rV27tl7fZ9SoUdq9e7cef/xxSdKxY8fUsWNHrV+/XtOmTZNUv6vP6ys8PFwZGRlKTU3V6NGjVV5ergMHDqioqEh9+/bVvHnztHfvXtlstq/tPQEAQNPCjKbBRowYcccZy1qJiYm3van7gAEDtGXLljrr9+/fL0mKjY2ts+1ON4k/c+aM321Wq1WTJk3SpEmTbvs9AAAAfGFGswl45513JElDhw41OQkAAMD/o2g2Ev/5z3+0YcMGr4tvXC6X/vSnPyklJUVWq1VTp041MSEAAIA3PjpvJM6ePaspU6YoJCREcXFxioyM1KlTp+RwOCRJycnJ6t+/v7khAQAAbsKMZiPx8MMP61e/+pW6d++uL7/8Urm5ubLZbBo7dqw+/PBDzZ8/3+yIAAAAXpjRbCS6du2qlStXmh0DAACg3pjRBAAAgCEomgAAADAERRMAAACG4BxNNLj4+HiFhoaaHcPLiy++aHYEv4YMGWJ2BL9cLpfZEXzKz883O4Jfgfo0rcjISLMj+BUeHm52BJ+++93vmh3Br2eeecbsCD5t2rTJ7Ah+VVVVmR3Bp86dO5sdwSeXy6WDBw/ecT9mNAEAAGAIiiYAAAAMQdEEAACAISiaAAAAMARFEwAAAIagaAIAAMAQFE0AAAAYgqIJAAAAQ1A0AQAAYAiKJgAAAAxB0QQAAIAhKJoAAAAwBEUTAAAAhqBoAgAAwBAUTQAAABiCotlMVFVV6a233tKwYcPUunVr2Ww2PfDAA1q4cKEcDofXvqtXr5bFYlFSUpKcTqcWL16sb3zjG7LZbIqJidFzzz2na9eumXQkAACgsaBoNgMOh0MjR47U9OnTtXv3brVu3VrdunXTZ599pt///vdKSEhQYWFhnXEul0uPPvqolixZIpvNptjYWF24cEHLly/X+PHjTTgSAADQmFA0m4Fp06Zp+/btGjlypE6fPq0zZ87o8OHDunTpkiZMmKDjx4/rl7/8ZZ1xGzduVFFRkU6cOKEjR47oxIkT2rlzpyIjI/Xxxx8rIyPjtu/rdDrlcDi8FgAA0HxQNJu4Q4cOacOGDercubPS0tLUpUsXz7Y2bdpo3bp1iomJ0aZNm3T27FmvsVVVVVqzZo26d+/uWZeQkKBnnnlGkrR169bbvndycrLsdrtniYmJ+RqPDAAABDqKZhOXlpYmSZo4caIiIiLqbG/ZsqVGjRolt9utHTt2eG3r37+/Bg4cWGfMoEGDJEmffvrpbd97wYIFKi0t9SwFBQV3exgAAKARCjY7AIx1+PBhSTcK565du3zuUzuT+fnnn3ut79q1q8/927VrJ0m6evXqbd87LCxMYWFh/1NeAADQdFA0m7jS0lJJUl5envLy8m677/Xr171et2rVyud+VuuNiXC32/01JAQAAE0VRbOJCw8PlyStWrXKc24lAABAQ+AczSauV69ekqQjR46YnAQAADQ3FM0mrvZ+l++++66Ki4tNTgMAAJoTimYTN3DgQE2cOFHFxcV65JFHlJOT47W9urpa27Zt0+TJk+V0Ok1KCQAAmiLO0WwG3n77bZWUlOjjjz/WgAEDdP/996t9+/YqLy9XXl6e5yKgt99+2+SkAACgKWFGsxkIDw9XRkaGUlNTNXr0aJWXl+vAgQMqKipS3759NW/ePO3du1c2m83sqAAAoAlhRrOZsFqtmjRpkiZNmnTHfZOSkpSUlOR3e2JiIrc2AgAAd8SMJgAAAAxB0QQAAIAhKJoAAAAwBEUTAAAAhqBoAgAAwBAUTQAAABiCogkAAABDUDQBAABgCG7YjgZ3+PBhBQUFmR3Dyz333GN2BL+CgwP3v+m+ffvMjuBTYWGh2RH8unLlitkRfLJaA3feITIy0uwIPnXo0MHsCH5t2LDB7Ag+ZWZmmh3Brz59+pgdwaeqqiqzI/hU31yB+5cFAAAAjRpFEwAAAIagaAIAAMAQFE0AAAAYgqIJAAAAQ1A0AQAAYAiKJgAAAAxB0QQAAIAhKJoAAAAwBEWzmbty5YratWunoKAg5eTkmB0HAAA0IRTNZu6ll15SaWmpoqKiNGPGDLPjAACAJoSi2YwdPXpUKSkpWrhwod555x3t3LlTqampfvdPT0/X4sWLlZub23AhAQBAo0XRbMZmzpyp3r17a/78+Ro9erR++tOf6oUXXtDVq1d97p+enq7f/va3FE0AAFAvFM1m6osvvtDDDz+sd999VyEhIZKk5cuX6+c//7lOnz5tcjoAANAUBJsdAOaIiorS4sWLvda1adNGixYtMicQAABocpjRDCBHjhzRokWLNGTIELVv316hoaFq3769JkyYoF27dvkdt2vXLk2YMEFRUVEKDQ1Vp06d9NRTT+n48eM+94+NjZXFYtGZM2d8bk9MTJTFYtG2bdskSWfOnJHFYtGaNWskSVOmTJHFYvEstxZWAAAAiaIZUGbPnq0lS5boxIkTatOmjfr06aOqqiqlpaVp+PDh+utf/1pnTEpKioYNG6a0tDRJUr9+/XTt2jWtW7dOAwYM0D/+8Y+vnMtms2no0KFq166dJKlbt24aOnSoZ7n//vu/8nsAAICmh6IZQJ599lkdOnRIJSUlOnbsmPbv36/CwkKlp6erRYsWmj59usrKyjz75+bmaubMmXK73XrllVd08eJF7du3T5cuXdIvfvELVVRUaPLkybp48eJXyhUdHa2srCw99thjkqQXX3xRWVlZnuVnP/vZV/r+AACgaaJoBpDvf//76tOnj9c6i8WicePGafbs2XI4HNq8ebNn27Jly1RVVaVx48bp17/+tazWG7/OsLAwvf7664qPj1dpaalSUlIa9DhqOZ1OORwOrwUAADQfFM0Ac+7cOb388suaOHGivvOd72jYsGEaNmyY3nvvPUnSwYMHPft+9NFHkuTzRusWi0UzZ8702q+hJScny263e5aYmBhTcgAAAHNw1XkAWbNmjZ599llVVFT43efy5cuSbjw68ssvv5Qk9erVy+e+8fHxkqRTp059zUnrZ8GCBXruuec8rx0OB2UTAIBmhBnNAJGfn6+pU6eqoqJCzz//vHJycuRwOFRTUyO3261Vq1ZJklwulyR53VS99iKdW0VFRUmS13mdDSksLEyRkZFeCwAAaD6Y0QwQ77//vlwul5588kktW7aszvaCggKv1+Hh4Z6vCwsL1b59+zpjvvjiC0lSRESE13qLxSJJcrvdPrNcu3btfwsPAADgAzOaAaL2npbf+ta3fG6/+dxMSWrdurXuu+8+SdKxY8d8jjl69KgkqXv37l7rW7VqJUmej95vlZ+f73N9bUEFAACoD4pmgGjRooWk/5+FvNmJEye8rjavNXr0aEnSypUr62xzu92e9bX71erSpYskad++fXXGbdq0SSUlJbfNeP36db/HAQAAUIuiGSCGDRsmSXrzzTeVm5vrWX/q1Cn94Ac/UGhoaJ0xzz//vIKDg/XBBx/o1VdfVU1NjSSpsrJSs2bN0pEjR2S32zV9+nSvcbX3w3zllVe8nmu+b98+zZw50/Ps81vVFtTt27f7/dgdAACgFkUzQDzxxBNKSEhQSUmJBg4cqF69eqlPnz564IEHVFxcrIULF9YZ079/f/35z3+WxWLR3Llz1aFDBw0ePFhRUVFauXKlwsLClJqaqujoaK9xU6ZMUXx8vM6dO+d5nx49emjw4MEaPny434/vx48fr9DQUG3YsEFxcXEaPny4EhMTtXr1aiN+JAAAoJGjaAaI4OBgffjhh5oxY4aioqKUl5enK1eu6Omnn9b+/fvVsWNHn+OmT5+uHTt26IknnlBNTY1yc3PVsmVL/fjHP9aBAwf0+OOP1xljs9n073//W08//bTatm2r06dPy2q1atmyZUpNTfWbsWvXrtq8ebO+/e1vq6SkRFlZWfrvf//r95npAACgebO4+QwUDcThcMhut+vBBx9UUFCQ2XG82O12syP45e8+qYFg9+7dZkfwqbCw0OwIfp07d87sCD61bdvW7Ah+/ehHPzI7gk8XLlwwO4JfgfoktszMTLMj+HXrk/kCRVxcnNkRfHK5XMrIyFBpaeltb1/IjCYAAAAMQdEEAACAISiaAAAAMARFEwAAAIagaAIAAMAQFE0AAAAYgqIJAAAAQ1A0AQAAYAiKJgAAAAzBk4HQYGqfDDRq1CiFhISYHQcAANwll8ulzMxMngwEAAAAc1A0AQAAYAiKJgAAAAxB0QQAAIAhKJoAAAAwBEUTAAAAhqBoAgAAwBAUTQAAABiCogkAAABDUDQBAABgCIomAAAADEHRNMFnn32mVatWaerUqerXr5+Cg4NlsVj0u9/97rbjSktL9dJLL6l3795q2bKlWrdureHDh2v9+vW3Hed0OvXqq6/qoYceUnh4uCIiIjRo0CC9+eabqqmp8Tnm/PnzWrFihcaOHatOnTopNDRUdrtdQ4YM0fLly+V0Ou/6+AEAQPMQbHaA5ui1117Ta6+99j+N+fzzzzVixAidPn1aQUFB6t27t1wul7KysrRjxw5t375dKSkpdcaVlZXpkUceUXZ2tiwWi3r27KmQkBDl5OTok08+0datW5WWlqbgYO9/CkOGDNH58+clSVFRUerXr58uXryoPXv2aM+ePVq7dq0yMzN1zz333P0PAgAANGnMaJrg3nvv1ZgxY7RkyRJt3bpV3/ve9+445ic/+YlOnz6t+Ph45eXlKTc3V0ePHlVOTo46dOigt956S+vWraszbtasWcrOzlaHDh2Uk5Ojo0ePKjc3V3l5eYqPj9eWLVuUnJxcZ5zNZtPMmTN16NAhXbp0Sfv27dP58+eVmZmpdu3aKTc3V9OmTftafh4AAKBpsrjdbrfZIZq7pKQkrVmzRkuXLtXChQvrbD948KD69+8vSdq9e7cSEhK8tr/33nt68skn1aVLF+Xn53vWFxcXKyoqStXV1dqwYYN++MMfeo3bs2ePhgwZooiICF28eFGtWrXybLt8+bLatm3rM2/t+1mtVhUWFtZ7VtPhcMhut2vUqFEKCQmp1xgAABB4XC6XMjMzVVpaqsjISL/7MaPZCOzcuVOS1KlTpzolU5LGjx8vq9WqTz/9VPv37/esz87OVnV1taxWq8aPH19nXEJCgjp27KiysjJlZGR4bfNXMiXp0UcflSTV1NQoLy/vro4JAAA0fRTNRqCkpESS1LFjR5/bQ0NDde+990q6MUt567j77rtPoaGhPsfWfs+bx91JRUWF5+sWLVrUexwAAGheuBioEbDb7ZJuXBDkS2VlpYqKiiRJJ0+erDOuqKhIlZWVPstm7fe8edydvP/++5KkNm3aqFevXn73czqdXlenOxyOer8HAABo/JjRbAQGDRok6cYth/bu3Vtne3p6uuc2RbWzmJI0cOBAWSwWVVdX64MPPqgzbu/evZ6iefO427l48aKWLl0qSZozZ06dq9VvlpycLLvd7lliYmLq9R4AAKBpoGg2At/85jf10EMPSbpx4dCpU6c827KzszVnzhzP6+vXr3u+jo6O9pybOXv2bGVnZ3u2nTp1SklJST7H+VNZWamJEyequLhY/fv317x58267/4IFC1RaWupZCgoK7vgeAACg6aBoNhKpqamKjo7W8ePH1bNnT/Xo0UNxcXFKSEhQeXm5xo4dK0kKDw/3GpeSkqIePXrowoULSkhIUFxcnHr06KGePXsqPz9fEydO9DnuVm63W0lJScrKylL79u2Vlpbm97zPWmFhYYqMjPRaAABA80HRbCR69OihnJwczZo1S7GxsTpz5oyuXbumyZMn68CBA54SFx0d7TWuXbt2ys7O1sKFC9WzZ09dunRJhYWFGjNmjLKzs9WtWzef4241Y8YMrV+/Xm3bttVHH32k2NhYQ44TAAA0HVwM1IhER0drxYoVWrFiRZ1tn3zyiSR5PmK/md1u19KlSz3nVt5s/vz5fsfV+s1vfqM33nhD4eHh2rp1q3r37n2XRwAAAJoTZjSbgKNHj+rkyZOy2WwaNWpUvcddvnxZ27ZtkySNGTPG5z5//OMf9Yc//EE2m02bN2/W4MGDv47IAACgGaBoNnJut1sLFiyQJE2ePFlt2rSp99hFixbJ6XRq5MiR6tmzZ53tf/nLX/TCCy8oJCREGzduVGJi4tcVGwAANAMUzUYiKytL//rXv3TzE0OLi4s1ZcoUbd68WVFRUXr55ZfrjDt8+LDS09NVVVXlWXf16lXNnz9fr7/+ulq2bKk33nijzriNGzdq+vTpslqtWrt2rd8ZTwAAAH941rkJdu7cqXHjxnleX716VU6nUy1btvR60k5OTo7n3pMrVqzQnDlzFBERobi4OLndbh0/flxVVVXq2LGjMjIyfJ47mZ6ervHjx6tFixaKi4tTaGioTpw4oYqKCrVu3Vp/+9vfNGLEiDrjwsLCVFlZqcjISPXp08fvsaxcuVIPPvhgvY6bZ50DANA01PdZ51wMZAKXy6Xi4uI668vLy1VeXu55XV1d7fk6MTFRTz31lHbv3q38/HxZLBb16tVLEyZM0Jw5c/z+kvv166dp06Zpx44dKigoUFVVlTp37qwxY8Zo7ty5fq82r6yslHSjHNY+a92X0tLSeh0zAABofpjRRINhRhMAgKahvjOanKMJAAAAQ1A0AQAAYAiKJgAAAAxB0QQAAIAhKJoAAAAwBEUTAAAAhqBoAgAAwBAUTQAAABiCJwOhwXXu3FmhoaFmx/CyZcsWsyP49dhjj5kdwa+goCCzI/iUnp5udgS/bn76VyCJjY01O4Jfdrvd7Ag+nTx50uwIfs2aNcvsCD7985//NDuCX+fPnzc7gk+ZmZlmR/CprKysXo+gZkYTAAAAhqBoAgAAwBAUTQAAABiCogkAAABDUDQBAABgCIomAAAADEHRBAAAgCEomgAAADAERRMAAACGoGgCAADAEBRNAAAAGIKiCQAAAENQNAEAAGAIiiYAAAAMQdEEAACAISiaAAAAMARFEwAAAIagaAIAAMAQwWYHQNPldDrldDo9rx0Oh4lpAABAQ2NGE4ZJTk6W3W73LDExMWZHAgAADYiiCcMsWLBApaWlnqWgoMDsSAAAoAHx0TkMExYWprCwMLNjAAAAkzCjCQAAAENQNAEAAGAIiiYAAAAMQdHEXZk7d65iY2M1d+5cs6MAAIAARdHEXSkqKtLZs2dVVFRkdhQAABCgKJoAAAAwBLc3wl1ZvXq1Vq9ebXYMAAAQwJjRBAAAgCEomgAAADAERRMAAACGoGgCAADAEBRNAAAAGIKiCQAAAENQNAEAAGAIiiYAAAAMwQ3b0WDcbrckqbKy0uQkddXU1Jgdwa9A/HnVCgoKMjuCT4H8+6z9fxBoqqurzY7gV1VVldkRfArkf2cVFRVmR/ApUH+XUuD+PsvKysyO4NPVq1cl3flvmsUdqH/10OScP39eMTExZscAAABfk4KCAnXq1MnvdoomGkxNTY0uXLigiIgIWSwWs+MAAIC75Ha7VVZWpg4dOshq9X8mJkUTAAAAhuBiIAAAABiCogkAAABDUDQBAABgCIomAAAADEHRBAAAgCEomgAAADAERRMAAACG+D+eSyhaTYOgTgAAAABJRU5ErkJggg==",
+      "text/plain": [
+       "<Figure size 700x700 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Tokenize example sentences in English and French, then get their embeddings\n",
+    "sentence_en = \"The agreement on the European Economic Area was signed in August 1992 .\"\n",
+    "tokenized_en = tokenize(sentence_en, en_words)\n",
+    "embedded_en = embed(tokenized_en, en_embeddings)\n",
+    "\n",
+    "sentence_fr = \"L accord sur la zone économique européenne a été signé en août 1992 .\"\n",
+    "tokenized_fr = tokenize(sentence_fr, fr_words)\n",
+    "embedded_fr = embed(tokenized_fr, fr_embeddings)\n",
+    "\n",
+    "# These weights indicate alignment between words in English and French\n",
+    "alignment = calculate_weights(embedded_fr, embedded_en)\n",
+    "\n",
+    "# Visualize weights to check for alignment\n",
+    "fig, ax = plt.subplots(figsize=(7,7))\n",
+    "ax.imshow(alignment, cmap='gray')\n",
+    "ax.xaxis.tick_top()\n",
+    "ax.set_xticks(np.arange(alignment.shape[1]))\n",
+    "ax.set_xticklabels(sentence_en.split(\" \"), rotation=90, size=16);\n",
+    "ax.set_yticks(np.arange(alignment.shape[0]));\n",
+    "ax.set_yticklabels(sentence_fr.split(\" \"), size=16);"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d634f0ec",
+   "metadata": {},
+   "source": [
+    "If you implemented the weights calculations correctly, the alignment matrix should look like this:\n",
+    "\n",
+    "![alignment visualization](./images/alignment.png)\n",
+    "\n",
+    "This is a demonstration of alignment where the model has learned which words in English correspond to words in French. For example, the words *signed* and *signé* have a large weight because they have the same meaning. Typically, these alignments are learned using linear layers in the model, but you've used pre-trained embeddings here.\n",
+    "\n",
+    "### Exercise 2\n",
+    "Complete the implementation of scaled dot-product attention using your `calculate_weights` function (ignore the mask)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fbfc157e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def attention_qkv(queries, keys, values):\n",
+    "    \"\"\" Calculate scaled dot-product attention from queries, keys, and values matrices \"\"\"\n",
+    "    \n",
+    "    # Replace pass with your code.\n",
+    "    return calculate_weights(queries, keys).dot(values)\n",
+    "\n",
+    "\n",
+    "attention_qkv_result = attention_qkv(embedded_fr, embedded_en, embedded_en)\n",
+    "\n",
+    "print(f\"The shape of the attention_qkv function is {attention_qkv_result.shape}\")\n",
+    "print(f\"Some elements of the attention_qkv function are \\n{attention_qkv_result[0:2,:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f98335f0",
+   "metadata": {},
+   "source": [
+    "**Expected output**\n",
+    "\n",
+    "The shape of the attention_qkv function is `(14, 300)`\n",
+    "\n",
+    "Some elements of the attention_qkv function are \n",
+    "```python\n",
+    "[[-0.04039161 -0.00275749  0.00389873  0.04842744 -0.02472726  0.01435613\n",
+    "  -0.00370253 -0.0619686  -0.00206159  0.01615228]\n",
+    " [-0.04083253 -0.00245985  0.00409068  0.04830341 -0.02479128  0.01447497\n",
+    "  -0.00355203 -0.06196036 -0.00241327  0.01582606]]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f87131fb",
+   "metadata": {},
+   "source": [
+    "## Solutions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8470a024",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "def softmax(x, axis=0):\n",
+    "    \"\"\" Calculate softmax function for an array x\n",
+    "    \n",
+    "        axis=0 calculates softmax across rows which means each column sums to 1 \n",
+    "        axis=1 calculates softmax across columns which means each row sums to 1\n",
+    "    \"\"\"\n",
+    "    y = np.exp(x) \n",
+    "    return y / np.expand_dims(np.sum(y, axis=axis), axis)\n",
+    "\n",
+    "def calculate_weights(queries, keys):\n",
+    "    \"\"\" Calculate the weights for scaled dot-product attention\"\"\"\n",
+    "    dot = np.matmul(queries, keys.T)/np.sqrt(keys.shape[1])\n",
+    "    weights = softmax(dot, axis=1)\n",
+    "    \n",
+    "    assert weights.sum(axis=1)[0] == 1, \"Each row in weights must sum to 1\"\n",
+    "    \n",
+    "    return weights\n",
+    "\n",
+    "def attention_qkv(queries, keys, values):\n",
+    "    \"\"\" Calculate scaled dot-product attention from queries, keys, and values matrices \"\"\"\n",
+    "    weights = calculate_weights(queries, keys)\n",
+    "    return np.matmul(weights, values)\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/embeddings_en.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dde813af6f3db6e520041567fe87619549e9a966515c534f050fe08413e0bc5
+size 6681346

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/embeddings_fr.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02aeb7d162339f0440134f501e867b15203f0dfac2f4cde6c7333840642b03ad
+size 6685746

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/wmt19_can.txt

@@ -0,0 +1 @@
+Walisische AMs machten sich Sorgen, dass sie wie Muppets aussehen könnten

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/wmt19_ref.txt

@@ -0,0 +1 @@
+Walisische Ageordnete sorgen sich "wie Dödel auszusehen"

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/wmt19_src.txt

@@ -0,0 +1 @@
+Welsh AMs worried about 'looking like muppets'

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/word2int_en.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42b2af079f58c498b5f437e1b6922d486a8baa721f254c15ccaba44e5a7c8165
+size 127796

NLP with Attention Models/NMT_with_Attention/Basic Attention-BLEU-QKV Attention/Files/home/jovyan/work/data/word2int_fr.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16e1f4953589bfcad3daae8002476a5727f9902dc8f48f513a20bcb1c94f81dd
+size 133064

NLP with Attention Models/NMT_with_Attention/NMT with MBR/Files/tf/.ipynb_checkpoints/C4W1_Assignment-checkpoint.ipynb

@@ -0,0 +1,1994 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9cb49525",
+   "metadata": {},
+   "source": [
+    "# Assignment 1: Neural Machine Translation\n",
+    "\n",
+    "Welcome to the first assignment of Course 4. Here, you will build an English-to-Portuguese neural machine translation (NMT) model using Long Short-Term Memory (LSTM) networks with attention. Machine translation is an important task in natural language processing and could be useful not only for translating one language to another but also for word sense disambiguation (e.g. determining whether the word \"bank\" refers to the financial bank, or the land alongside a river). Implementing this using just a Recurrent Neural Network (RNN) with LSTMs can work for short to medium length sentences but can result in vanishing gradients for very long sequences. To help with this, you will be adding an attention mechanism to allow the decoder to access all relevant parts of the input sentence regardless of its length. By completing this assignment, you will:\n",
+    "\n",
+    "- Implement an encoder-decoder system with attention\n",
+    "- Build the NMT model from scratch using Tensorflow\n",
+    "- Generate translations using greedy and Minimum Bayes Risk (MBR) decoding\n",
+    "\n",
+    "## Table of Contents\n",
+    "- [1 - Data Preparation](#1)\n",
+    "- [2 - NMT model with attention](#2)\n",
+    "    - [Exercise 1 - Encoder](#ex1)\n",
+    "    - [Exercise 2 - CrossAttention](#ex2)\n",
+    "    - [Exercise 3 - Decoder](#ex3)    \n",
+    "    - [Exercise 4 - Translator](#ex4)\n",
+    "- [3 - Training](#3)\n",
+    "- [4 - Using the model for inference ](#4)\n",
+    "    - [Exercise 5 - translate](#ex5)\n",
+    "- [5 - Minimum Bayes-Risk Decoding](#5)\n",
+    "    - [Exercise 6 - rouge1_similarity](#ex6)\n",
+    "    - [Exercise 7 - average_overlap](#ex7)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f9ef370d",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # Setting this env variable prevents TF warnings from showing up\n",
+    "\n",
+    "import numpy as np\n",
+    "import tensorflow as tf\n",
+    "from collections import Counter\n",
+    "from utils import (sentences, train_data, val_data, english_vectorizer, portuguese_vectorizer, \n",
+    "                   masked_loss, masked_acc, tokens_to_text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8adb8fd6",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import w1_unittest"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e76be1dc",
+   "metadata": {},
+   "source": [
+    "<a name=\"1\"></a>\n",
+    "## 1. Data Preparation\n",
+    "\n",
+    "The text pre-processing bits have already been taken care of (if you are interested in this be sure to check the `utils.py` file). The steps performed can be summarized as:\n",
+    "\n",
+    "- Reading the raw data from the text files\n",
+    "- Cleaning the data (using lowercase, adding space around punctuation, trimming whitespaces, etc)\n",
+    "- Splitting it into training and validation sets\n",
+    "- Adding the start-of-sentence and end-of-sentence tokens to every sentence\n",
+    "- Tokenizing the sentences\n",
+    "- Creating a Tensorflow dataset out of the tokenized sentences\n",
+    "\n",
+    "Take a moment to inspect the raw sentences:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "226033a1",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "portuguese_sentences, english_sentences = sentences\n",
+    "\n",
+    "print(f\"English (to translate) sentence:\\n\\n{english_sentences[-5]}\\n\")\n",
+    "print(f\"Portuguese (translation) sentence:\\n\\n{portuguese_sentences[-5]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ba90eb9",
+   "metadata": {},
+   "source": [
+    "You don't have much use for the raw sentences so delete them to save memory:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d9f081b0",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "del portuguese_sentences\n",
+    "del english_sentences\n",
+    "del sentences"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a2ff83d2",
+   "metadata": {},
+   "source": [
+    "Notice that you imported an `english_vectorizer` and a `portuguese_vectorizer` from `utils.py`. These were created using [tf.keras.layers.TextVectorization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization) and they provide interesting features such as ways to visualize the vocabulary and convert text into tokenized ids and vice versa. In fact, you can inspect the first ten words of the vocabularies for both languages:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2c1cfc17",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "print(f\"First 10 words of the english vocabulary:\\n\\n{english_vectorizer.get_vocabulary()[:10]}\\n\")\n",
+    "print(f\"First 10 words of the portuguese vocabulary:\\n\\n{portuguese_vectorizer.get_vocabulary()[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3152b075",
+   "metadata": {},
+   "source": [
+    "Notice that the first 4 words are reserved for special words. In order, these are:\n",
+    "\n",
+    "- the empty string\n",
+    "- a special token to represent an unknown word\n",
+    "- a special token to represent the start of a sentence\n",
+    "- a special token to represent the end of a sentence\n",
+    "\n",
+    "You can see how many words are in a vocabulary by using the `vocabulary_size` method:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5facaa0c",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Size of the vocabulary\n",
+    "vocab_size_por = portuguese_vectorizer.vocabulary_size()\n",
+    "vocab_size_eng = english_vectorizer.vocabulary_size()\n",
+    "\n",
+    "print(f\"Portuguese vocabulary is made up of {vocab_size_por} words\")\n",
+    "print(f\"English vocabulary is made up of {vocab_size_eng} words\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53e4b615",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "You can define [tf.keras.layers.StringLookup](https://www.tensorflow.org/api_docs/python/tf/keras/layers/StringLookup) objects that will help you map from words to ids and vice versa. Do this for the portuguese vocabulary since this will be useful later on when you decode the predictions from your model:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "218f7a36",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# This helps you convert from words to ids\n",
+    "word_to_id = tf.keras.layers.StringLookup(\n",
+    "    vocabulary=portuguese_vectorizer.get_vocabulary(), \n",
+    "    mask_token=\"\", \n",
+    "    oov_token=\"[UNK]\"\n",
+    ")\n",
+    "\n",
+    "# This helps you convert from ids to words\n",
+    "id_to_word = tf.keras.layers.StringLookup(\n",
+    "    vocabulary=portuguese_vectorizer.get_vocabulary(),\n",
+    "    mask_token=\"\",\n",
+    "    oov_token=\"[UNK]\",\n",
+    "    invert=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4af8b623",
+   "metadata": {},
+   "source": [
+    "Try it out for the special tokens and a random word:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "20076b9a",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "unk_id = word_to_id(\"[UNK]\")\n",
+    "sos_id = word_to_id(\"[SOS]\")\n",
+    "eos_id = word_to_id(\"[EOS]\")\n",
+    "baunilha_id = word_to_id(\"baunilha\")\n",
+    "\n",
+    "print(f\"The id for the [UNK] token is {unk_id}\")\n",
+    "print(f\"The id for the [SOS] token is {sos_id}\")\n",
+    "print(f\"The id for the [EOS] token is {eos_id}\")\n",
+    "print(f\"The id for baunilha (vanilla) is {baunilha_id}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2f1d744c",
+   "metadata": {},
+   "source": [
+    "Finally take a look at how the data that is going to be fed to the neural network looks like. Both `train_data` and `val_data` are of type `tf.data.Dataset` and are already arranged in batches of 64 examples. To get the first batch out of a tf dataset you can use the `take` method. To get the first example out of the batch you can slice the tensor and use the `numpy` method for nicer printing:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "739777eb",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "for (to_translate, sr_translation), translation in train_data.take(1):\n",
+    "    print(f\"Tokenized english sentence:\\n{to_translate[0, :].numpy()}\\n\\n\")\n",
+    "    print(f\"Tokenized portuguese sentence (shifted to the right):\\n{sr_translation[0, :].numpy()}\\n\\n\")\n",
+    "    print(f\"Tokenized portuguese sentence:\\n{translation[0, :].numpy()}\\n\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bdd9ee3c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "There are a couple of important details to notice.\n",
+    "\n",
+    "- Padding has already been applied to the tensors and the value used for this is 0\n",
+    "- Each example consists of 3 different tensors:\n",
+    "    - The sentence to translate\n",
+    "    - The shifted-to-the-right translation\n",
+    "    - The translation\n",
+    "    \n",
+    "The first two can be considered as the features, while the third one as the target. By doing this your model can perform Teacher Forcing as you saw in the lectures.\n",
+    "\n",
+    "Now it is time to begin coding!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dd41cb52",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "<a name=\"2\"></a>\n",
+    "## 2. NMT model with attention\n",
+    "\n",
+    "The model you will build uses an encoder-decoder architecture. This Recurrent Neural Network (RNN) takes in a tokenized version of a sentence in its encoder, then passes it on to the decoder for translation. As mentioned in the lectures, just using a a regular sequence-to-sequence model with LSTMs will work effectively for short to medium sentences but will start to degrade for longer ones. You can picture it like the figure below where all of the context of the input sentence is compressed into one vector that is passed into the decoder block. You can see how this will be an issue for very long sentences (e.g. 100 tokens or more) because the context of the first parts of the input will have very little effect on the final vector passed to the decoder.\n",
+    "\n",
+    "<img src='images/plain_rnn.png'>\n",
+    "\n",
+    "Adding an attention layer to this model avoids this problem by giving the decoder access to all parts of the input sentence. To illustrate, let's just use a 4-word input sentence as shown below. Remember that a hidden state is produced at each timestep of the encoder (represented by the orange rectangles). These are all passed to the attention layer and each are given a score given the current activation (i.e. hidden state) of the decoder. For instance, let's consider the figure below where the first prediction \"como\" is already made. To produce the next prediction, the attention layer will first receive all the encoder hidden states (i.e. orange rectangles) as well as the decoder hidden state when producing the word \"como\" (i.e. first green rectangle). Given this information, it will score each of the encoder hidden states to know which one the decoder should focus on to produce the next word. As a result of training, the model might have learned that it should align to the second encoder hidden state and subsequently assigns a high probability to the word \"você\". If we are using greedy decoding, we will output the said word as the next symbol, then restart the process to produce the next word until we reach an end-of-sentence prediction.\n",
+    "\n",
+    "<img src='images/attention_overview.png'>\n",
+    "\n",
+    "\n",
+    "There are different ways to implement attention and the one we'll use for this assignment is the Scaled Dot Product Attention which has the form:\n",
+    "\n",
+    "$$Attention(Q, K, V) = softmax(\\frac{QK^T}{\\sqrt{d_k}})V$$\n",
+    "\n",
+    "You will dive deeper into this equation in the next week but for now, you can think of it as computing scores using queries (Q) and keys (K), followed by a multiplication of values (V) to get a context vector at a particular timestep of the decoder. This context vector is fed to the decoder RNN to get a set of probabilities for the next predicted word. The division by square root of the keys dimensionality ($\\sqrt{d_k}$) is for improving model performance and you'll also learn more about it next week. For our machine translation application, the encoder activations (i.e. encoder hidden states) will be the keys and values, while the decoder activations (i.e. decoder hidden states) will be the queries.\n",
+    "\n",
+    "You will see in the upcoming sections that this complex architecture and mechanism can be implemented with just a few lines of code. \n",
+    "\n",
+    "First you will define two important global variables:\n",
+    "\n",
+    "- The size of the vocabulary\n",
+    "- The number of units in the LSTM layers (the same number will be used for all LSTM layers)\n",
+    "\n",
+    "In this assignment, the vocabulary sizes for English and Portuguese are the same. Therefore, we use a single constant VOCAB_SIZE throughout the notebook. While in other settings, vocabulary sizes could differ, that is not the case in our assignment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2e484abf",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "VOCAB_SIZE = 12000\n",
+    "UNITS = 256"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cc251965",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex1\"></a>\n",
+    "## Exercise 1 - Encoder\n",
+    "\n",
+    "Your first exercise is to code the encoder part of the neural network. For this, complete the `Encoder` class below. Notice that in the constructor (the `__init__` method) you need to define all of the sublayers of the encoder and then use these sublayers during the forward pass (the `call` method).\n",
+    "\n",
+    "The encoder consists of the following layers:\n",
+    "\n",
+    "- [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding). For this layer you need to define the appropriate `input_dim` and `output_dim` and let it know that you are using '0' as padding, which can be done by using the appropriate value for the `mask_zero` parameter.\n",
+    "    \n",
+    "+ [Bidirectional](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Bidirectional) [LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM). In TF you can implement bidirectional behaviour for RNN-like layers. This part is already taken care of but you will need to specify the appropriate type of layer as well as its parameters. In particular you need to set the appropriate number of units and make sure that the LSTM returns the full sequence and not only the last output, which can be done by using the appropriate value for the `return_sequences` parameter.\n",
+    "\n",
+    "\n",
+    "You need to define the forward pass using the syntax of TF's [functional API](https://www.tensorflow.org/guide/keras/functional_api). What this means is that you chain function calls together to define your network like this:\n",
+    "\n",
+    "```python\n",
+    "encoder_input = keras.Input(shape=(28, 28, 1), name=\"original_img\")\n",
+    "x = layers.Conv2D(16, 3, activation=\"relu\")(encoder_input)\n",
+    "x = layers.MaxPooling2D(3)(x)\n",
+    "x = layers.Conv2D(16, 3, activation=\"relu\")(x)\n",
+    "encoder_output = layers.GlobalMaxPooling2D()(x)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1db0a1d",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: Encoder\n",
+    "class Encoder(tf.keras.layers.Layer):\n",
+    "    def __init__(self, vocab_size, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            vocab_size (int): Size of the vocabulary\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super(Encoder, self).__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        self.embedding = tf.keras.layers.Embedding(  \n",
+    "            input_dim=None,\n",
+    "            output_dim=None,\n",
+    "            mask_zero=None\n",
+    "        )  \n",
+    "\n",
+    "        self.rnn = tf.keras.layers.Bidirectional(  \n",
+    "            merge_mode=\"sum\",  \n",
+    "            layer=tf.keras.layers.None(\n",
+    "                units=None,\n",
+    "                return_sequences=None\n",
+    "            ),  \n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "    def call(self, context):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            context (tf.Tensor): The sentence to translate\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: Encoded sentence to translate\n",
+    "        \"\"\"\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Pass the context through the embedding layer\n",
+    "        x = None\n",
+    "\n",
+    "        # Pass the output of the embedding through the RNN\n",
+    "        x = None\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        return x"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "65034ffd",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "encoder = Encoder(VOCAB_SIZE, UNITS)\n",
+    "\n",
+    "# Pass a batch of sentences to translate from english to portuguese\n",
+    "encoder_output = encoder(to_translate)\n",
+    "\n",
+    "print(f'Tensor of sentences in english has shape: {to_translate.shape}\\n')\n",
+    "print(f'Encoder output has shape: {encoder_output.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a909aea1",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of sentences in english has shape: (64, 14)\n",
+    "\n",
+    "Encoder output has shape: (64, 14, 256)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3031bb14",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Test your code!\n",
+    "\n",
+    "w1_unittest.test_encoder(Encoder)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1afe83f4",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex2\"></a>\n",
+    "## Exercise 2 - CrossAttention\n",
+    "\n",
+    "Your next exercise is to code the layer that will perform cross attention between the original sentences and the translations. For this, complete the `CrossAttention` class below. Notice that in the constructor (the `__init__` method) you need to define all of the sublayers and then use these sublayers during the forward pass (the `call` method). For this particular case some of these bits are already taken care of.\n",
+    "\n",
+    "The cross attention consists of the following layers:\n",
+    "\n",
+    "- [MultiHeadAttention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention). For this layer you need to define the appropriate `key_dim`, which is the size of the key and query tensors. You will also need to set the number of heads to 1 since you aren't implementing multi head attention but attention between two tensors. The reason why this layer is preferred over [Attention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention) is that it allows simpler code during the forward pass.\n",
+    "    \n",
+    "A couple of things to notice:\n",
+    "- You need a way to pass both the output of the attention alongside the shifted-to-the-right translation (since this cross attention happens in the decoder side). For this you will use an [Add](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Add) layer so that the original dimension is preserved, which would not happen if you use something like a [Concatenate](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Concatenate) layer.\n",
+    "\n",
+    "+ Layer normalization is also performed for better stability of the network by using a [LayerNormalization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LayerNormalization) layer.\n",
+    "\n",
+    "- You don't need to worry about these last steps as these are already solved.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74e71f3d",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: CrossAttention\n",
+    "class CrossAttention(tf.keras.layers.Layer):\n",
+    "    def __init__(self, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super().__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        self.mha = ( \n",
+    "            tf.keras.layers.None(\n",
+    "                key_dim=None,\n",
+    "                num_heads=None\n",
+    "            ) \n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        self.layernorm = tf.keras.layers.LayerNormalization()\n",
+    "        self.add = tf.keras.layers.Add()\n",
+    "\n",
+    "    def call(self, context, target):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            context (tf.Tensor): Encoded sentence to translate\n",
+    "            target (tf.Tensor): The embedded shifted-to-the-right translation\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: Cross attention between context and target\n",
+    "        \"\"\"\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Call the MH attention by passing in the query and value\n",
+    "        # For this case the query should be the translation and the value the encoded sentence to translate\n",
+    "        # Hint: Check the call arguments of MultiHeadAttention in the docs\n",
+    "        attn_output = None(\n",
+    "            query=None,\n",
+    "            value=None\n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        x = self.add([target, attn_output])\n",
+    "\n",
+    "        x = self.layernorm(x)\n",
+    "\n",
+    "        return x"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4c62796f",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "attention_layer = CrossAttention(UNITS)\n",
+    "\n",
+    "# The attention layer expects the embedded sr-translation and the context\n",
+    "# The context (encoder_output) is already embedded so you need to do this for sr_translation:\n",
+    "sr_translation_embed = tf.keras.layers.Embedding(VOCAB_SIZE, output_dim=UNITS, mask_zero=True)(sr_translation)\n",
+    "\n",
+    "# Compute the cross attention\n",
+    "attention_result = attention_layer(encoder_output, sr_translation_embed)\n",
+    "\n",
+    "print(f'Tensor of contexts has shape: {encoder_output.shape}')\n",
+    "print(f'Tensor of translations has shape: {sr_translation_embed.shape}')\n",
+    "print(f'Tensor of attention scores has shape: {attention_result.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "41d4f99a",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of contexts has shape: (64, 14, 256)\n",
+    "Tensor of translations has shape: (64, 15, 256)\n",
+    "Tensor of attention scores has shape: (64, 15, 256)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4f658975",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Test your code!\n",
+    "\n",
+    "w1_unittest.test_cross_attention(CrossAttention)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aa296ee2",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex3\"></a>\n",
+    "## Exercise 3 - Decoder\n",
+    "\n",
+    "\n",
+    "Now you will implement the decoder part of the neural network by completing the `Decoder` class below. Notice that in the constructor (the `__init__` method) you need to define all of the sublayers of the decoder and then use these sublayers during the forward pass (the `call` method).\n",
+    "\n",
+    "The decoder consists of the following layers:\n",
+    "\n",
+    "- [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding). For this layer you need to define the appropriate `input_dim` and `output_dim` and let it know that you are using '0' as padding, which can be done by using the appropriate value for the `mask_zero` parameter.\n",
+    "  \n",
+    "  \n",
+    "+ Pre-attention [LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM). Unlike in the encoder in which you used a Bidirectional LSTM, here you will use a vanilla LSTM. Don't forget to set the appropriate number of units and make sure that the LSTM returns the full sequence and not only the last output, which can be done by using the appropriate value for the `return_sequences` parameter. It is very important that this layer returns the state since this will be needed for inference so make sure to set the `return_state` parameter accordingly. Notice that LSTM layers return state as a tuple of two tensors called `memory_state` and `carry_state`, **however these names have been changed to better reflect what you have seen in the lectures to `hidden_state` and `cell_state` respectively**.\n",
+    "\n",
+    "- The attention layer that performs cross attention between the sentence to translate and the right-shifted translation. Here you need to use the `CrossAttention` layer you defined in the previous exercise.\n",
+    "\n",
+    "+ Post-attention [LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM). Another LSTM layer. For this one you don't need it to return the state.\n",
+    "\n",
+    "- Finally a [Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) layer. This one should have the same number of units as the size of the vocabulary since you expect it to compute the logits for every possible word in the vocabulary. Make sure to use a `logsoftmax` activation function for this one, which you can get as [tf.nn.log_softmax](https://www.tensorflow.org/api_docs/python/tf/nn/log_softmax).\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e9639bdb",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: Decoder\n",
+    "class Decoder(tf.keras.layers.Layer):\n",
+    "    def __init__(self, vocab_size, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            vocab_size (int): Size of the vocabulary\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super(Decoder, self).__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # The embedding layer\n",
+    "        self.embedding = tf.keras.layers.None(\n",
+    "            input_dim=None,\n",
+    "            output_dim=None,\n",
+    "            mask_zero=None\n",
+    "        )  \n",
+    "\n",
+    "        # The RNN before attention\n",
+    "        self.pre_attention_rnn = tf.keras.layers.None(\n",
+    "            units=None,\n",
+    "            return_sequences=None,\n",
+    "            return_state=None\n",
+    "        )  \n",
+    "\n",
+    "        # The attention layer\n",
+    "        self.attention = None(None)\n",
+    "\n",
+    "        # The RNN after attention\n",
+    "        self.post_attention_rnn = tf.keras.layers.None(\n",
+    "            units=None,\n",
+    "            return_sequences=None\n",
+    "        )  \n",
+    "\n",
+    "        # The dense layer with logsoftmax activation\n",
+    "        self.output_layer = tf.keras.layers.None(\n",
+    "            units=None,\n",
+    "            activation=None\n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "    def call(self, context, target, state=None, return_state=False):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            context (tf.Tensor): Encoded sentence to translate\n",
+    "            target (tf.Tensor): The shifted-to-the-right translation\n",
+    "            state (list[tf.Tensor, tf.Tensor], optional): Hidden state of the pre-attention LSTM. Defaults to None.\n",
+    "            return_state (bool, optional): If set to true return the hidden states of the LSTM. Defaults to False.\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: The log_softmax probabilities of predicting a particular token\n",
+    "        \"\"\"\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Get the embedding of the input\n",
+    "        x = self.None(None)\n",
+    "\n",
+    "        # Pass the embedded input into the pre attention LSTM\n",
+    "        # Hints:\n",
+    "        # - The LSTM you defined earlier should return the output alongside the state (made up of two tensors)\n",
+    "        # - Pass in the state to the LSTM (needed for inference)\n",
+    "        x, hidden_state, cell_state = self.None(None, initial_state=None)\n",
+    "\n",
+    "        # Perform cross attention between the context and the output of the LSTM (in that order)\n",
+    "        x = self.None(None, None)\n",
+    "\n",
+    "        # Do a pass through the post attention LSTM\n",
+    "        x = self.None(None)\n",
+    "\n",
+    "        # Compute the logits\n",
+    "        logits = self.None(None)\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        if return_state:\n",
+    "            return logits, [hidden_state, cell_state]\n",
+    "\n",
+    "        return logits"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f6165cf2",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "decoder = Decoder(VOCAB_SIZE, UNITS)\n",
+    "\n",
+    "# Notice that you don't need the embedded version of sr_translation since this is done inside the class\n",
+    "logits = decoder(encoder_output, sr_translation)\n",
+    "\n",
+    "print(f'Tensor of contexts has shape: {encoder_output.shape}')\n",
+    "print(f'Tensor of right-shifted translations has shape: {sr_translation.shape}')\n",
+    "print(f'Tensor of logits has shape: {logits.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f2b5d7d",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of contexts has shape: (64, 14, 256)\n",
+    "Tensor of right-shifted translations has shape: (64, 15)\n",
+    "Tensor of logits has shape: (64, 15, 12000)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1b61093a",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Test your code!\n",
+    "\n",
+    "w1_unittest.test_decoder(Decoder, CrossAttention)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9dcce3a7",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex4\"></a>\n",
+    "## Exercise 4 - Translator\n",
+    "\n",
+    "Now you have to put together all of the layers you previously coded into an actual model. For this, complete the `Translator` class below. Notice how unlike the Encoder and Decoder classes inherited from `tf.keras.layers.Layer`, the Translator class inherits from `tf.keras.Model`.\n",
+    "\n",
+    "Remember that `train_data` will yield a tuple with the sentence to translate and the shifted-to-the-right translation, which are the \"features\" of the model. This means that the inputs of your network will be tuples containing context and targets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "205fcf31",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: Translator\n",
+    "class Translator(tf.keras.Model):\n",
+    "    def __init__(self, vocab_size, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            vocab_size (int): Size of the vocabulary\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super().__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Define the encoder with the appropriate vocab_size and number of units\n",
+    "        self.encoder = None\n",
+    "\n",
+    "        # Define the decoder with the appropriate vocab_size and number of units\n",
+    "        self.decoder = None\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "    def call(self, inputs):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            inputs (tuple(tf.Tensor, tf.Tensor)): Tuple containing the context (sentence to translate) and the target (shifted-to-the-right translation)\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: The log_softmax probabilities of predicting a particular token\n",
+    "        \"\"\"\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # In this case inputs is a tuple consisting of the context and the target, unpack it into single variables\n",
+    "        context, target = None\n",
+    "\n",
+    "        # Pass the context through the encoder\n",
+    "        encoded_context = None\n",
+    "\n",
+    "        # Compute the logits by passing the encoded context and the target to the decoder\n",
+    "        logits = None\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        return logits"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4d4a231c",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "translator = Translator(VOCAB_SIZE, UNITS)\n",
+    "\n",
+    "# Compute the logits for every word in the vocabulary\n",
+    "logits = translator((to_translate, sr_translation))\n",
+    "\n",
+    "print(f'Tensor of sentences to translate has shape: {to_translate.shape}')\n",
+    "print(f'Tensor of right-shifted translations has shape: {sr_translation.shape}')\n",
+    "print(f'Tensor of logits has shape: {logits.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e3a162dd",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of sentences to translate has shape: (64, 14)\n",
+    "Tensor of right-shifted translations has shape: (64, 15)\n",
+    "Tensor of logits has shape: (64, 15, 12000)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "37009022",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "w1_unittest.test_translator(Translator, Encoder, Decoder)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f81bc228",
+   "metadata": {},
+   "source": [
+    "<a name=\"3\"></a>\n",
+    "## 3. Training\n",
+    "\n",
+    "Now that you have an untrained instance of the NMT model, it is time to train it. You can use the `compile_and_train` function below to achieve this:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8a61ef65",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def compile_and_train(model, epochs=20, steps_per_epoch=500):\n",
+    "    model.compile(optimizer=\"adam\", loss=masked_loss, metrics=[masked_acc, masked_loss])\n",
+    "\n",
+    "    history = model.fit(\n",
+    "        train_data.repeat(),\n",
+    "        epochs=epochs,\n",
+    "        steps_per_epoch=steps_per_epoch,\n",
+    "        validation_data=val_data,\n",
+    "        validation_steps=50,\n",
+    "        callbacks=[tf.keras.callbacks.EarlyStopping(patience=3)],\n",
+    "    )\n",
+    "\n",
+    "    return model, history"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "87d9bf9f",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Train the translator (this takes some minutes so feel free to take a break)\n",
+    "\n",
+    "trained_translator, history = compile_and_train(translator)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d23b9301",
+   "metadata": {},
+   "source": [
+    "<a name=\"4\"></a>\n",
+    "## 4. Using the model for inference \n",
+    "\n",
+    "\n",
+    "Now that your model is trained you can use it for inference. To help you with this the `generate_next_token` function is provided. Notice that this function is meant to be used inside a for-loop, so you feed to it the information of the previous step to generate the information of the next step. In particular you need to keep track of the state of the pre-attention LSTM in the decoder and if you are done with the translation. Also notice that a `temperature` variable is introduced which determines how to select the next token given the predicted logits:  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "522f6b6f",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def generate_next_token(decoder, context, next_token, done, state, temperature=0.0):\n",
+    "    \"\"\"Generates the next token in the sequence\n",
+    "\n",
+    "    Args:\n",
+    "        decoder (Decoder): The decoder\n",
+    "        context (tf.Tensor): Encoded sentence to translate\n",
+    "        next_token (tf.Tensor): The predicted next token\n",
+    "        done (bool): True if the translation is complete\n",
+    "        state (list[tf.Tensor, tf.Tensor]): Hidden states of the pre-attention LSTM layer\n",
+    "        temperature (float, optional): The temperature that controls the randomness of the predicted tokens. Defaults to 0.0.\n",
+    "\n",
+    "    Returns:\n",
+    "        tuple(tf.Tensor, np.float, list[tf.Tensor, tf.Tensor], bool): The next token, log prob of said token, hidden state of LSTM and if translation is done\n",
+    "    \"\"\"\n",
+    "    # Get the logits and state from the decoder\n",
+    "    logits, state = decoder(context, next_token, state=state, return_state=True)\n",
+    "    \n",
+    "    # Trim the intermediate dimension \n",
+    "    logits = logits[:, -1, :]\n",
+    "        \n",
+    "    # If temp is 0 then next_token is the argmax of logits\n",
+    "    if temperature == 0.0:\n",
+    "        next_token = tf.argmax(logits, axis=-1)\n",
+    "        \n",
+    "    # If temp is not 0 then next_token is sampled out of logits\n",
+    "    else:\n",
+    "        logits = logits / temperature\n",
+    "        next_token = tf.random.categorical(logits, num_samples=1)\n",
+    "    \n",
+    "    # Trim dimensions of size 1\n",
+    "    logits = tf.squeeze(logits)\n",
+    "    next_token = tf.squeeze(next_token)\n",
+    "    \n",
+    "    # Get the logit of the selected next_token\n",
+    "    logit = logits[next_token].numpy()\n",
+    "    \n",
+    "    # Reshape to (1,1) since this is the expected shape for text encoded as TF tensors\n",
+    "    next_token = tf.reshape(next_token, shape=(1,1))\n",
+    "    \n",
+    "    # If next_token is End-of-Sentence token you are done\n",
+    "    if next_token == eos_id:\n",
+    "        done = True\n",
+    "    \n",
+    "    return next_token, logit, state, done"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "190d2d76",
+   "metadata": {},
+   "source": [
+    "See how it works by running the following cell:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9937547a",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# PROCESS SENTENCE TO TRANSLATE AND ENCODE\n",
+    "\n",
+    "# A sentence you wish to translate\n",
+    "eng_sentence = \"I love languages\"\n",
+    "\n",
+    "# Convert it to a tensor\n",
+    "texts = tf.convert_to_tensor(eng_sentence)[tf.newaxis]\n",
+    "\n",
+    "# Vectorize it and pass it through the encoder\n",
+    "context = english_vectorizer(texts).to_tensor()\n",
+    "context = encoder(context)\n",
+    "\n",
+    "# SET STATE OF THE DECODER\n",
+    "\n",
+    "# Next token is Start-of-Sentence since you are starting fresh\n",
+    "next_token = tf.fill((1,1), sos_id)\n",
+    "\n",
+    "# Hidden and Cell states of the LSTM can be mocked using uniform samples\n",
+    "state = [tf.random.uniform((1, UNITS)), tf.random.uniform((1, UNITS))]\n",
+    "\n",
+    "# You are not done until next token is EOS token\n",
+    "done = False\n",
+    "\n",
+    "# Generate next token\n",
+    "next_token, logit, state, done = generate_next_token(decoder, context, next_token, done, state, temperature=0.5)\n",
+    "print(f\"Next token: {next_token}\\nLogit: {logit:.4f}\\nDone? {done}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "170323dd",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex5\"></a>\n",
+    "## Exercise 5 - translate\n",
+    "\n",
+    "Now you can put everything together to translate a given sentence. For this, complete the `translate` function below. This function will take care of the following steps: \n",
+    "- Process the sentence to translate and encode it\n",
+    "\n",
+    "+ Set the initial state of the decoder\n",
+    "\n",
+    "- Get predictions of the next token (starting with the \\<SOS> token) for a maximum of iterations (in case the \\<EOS> token is never returned)\n",
+    "    \n",
+    "+ Return the translated text (as a string), the logit of the last iteration (this helps measure how certain was that the sequence was translated in its totality) and the translation in token format.\n",
+    "\n",
+    "\n",
+    "Hints: \n",
+    "\n",
+    "- The previous cell provides a lot of insights on how this function should work, so if you get stuck refer to it.\n",
+    "\n",
+    "+ Some useful docs:\n",
+    "    + [tf.newaxis](https://www.tensorflow.org/api_docs/python/tf#newaxis)\n",
+    "\n",
+    "    - [tf.fill](https://www.tensorflow.org/api_docs/python/tf/fill)\n",
+    "\n",
+    "    + [tf.zeros](https://www.tensorflow.org/api_docs/python/tf/zeros)\n",
+    "\n",
+    "\n",
+    "**IMPORTANT NOTE**: Due to randomness processes involving tensorflow training and weight initializing, the results below may vary a lot, even if you retrain your model in the same session. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "42c74f1f",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED FUNCTION: translate\n",
+    "def translate(model, text, max_length=50, temperature=0.0):\n",
+    "    \"\"\"Translate a given sentence from English to Portuguese\n",
+    "\n",
+    "    Args:\n",
+    "        model (tf.keras.Model): The trained translator\n",
+    "        text (string): The sentence to translate\n",
+    "        max_length (int, optional): The maximum length of the translation. Defaults to 50.\n",
+    "        temperature (float, optional): The temperature that controls the randomness of the predicted tokens. Defaults to 0.0.\n",
+    "\n",
+    "    Returns:\n",
+    "        tuple(str, np.float, tf.Tensor): The translation, logit that predicted <EOS> token and the tokenized translation\n",
+    "    \"\"\"\n",
+    "    # Lists to save tokens and logits\n",
+    "    tokens, logits = [], []\n",
+    "\n",
+    "    ### START CODE HERE ###\n",
+    "    \n",
+    "    # PROCESS THE SENTENCE TO TRANSLATE\n",
+    "    \n",
+    "    # Convert the original string into a tensor\n",
+    "    text = tf.None(None)[tf.None]\n",
+    "    \n",
+    "    # Vectorize the text using the correct vectorizer\n",
+    "    context = None(None).to_tensor()\n",
+    "    \n",
+    "    # Get the encoded context (pass the context through the encoder)\n",
+    "    # Hint: Remember you can get the encoder by using model.encoder\n",
+    "    context = None.None(None)\n",
+    "    \n",
+    "    # INITIAL STATE OF THE DECODER\n",
+    "    \n",
+    "    # First token should be SOS token with shape (1,1)\n",
+    "    next_token = tf.None((None, None), None)\n",
+    "    \n",
+    "    # Initial hidden and cell states should be tensors of zeros with shape (1, UNITS)\n",
+    "    state = [tf.None((None, None)), tf.None((None, None))]\n",
+    "    \n",
+    "    # You are done when you draw a EOS token as next token (initial state is False)\n",
+    "    done = None\n",
+    "\n",
+    "    # Iterate for max_length iterations\n",
+    "    for None in None(None):\n",
+    "        # Generate the next token\n",
+    "        try:\n",
+    "            next_token, logit, state, done = None(\n",
+    "                decoder=None,\n",
+    "                context=None,\n",
+    "                next_token=None,\n",
+    "                done=None,\n",
+    "                state=None,\n",
+    "                temperature=None\n",
+    "            )\n",
+    "        except:\n",
+    "             raise Exception(\"Problem generating the next token\")\n",
+    "        \n",
+    "        # If done then break out of the loop\n",
+    "        if None:\n",
+    "            None\n",
+    "        \n",
+    "        # Add next_token to the list of tokens\n",
+    "        None.None(None)\n",
+    "        \n",
+    "        # Add logit to the list of logits\n",
+    "        None.None(None)\n",
+    "    \n",
+    "    ### END CODE HERE ###\n",
+    "    \n",
+    "    # Concatenate all tokens into a tensor\n",
+    "    tokens = tf.concat(tokens, axis=-1)\n",
+    "    \n",
+    "    # Convert the translated tokens into text\n",
+    "    translation = tf.squeeze(tokens_to_text(tokens, id_to_word))\n",
+    "    translation = translation.numpy().decode()\n",
+    "    \n",
+    "    return translation, logits[-1], tokens"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3525e8ba",
+   "metadata": {},
+   "source": [
+    "Try your function with temperature of 0, which will yield a deterministic output and is equivalent to a greedy decoding:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "daaea8c5",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Running this cell multiple times should return the same output since temp is 0\n",
+    "\n",
+    "temp = 0.0 \n",
+    "original_sentence = \"I love languages\"\n",
+    "\n",
+    "translation, logit, tokens = translate(trained_translator, original_sentence, temperature=temp)\n",
+    "\n",
+    "print(f\"Temperature: {temp}\\n\\nOriginal sentence: {original_sentence}\\nTranslation: {translation}\\nTranslation tokens:{tokens}\\nLogit: {logit:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7d05129b",
+   "metadata": {},
+   "source": [
+    "Try your function with temperature of 0.7 (stochastic output):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0e0697db",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Running this cell multiple times should return different outputs since temp is not 0\n",
+    "# You can try different temperatures\n",
+    "\n",
+    "temp = 0.7\n",
+    "original_sentence = \"I love languages\"\n",
+    "\n",
+    "translation, logit, tokens = translate(trained_translator, original_sentence, temperature=temp)\n",
+    "\n",
+    "print(f\"Temperature: {temp}\\n\\nOriginal sentence: {original_sentence}\\nTranslation: {translation}\\nTranslation tokens:{tokens}\\nLogit: {logit:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a3a9ea35",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "w1_unittest.test_translate(translate, trained_translator)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ba027524",
+   "metadata": {},
+   "source": [
+    "<a name=\"5\"></a>\n",
+    "## 5. Minimum Bayes-Risk Decoding\n",
+    "\n",
+    "As mentioned in the lectures, getting the most probable token at each step may not necessarily produce the best results. Another approach is to do Minimum Bayes Risk Decoding or MBR. The general steps to implement this are:\n",
+    "\n",
+    "- Take several random samples\n",
+    "+ Score each sample against all other samples\n",
+    "- Select the one with the highest score\n",
+    "\n",
+    "You will be building helper functions for these steps in the following sections.\n",
+    "\n",
+    "With the ability to generate different translations by setting different temperature values you can do what you saw in the lectures and generate a bunch of translations and then determine which one is the best candidate. You will now do this by using the provided `generate_samples` function. This function will return any desired number of candidate translations alongside the log-probability for each one:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62301cd5",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def generate_samples(model, text, n_samples=4, temperature=0.6):\n",
+    "    \n",
+    "    samples, log_probs = [], []\n",
+    "\n",
+    "    # Iterate for n_samples iterations\n",
+    "    for _ in range(n_samples):\n",
+    "        \n",
+    "        # Save the logit and the translated tensor\n",
+    "        _, logp, sample = translate(model, text, temperature=temperature)\n",
+    "        \n",
+    "        # Save the translated tensors\n",
+    "        samples.append(np.squeeze(sample.numpy()).tolist())\n",
+    "        \n",
+    "        # Save the logits\n",
+    "        log_probs.append(logp)\n",
+    "                \n",
+    "    return samples, log_probs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "06bd792c",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "samples, log_probs = generate_samples(trained_translator, 'I love languages')\n",
+    "\n",
+    "for s, l in zip(samples, log_probs):\n",
+    "    print(f\"Translated tensor: {s} has logit: {l:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "29b10677",
+   "metadata": {},
+   "source": [
+    "## Comparing overlaps\n",
+    "\n",
+    "Now that you can generate multiple translations it is time to come up with a method to measure the goodness of each one. As you saw in the lectures, one way to achieve this is by comparing each sample against the others. \n",
+    "\n",
+    "There are several metrics you can use for this purpose, as shown in the lectures and you can try experimenting with any one of these. For this assignment, you will be calculating scores for **unigram overlaps**. \n",
+    "\n",
+    "One of these metrics is the widely used yet simple [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index) which gets the intersection over union of two sets. The `jaccard_similarity` function returns this metric for any pair of candidate and reference translations:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "edb54a71",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def jaccard_similarity(candidate, reference):\n",
+    "        \n",
+    "    # Convert the lists to sets to get the unique tokens\n",
+    "    candidate_set = set(candidate)\n",
+    "    reference_set = set(reference)\n",
+    "    \n",
+    "    # Get the set of tokens common to both candidate and reference\n",
+    "    common_tokens = candidate_set.intersection(reference_set)\n",
+    "    \n",
+    "    # Get the set of all tokens found in either candidate or reference\n",
+    "    all_tokens = candidate_set.union(reference_set)\n",
+    "    \n",
+    "    # Compute the percentage of overlap (divide the number of common tokens by the number of all tokens)\n",
+    "    overlap = len(common_tokens) / len(all_tokens)\n",
+    "        \n",
+    "    return overlap"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fc3384bf",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 3, 4]\n",
+    "\n",
+    "js = jaccard_similarity(l1, l2)\n",
+    "\n",
+    "print(f\"jaccard similarity between lists: {l1} and {l2} is {js:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a6997662",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "jaccard similarity between tensors: [1, 2, 3] and [1, 2, 3, 4] is 0.750\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b2510e3d",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex6\"></a>\n",
+    "## Exercise 6 - rouge1_similarity\n",
+    "\n",
+    "Jaccard similarity is good but a more commonly used metric in machine translation is the ROUGE score. For unigrams, this is called ROUGE-1 and as shown in the lectures, you can output the scores for both precision and recall when comparing two samples. To get the final score, you will want to compute the F1-score as given by:\n",
+    "\n",
+    "$$score = 2* \\frac{(precision * recall)}{(precision + recall)}$$\n",
+    "\n",
+    "For the implementation of the `rouge1_similarity` function you want to use the [Counter](https://docs.python.org/3/library/collections.html#collections.Counter) class from the Python standard library:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fb2e0a00",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED FUNCTION: rouge1_similarity\n",
+    "def rouge1_similarity(candidate, reference):\n",
+    "    \"\"\"Computes the ROUGE 1 score between two token lists\n",
+    "\n",
+    "    Args:\n",
+    "        candidate (list[int]): Tokenized candidate translation\n",
+    "        reference (list[int]): Tokenized reference translation\n",
+    "\n",
+    "    Returns:\n",
+    "        float: Overlap between the two token lists\n",
+    "    \"\"\"\n",
+    "    ### START CODE HERE ###\n",
+    "    \n",
+    "    # Make a frequency table of the candidate and reference tokens\n",
+    "    # Hint: use the Counter class (already imported)\n",
+    "    candidate_word_counts = None\n",
+    "    reference_word_counts = None\n",
+    "    \n",
+    "    # Initialize overlap at 0\n",
+    "    overlap = None\n",
+    "    \n",
+    "    # Iterate over the tokens in the candidate frequency table\n",
+    "    # Hint: Counter is a subclass of dict and you can get the keys \n",
+    "    #       out of a dict using the keys method like this: dict.keys()\n",
+    "    for token in None:\n",
+    "        \n",
+    "        # Get the count of the current token in the candidate frequency table\n",
+    "        # Hint: You can access the counts of a token as you would access values of a dictionary\n",
+    "        token_count_candidate = None\n",
+    "        \n",
+    "        # Get the count of the current token in the reference frequency table\n",
+    "        # Hint: You can access the counts of a token as you would access values of a dictionary\n",
+    "        token_count_reference = None\n",
+    "        \n",
+    "        # Update the overlap by getting the minimum between the two token counts above\n",
+    "        overlap += None\n",
+    "    \n",
+    "    # Compute the precision\n",
+    "    # Hint: precision = overlap / (number of tokens in candidate list) \n",
+    "    precision = None\n",
+    "    \n",
+    "    # Compute the recall\n",
+    "    # Hint: recall = overlap / (number of tokens in reference list) \n",
+    "    recall = None\n",
+    "    \n",
+    "    if precision + recall != 0:\n",
+    "        # Compute the Rouge1 Score\n",
+    "        # Hint: This is equivalent to the F1 score\n",
+    "        f1_score = None\n",
+    "        \n",
+    "        return f1_score\n",
+    "    \n",
+    "    ### END CODE HERE ###\n",
+    "        \n",
+    "    return 0 # If precision + recall = 0 then return 0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14bb5295",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 3, 4]\n",
+    "\n",
+    "r1s = rouge1_similarity(l1, l2)\n",
+    "\n",
+    "print(f\"rouge 1 similarity between lists: {l1} and {l2} is {r1s:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "afb8c61a",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "rouge 1 similarity between lists: [1, 2, 3] and [1, 2, 3, 4] is 0.857\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a680132e",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "w1_unittest.test_rouge1_similarity(rouge1_similarity)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aaf8a058",
+   "metadata": {},
+   "source": [
+    "## Computing the Overall Score\n",
+    "\n",
+    "\n",
+    "You will now build a function to generate the overall score for a particular sample. As mentioned in the lectures, you need to compare each sample with all other samples. For instance, if we generated 30 sentences, we will need to compare sentence 1 to sentences 2 through 30. Then, we compare sentence 2 to sentences 1 and 3 through 30, and so forth. At each step, we get the average score of all comparisons to get the overall score for a particular sample. To illustrate, these will be the steps to generate the scores of a 4-sample list.\n",
+    "\n",
+    "- Get similarity score between sample 1 and sample 2\n",
+    "+ Get similarity score between sample 1 and sample 3\n",
+    "- Get similarity score between sample 1 and sample 4\n",
+    "+ Get average score of the first 3 steps. This will be the overall score of sample 1\n",
+    "- Iterate and repeat until samples 1 to 4 have overall scores.\n",
+    "\n",
+    "\n",
+    "The results will be stored in a dictionary for easy lookups.\n",
+    "\n",
+    "<a name=\"ex7\"></a>\n",
+    "## Exercise 7 - average_overlap\n",
+    "\n",
+    "Complete the `average_overlap` function below which should implement the process described above:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "142264ff",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED FUNCTION: average_overlap\n",
+    "def average_overlap(samples, similarity_fn):\n",
+    "    \"\"\"Computes the arithmetic mean of each candidate sentence in the samples\n",
+    "\n",
+    "    Args:\n",
+    "        samples (list[list[int]]): Tokenized version of translated sentences\n",
+    "        similarity_fn (Function): Similarity function used to compute the overlap\n",
+    "\n",
+    "    Returns:\n",
+    "        dict[int, float]: A dictionary mapping the index of each translation to its score\n",
+    "    \"\"\"\n",
+    "    # Initialize dictionary\n",
+    "    scores = {}\n",
+    "    \n",
+    "    # Iterate through all samples (enumerate helps keep track of indexes)\n",
+    "    for index_candidate, candidate in enumerate(samples):    \n",
+    "        \n",
+    "        ### START CODE HERE ###\n",
+    "                \n",
+    "        # Initially overlap is zero\n",
+    "        overlap = None\n",
+    "        \n",
+    "        # Iterate through all samples (enumerate helps keep track of indexes)\n",
+    "        for index_sample, sample in enumerate(samples):\n",
+    "\n",
+    "            # Skip if the candidate index is the same as the sample index\n",
+    "            if None == None:\n",
+    "                None\n",
+    "                \n",
+    "            # Get the overlap between candidate and sample using the similarity function\n",
+    "            sample_overlap = None(None, None)\n",
+    "            \n",
+    "            # Add the sample overlap to the total overlap\n",
+    "            overlap += None\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "        \n",
+    "        # Get the score for the candidate by computing the average\n",
+    "        score = overlap / (len(samples) - 1)\n",
+    "\n",
+    "        # Only use 3 decimal points\n",
+    "        score = round(score, 3)\n",
+    "        \n",
+    "        # Save the score in the dictionary. use index as the key.\n",
+    "        scores[index_candidate] = score\n",
+    "        \n",
+    "    return scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f36cf403",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Test with Jaccard similarity\n",
+    "\n",
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 4]\n",
+    "l3 = [1, 2, 4, 5]\n",
+    "\n",
+    "avg_ovlp = average_overlap([l1, l2, l3], jaccard_similarity)\n",
+    "\n",
+    "print(f\"average overlap between lists: {l1}, {l2} and {l3} using Jaccard similarity is:\\n\\n{avg_ovlp}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e277aed2-a5c9-4ed0-9ee2-614939f2df7b",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "average overlap between lists: [1, 2, 3], [1, 2, 4] and [1, 2, 4, 5] using Jaccard similarity is:\n",
+    "\n",
+    "{0: 0.45, 1: 0.625, 2: 0.575}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d961a304-7c03-4ecb-ba5f-c8747ed3ec39",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# Test with Rouge1 similarity\n",
+    "\n",
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 4]\n",
+    "l3 = [1, 2, 4, 5]\n",
+    "l4 = [5,6]\n",
+    "\n",
+    "avg_ovlp = average_overlap([l1, l2, l3, l4], rouge1_similarity)\n",
+    "\n",
+    "print(f\"average overlap between lists: {l1}, {l2}, {l3} and {l4} using Rouge1 similarity is:\\n\\n{avg_ovlp}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30adc749-ffcb-4e82-a8f0-c04a7e39da0a",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "average overlap between lists: [1, 2, 3], [1, 4], [1, 2, 4, 5] and [5, 6] using Rouge1 similarity is:\n",
+    "\n",
+    "{0: 0.324, 1: 0.356, 2: 0.524, 3: 0.111}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c41b1fba-fd0f-41e6-9b07-746f64030fe3",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "w1_unittest.test_average_overlap(average_overlap)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4482249",
+   "metadata": {},
+   "source": [
+    "In practice, it is also common to see the weighted mean being used to calculate the overall score instead of just the arithmetic mean. This is implemented in the `weighted_avg_overlap` function below and you can use it in your experiments to see which one will give better results:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "398714be",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def weighted_avg_overlap(samples, log_probs, similarity_fn):\n",
+    "    \n",
+    "    # Scores dictionary\n",
+    "    scores = {}\n",
+    "    \n",
+    "    # Iterate over the samples\n",
+    "    for index_candidate, candidate in enumerate(samples):    \n",
+    "        \n",
+    "        # Initialize overlap and weighted sum\n",
+    "        overlap, weight_sum = 0.0, 0.0\n",
+    "        \n",
+    "        # Iterate over all samples and log probabilities\n",
+    "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):\n",
+    "\n",
+    "            # Skip if the candidate index is the same as the sample index            \n",
+    "            if index_candidate == index_sample:\n",
+    "                continue\n",
+    "                \n",
+    "            # Convert log probability to linear scale\n",
+    "            sample_p = float(np.exp(logp))\n",
+    "\n",
+    "            # Update the weighted sum\n",
+    "            weight_sum += sample_p\n",
+    "\n",
+    "            # Get the unigram overlap between candidate and sample\n",
+    "            sample_overlap = similarity_fn(candidate, sample)\n",
+    "            \n",
+    "            # Update the overlap\n",
+    "            overlap += sample_p * sample_overlap\n",
+    "            \n",
+    "        # Compute the score for the candidate\n",
+    "        score = overlap / weight_sum\n",
+    "\n",
+    "        # Only use 3 decimal points\n",
+    "        score = round(score, 3)\n",
+    "        \n",
+    "        # Save the score in the dictionary. use index as the key.\n",
+    "        scores[index_candidate] = score\n",
+    "    \n",
+    "    return scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e3dfd6d3",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 4]\n",
+    "l3 = [1, 2, 4, 5]\n",
+    "log_probs = [0.4, 0.2, 0.5]\n",
+    "\n",
+    "w_avg_ovlp = weighted_avg_overlap([l1, l2, l3], log_probs, jaccard_similarity)\n",
+    "\n",
+    "print(f\"weighted average overlap using Jaccard similarity is:\\n\\n{w_avg_ovlp}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cdb0b4db",
+   "metadata": {},
+   "source": [
+    "## mbr_decode\n",
+    "\n",
+    "You will now put everything together in the the `mbr_decode` function below. This final step is not graded as this function is just a wrapper around all the cool stuff you have coded so far! \n",
+    "\n",
+    "You can use it to play around, trying different numbers of samples, temperatures and similarity functions!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6fcfa640",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def mbr_decode(model, text, n_samples=5, temperature=0.6, similarity_fn=jaccard_similarity):\n",
+    "    \n",
+    "    # Generate samples\n",
+    "    samples, log_probs = generate_samples(model, text, n_samples=n_samples, temperature=temperature)\n",
+    "    \n",
+    "    # Compute the overlap scores\n",
+    "    scores = weighted_avg_overlap(samples, log_probs, similarity_fn)\n",
+    "\n",
+    "    # Decode samples\n",
+    "    decoded_translations = [tokens_to_text(s, id_to_word).numpy().decode('utf-8') for s in samples]\n",
+    "    \n",
+    "    # Find the key with the highest score\n",
+    "    max_score_key = max(scores, key=lambda k: scores[k])\n",
+    "    \n",
+    "    # Get the translation \n",
+    "    translation = decoded_translations[max_score_key]\n",
+    "    \n",
+    "    return translation, decoded_translations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "99507fcc-7727-45e7-933b-d3a08034f731",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "english_sentence = \"I love languages\"\n",
+    "\n",
+    "translation, candidates = mbr_decode(trained_translator, english_sentence, n_samples=10, temperature=0.6)\n",
+    "\n",
+    "print(\"Translation candidates:\")\n",
+    "for c in candidates:\n",
+    "    print(c)\n",
+    "\n",
+    "print(f\"\\nSelected translation: {translation}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "801b193f-4ea6-4ca1-ae29-a506cce656d9",
+   "metadata": {},
+   "source": [
+    "**Congratulations!** Next week, you'll dive deeper into attention models and study the Transformer architecture. You will build another network but without the recurrent part. It will show that attention is all you need! It should be fun!\n",
+    "\n",
+    "**Keep up the good work!**"
+   ]
+  }
+ ],
+ "metadata": {
+  "grader_version": "1",
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

NLP with Attention Models/NMT_with_Attention/NMT with MBR/Files/tf/.ipynb_checkpoints/w1_unittest-checkpoint.py

@@ -0,0 +1,654 @@
+import math
+from itertools import combinations
+import tensorflow as tf
+import numpy as np
+from dlai_grader.grading import test_case, print_feedback
+from utils import train_data
+
+VOCAB_SIZE = 12000
+UNITS = 256
+
+
+def test_encoder(encoder_to_test):
+    def g():
+        vocab_sizes = [5, 20, 1000, 15000]
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        vocab_size = 15000
+        n_units = 512
+        encoder = encoder_to_test(vocab_size, n_units)
+
+        t = test_case()
+        if encoder.embedding.mask_zero != True:
+            t.failed = True
+            t.msg = "Embedding layer has incorrect value for 'mask_zero' attribute"
+            t.want = True
+            t.got = encoder.embedding.mask_zero
+        cases.append(t)
+
+        for vs, u in zip(vocab_sizes, units):
+            encoder = encoder_to_test(vs, u)
+
+            t = test_case()
+            if encoder.embedding.input_dim != vs:
+                t.failed = True
+                t.msg = "Incorrect input dim of embedding layer"
+                t.want = vs
+                t.got = encoder.embedding.input_dim
+            cases.append(t)
+
+            t = test_case()
+            if encoder.embedding.output_dim != u:
+                t.failed = True
+                t.msg = "Incorrect output dim of embedding layer"
+                t.want = u
+                t.got = encoder.embedding.output_dim
+            cases.append(t)
+
+        t = test_case()
+        if not isinstance(encoder.rnn.layer, tf.keras.layers.LSTM):
+            t.failed = True
+            t.msg = "Incorrect type of layer inside Bidirectional"
+            t.want = tf.keras.layers.LSTM
+            t.got = type(encoder.rnn.layer)
+            return [t]
+
+        for u in units:
+            encoder = encoder_to_test(vocab_size, u)
+            t = test_case()
+            if encoder.rnn.layer.units != u:
+                t.failed = True
+                t.msg = "Incorrect number of units in LSTM layer"
+                t.want = u
+                t.got = encoder.rnn.layer.units
+            cases.append(t)
+
+        t = test_case()
+        if encoder.rnn.layer.return_sequences != True:
+            t.failed = True
+            t.msg = "LSTM layer has incorrect value for 'return_sequences' attribute"
+            t.want = True
+            t.got = encoder.rnn.layer.return_sequences
+        cases.append(t)
+
+        encoder = encoder_to_test(vocab_size, n_units)
+
+        for (to_translate, _), _ in train_data.take(3):
+            first_dim_in, second_dim_in = to_translate.shape
+            encoder_output = encoder(to_translate)
+            t = test_case()
+            if len(encoder_output.shape) != 3:
+                t.failed = True
+                t.msg = "Incorrect shape of encoder output"
+                t.want = "a shape with 3 dimensions"
+                t.got = encoder_output.shape
+                return [t]
+
+            first_dim_out, second_dim_out, third_dim_out = encoder_output.shape
+
+            t = test_case()
+            if first_dim_in != first_dim_out:
+                t.failed = True
+                t.msg = "Incorrect first dimension of encoder output"
+                t.want = first_dim_in
+                t.got = first_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if second_dim_in != second_dim_out:
+                t.failed = True
+                t.msg = "Incorrect second dimension of encoder output"
+                t.want = second_dim_in
+                t.got = second_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if third_dim_out != n_units:
+                t.failed = True
+                t.msg = "Incorrect third dimension of encoder output"
+                t.want = units
+                t.got = third_dim_out
+            cases.append(t)
+
+            return cases
+
+    cases = g()
+    print_feedback(cases)
+
+
+def test_cross_attention(cross_attention_to_test):
+    def g():
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        n_units = 512
+        cross_attention = cross_attention_to_test(n_units)
+
+        t = test_case()
+        if not isinstance(cross_attention.mha, tf.keras.layers.MultiHeadAttention):
+            t.failed = True
+            t.msg = "Incorrect type of layer for Multi Head Attention"
+            t.want = tf.keras.layers.MultiHeadAttention
+            t.got = type(cross_attention.mha)
+            return [t]
+
+        #         for u in units:
+        #             cross_attention = cross_attention_to_test(u)
+
+        #             t = test_case()
+        #             if cross_attention.mha.key_dim != u:
+        #                 t.failed = True
+        #                 t.msg = "Incorrect key dim of Multi Head Attention layer"
+        #                 t.want = u
+        #                 t.got = cross_attention.mha.key_dim
+        #             cases.append(t)
+
+        cross_attention = cross_attention_to_test(n_units)
+        embed = tf.keras.layers.Embedding(VOCAB_SIZE, output_dim=UNITS, mask_zero=True)
+
+        for (to_translate, sr_translation), _ in train_data.take(3):
+            sr_translation_embed = embed(sr_translation)
+            first_dim_in, second_dim_in, third_dim_in = sr_translation_embed.shape
+            dummy_encoder_output = np.random.rand(64, 14, 512)
+            cross_attention_output = cross_attention(
+                dummy_encoder_output, sr_translation_embed
+            )
+            #             print(cross_attention_output.shape)
+
+            t = test_case()
+            if len(cross_attention_output.shape) != 3:
+                t.failed = True
+                t.msg = "Incorrect shape of cross_attention output"
+                t.want = "a shape with 3 dimensions"
+                t.got = cross_attention_output.shape
+                return [t]
+
+            first_dim_out, second_dim_out, third_dim_out = cross_attention_output.shape
+
+            t = test_case()
+            if first_dim_in != first_dim_out:
+                t.failed = True
+                t.msg = "Incorrect first dimension of cross_attention output"
+                t.want = first_dim_in
+                t.got = first_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if second_dim_in != second_dim_out:
+                t.failed = True
+                t.msg = "Incorrect second dimension of cross_attention output"
+                t.want = second_dim_in
+                t.got = second_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if third_dim_in != third_dim_out:
+                t.failed = True
+                t.msg = "Incorrect third dimension of cross_attention output"
+                t.want = third_dim_in
+                t.got = third_dim_out
+            cases.append(t)
+
+        _, n_heads, key_dim = cross_attention.mha.get_weights()[0].shape
+
+        t = test_case()
+        if n_heads != 1:
+            t.failed = True
+            t.msg = "Incorrect number of attention heads"
+            t.want = 1
+            t.got = n_heads
+        cases.append(t)
+
+        t = test_case()
+        if key_dim != n_units:
+            t.failed = True
+            t.msg = f"Incorrect size of query and key for every attention head when passing {n_units} units to the constructor"
+            t.want = n_units
+            t.got = key_dim
+        cases.append(t)
+
+        return cases
+
+    cases = g()
+    print_feedback(cases)
+
+
+def test_decoder(decoder_to_test, CrossAttention):
+    def g():
+        vocab_sizes = [5, 20, 1000, 15000]
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        vocab_size = 10000
+        n_units = 512
+        decoder = decoder_to_test(vocab_size, n_units)
+
+        t = test_case()
+        if not isinstance(decoder.embedding, tf.keras.layers.Embedding):
+            t.failed = True
+            t.msg = "Incorrect type of embedding layer"
+            t.want = tf.keras.layers.Embedding
+            t.got = type(decoder.embedding)
+            return [t]
+
+        t = test_case()
+        if decoder.embedding.mask_zero != True:
+            t.failed = True
+            t.msg = "Embedding layer has incorrect value for 'mask_zero' attribute"
+            t.want = True
+            t.got = decoder.embedding.mask_zero
+        cases.append(t)
+
+        for vs, u in zip(vocab_sizes, units):
+            decoder = decoder_to_test(vs, u)
+
+            t = test_case()
+            if decoder.embedding.input_dim != vs:
+                t.failed = True
+                t.msg = "Incorrect input dim of embedding layer"
+                t.want = vs
+                t.got = decoder.embedding.input_dim
+            cases.append(t)
+
+            t = test_case()
+            if decoder.embedding.output_dim != u:
+                t.failed = True
+                t.msg = "Incorrect output dim of embedding layer"
+                t.want = u
+                t.got = decoder.embedding.output_dim
+            cases.append(t)
+
+        t = test_case()
+        if not isinstance(decoder.pre_attention_rnn, tf.keras.layers.LSTM):
+            t.failed = True
+            t.msg = "Incorrect type of pre_attention_rnn layer"
+            t.want = tf.keras.layers.LSTM
+            t.got = type(decoder.pre_attention_rnn)
+            return [t]
+
+        for u in units:
+            decoder = decoder_to_test(vocab_size, u)
+            t = test_case()
+            if decoder.pre_attention_rnn.units != u:
+                t.failed = True
+                t.msg = "Incorrect number of units in pre_attention_rnn layer"
+                t.want = u
+                t.got = decoder.pre_attention_rnn.units
+            cases.append(t)
+
+            #             t = test_case()
+            #             if decoder.attention.units != u:
+            #                 t.failed = True
+            #                 t.msg = "Incorrect number of units in attention layer"
+            #                 t.want = u
+            #                 t.got = decoder.attention.units
+            #             cases.append(t)
+
+            t = test_case()
+            if decoder.post_attention_rnn.units != u:
+                t.failed = True
+                t.msg = "Incorrect number of units in post_attention_rnn layer"
+                t.want = u
+                t.got = decoder.post_attention_rnn.units
+            cases.append(t)
+
+        t = test_case()
+        if decoder.pre_attention_rnn.return_sequences != True:
+            t.failed = True
+            t.msg = "pre_attention_rnn layer has incorrect value for 'return_sequences' attribute"
+            t.want = True
+            t.got = decoder.pre_attention_rnn.return_sequences
+        cases.append(t)
+
+        t = test_case()
+        if decoder.pre_attention_rnn.return_state != True:
+            t.failed = True
+            t.msg = "pre_attention_rnn layer has incorrect value for 'return_state' attribute"
+            t.want = True
+            t.got = decoder.pre_attention_rnn.return_state
+        cases.append(t)
+
+        t = test_case()
+        if not isinstance(decoder.attention, CrossAttention):
+            t.failed = True
+            t.msg = "Incorrect type of attention layer"
+            t.want = CrossAttention
+            t.got = type(decoder.attention)
+            return [t]
+
+        t = test_case()
+        if decoder.post_attention_rnn.return_sequences != True:
+            t.failed = True
+            t.msg = "post_attention_rnn layer has incorrect value for 'return_sequences' attribute"
+            t.want = True
+            t.got = decoder.post_attention_rnn.return_sequences
+        cases.append(t)
+
+        t = test_case()
+        if not isinstance(decoder.post_attention_rnn, tf.keras.layers.LSTM):
+            t.failed = True
+            t.msg = "Incorrect type of pre_attention_rnn layer"
+            t.want = tf.keras.layers.LSTM
+            t.got = type(decoder.post_attention_rnn)
+            return [t]
+
+        t = test_case()
+        if not isinstance(decoder.output_layer, tf.keras.layers.Dense):
+            t.failed = True
+            t.msg = "Incorrect type of output_layer layer"
+            t.want = tf.keras.layers.Dense
+            t.got = type(decoder.output_layer)
+            return [t]
+
+        t = test_case()
+        if (
+            "log" not in decoder.output_layer.activation.__name__
+            or "softmax" not in decoder.output_layer.activation.__name__
+        ):
+            t.failed = True
+            t.msg = "output_layer layer has incorrect activation function"
+            t.want = "a log softmax activation function such as 'log_softmax_v2'"
+            t.got = decoder.output_layer.activation.__name__
+        cases.append(t)
+
+        vocab_size = 10000
+        n_units = 512
+        decoder = decoder_to_test(vocab_size, n_units)
+
+        for (_, sr_translation), _ in train_data.take(3):
+            encoder_output = np.random.rand(64, 15, 256)
+            decoder_output = decoder(encoder_output, sr_translation)
+
+            first_dim_in, second_dim_in = sr_translation.shape
+
+            t = test_case()
+            if len(decoder_output.shape) != 3:
+                t.failed = True
+                t.msg = "Incorrect shape of decoder output"
+                t.want = "a shape with 3 dimensions"
+                t.got = decoder_output.shape
+                return [t]
+
+            first_dim_out, second_dim_out, third_dim_out = decoder_output.shape
+
+            t = test_case()
+            if first_dim_in != first_dim_out:
+                t.failed = True
+                t.msg = "Incorrect first dimension of decoder output"
+                t.want = first_dim_in
+                t.got = first_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if second_dim_in != second_dim_out:
+                t.failed = True
+                t.msg = "Incorrect second dimension of decoder output"
+                t.want = second_dim_in
+                t.got = second_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if third_dim_out != vocab_size:
+                t.failed = True
+                t.msg = "Incorrect third dimension of decoder output"
+                t.want = vocab_size
+                t.got = third_dim_out
+            cases.append(t)
+        return cases
+
+    cases = g()
+    print_feedback(cases)
+
+
+def test_translator(translator_to_test, Encoder, Decoder):
+    def g():
+        vocab_sizes = [5, 20, 1000, 15000]
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        vocab_size = 10000
+        n_units = 512
+        translator = translator_to_test(vocab_size, n_units)
+
+        t = test_case()
+        if not isinstance(translator.encoder, Encoder):
+            t.failed = True
+            t.msg = "Incorrect type of encoder layer"
+            t.want = Encoder
+            t.got = type(translator.encoder)
+            return [t]
+
+        t = test_case()
+        if not isinstance(translator.decoder, Decoder):
+            t.failed = True
+            t.msg = "Incorrect type of encoder layer"
+            t.want = Decoder
+            t.got = type(translator.decoder)
+            return [t]
+
+        translator = translator_to_test(vocab_size, n_units)
+
+        for (to_translate, sr_translation), _ in train_data.take(3):
+            first_dim_in, second_dim_in = sr_translation.shape
+            translator_output = translator((to_translate, sr_translation))
+            t = test_case()
+            if len(translator_output.shape) != 3:
+                t.failed = True
+                t.msg = "Incorrect shape of translator output"
+                t.want = "a shape with 3 dimensions"
+                t.got = translator_output.shape
+                return [t]
+
+            first_dim_out, second_dim_out, third_dim_out = translator_output.shape
+
+            t = test_case()
+            if first_dim_in != first_dim_out:
+                t.failed = True
+                t.msg = "Incorrect first dimension of translator output"
+                t.want = first_dim_in
+                t.got = first_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if second_dim_in != second_dim_out:
+                t.failed = True
+                t.msg = "Incorrect second dimension of translator output"
+                t.want = second_dim_in
+                t.got = second_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if third_dim_out != vocab_size:
+                t.failed = True
+                t.msg = "Incorrect third dimension of translator output"
+                t.want = vocab_size
+                t.got = third_dim_out
+            cases.append(t)
+
+            return cases
+
+    cases = g()
+    print_feedback(cases)
+
+    
+
+def test_translate(learner_func, model):
+    def g():
+        
+        cases = []
+        
+        txt = "Hi, my name is Younes"
+        try:
+            translation, logit, tokens = learner_func(model, txt, temperature=0.9)
+        except Exception as e:
+            t = test_case()
+            t.failed = True
+            t.msg = "There was an exception when running your function"
+            t.want = "No exceptions"
+            t.got = f"{str(e)}"
+            return [t]
+        
+        txt = "Hi, my name is Alejandra"
+        translation, logit, tokens = learner_func(model, txt, temperature=0.0)
+        
+        t = test_case()
+        
+        if not isinstance(translation, str):
+            t.failed = True
+            t.msg = "'translation' has incorrect type"
+            t.want = str
+            t.got = type(translation)
+        cases.append(t)
+        
+        if not isinstance(logit, np.number):
+            t.failed = True
+            t.msg = "'logit' has incorrect type"
+            t.want = np.number
+            t.got = type(logit)
+        cases.append(t)
+        
+        if not isinstance(tokens, tf.Tensor):
+            t.failed = True
+            t.msg = "'tokens' has incorrect type"
+            t.want = tf.Tensor
+            t.got = type(tokens)
+        cases.append(t)
+        
+        translation2, logit2, tokens2 = learner_func(model, txt, temperature=0.0)
+        
+        t = test_case()
+        if translation != translation2:
+            t.failed = True
+            t.msg = "translate didn't return the same translation when using temperature of 0.0"
+            t.want = translation
+            t.got = translation2
+        cases.append(t)
+        
+        t = test_case()
+        if logit != logit2:
+            t.failed = True
+            t.msg = "translate didn't return the same logit when using temperature of 0.0"
+            t.want = logit
+            t.got = logit2
+        cases.append(t)
+        
+        t = test_case()
+        if not np.allclose(tokens, tokens2):
+            t.failed = True
+            t.msg = "translate didn't return the same tokens when using temperature of 0.0"
+            t.want = tokens
+            t.got = tokens2
+        cases.append(t)
+        
+        
+        return cases
+    
+    cases = g()
+    print_feedback(cases)
+
+    
+
+
+def test_rouge1_similarity(learner_func):
+    
+    def g():
+        
+        tensors = [
+            [0],
+            [0, 1],
+            [0, 1, 2],
+            [1, 2, 4, 5],
+            [5, 5, 7, 0, 232]
+        ]
+        
+        expected = [0.6666666666666666, 0.5, 0, 0.33333333333333337, 0.8, 0.3333333333333333, 0.28571428571428575, 0.5714285714285715, 0.25]
+
+        cases = []
+        pairs = list(combinations(tensors, 2))
+        
+        for (candidate, reference), solution in zip(pairs, expected):
+            answer = learner_func(candidate, reference)
+            t = test_case()
+            if not math.isclose(answer, solution):
+                t.failed = True
+                t.msg = f"Incorrect similarity for candidate={candidate} and reference={reference}"
+                t.want = solution
+                t.got = answer
+            cases.append(t)
+
+        return cases
+    
+    cases = g()
+    print_feedback(cases)
+
+
+def test_average_overlap(learner_func):
+
+    def jaccard_similarity(candidate, reference):
+        
+        # Convert the lists to sets to get the unique tokens
+        candidate_set = set(candidate)
+        reference_set = set(reference)
+        
+        # Get the set of tokens common to both candidate and reference
+        common_tokens = candidate_set.intersection(reference_set)
+        
+        # Get the set of all tokens found in either candidate or reference
+        all_tokens = candidate_set.union(reference_set)
+        
+        # Compute the percentage of overlap (divide the number of common tokens by the number of all tokens)
+        overlap = len(common_tokens) / len(all_tokens)
+            
+        return overlap
+    
+    def g():
+        
+        l1 = [1, 2, 3]
+        l2 = [1, 2, 4]
+        l3 = [1, 2, 4, 5]
+        l4 = [5,6]
+
+        elements = [l1, l2, l3, l4]
+
+        all_combinations = []
+
+        for r in range(2, len(elements) + 1):
+            # Generate combinations of length r
+            combinations_r = combinations(elements, r)
+            
+            # Append the combinations to the result list
+            all_combinations.extend(combinations_r)
+        
+        expected = [{0: 0.5, 1: 0.5},
+                     {0: 0.4, 1: 0.4},
+                     {0: 0.0, 1: 0.0},
+                     {0: 0.75, 1: 0.75},
+                     {0: 0.0, 1: 0.0},
+                     {0: 0.2, 1: 0.2},
+                     {0: 0.45, 1: 0.625, 2: 0.575},
+                     {0: 0.25, 1: 0.25, 2: 0.0},
+                     {0: 0.2, 1: 0.3, 2: 0.1},
+                     {0: 0.375, 1: 0.475, 2: 0.1},
+                     {0: 0.3, 1: 0.417, 2: 0.45, 3: 0.067}]
+
+        cases = []
+        
+        for combination, solution in zip(all_combinations, expected):
+            answer = learner_func(combination, jaccard_similarity)
+            t = test_case()
+            if answer != solution:
+                t.failed = True
+                t.msg = f"Incorrect overlap for lists={combination}"
+                t.want = solution
+                t.got = answer
+            cases.append(t)
+
+        return cases
+    
+    cases = g()
+    print_feedback(cases)

NLP with Attention Models/NMT_with_Attention/NMT with MBR/Files/tf/C4W1_Assignment.ipynb

@@ -0,0 +1,2312 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9cb49525",
+   "metadata": {},
+   "source": [
+    "# Assignment 1: Neural Machine Translation\n",
+    "\n",
+    "Welcome to the first assignment of Course 4. Here, you will build an English-to-Portuguese neural machine translation (NMT) model using Long Short-Term Memory (LSTM) networks with attention. Machine translation is an important task in natural language processing and could be useful not only for translating one language to another but also for word sense disambiguation (e.g. determining whether the word \"bank\" refers to the financial bank, or the land alongside a river). Implementing this using just a Recurrent Neural Network (RNN) with LSTMs can work for short to medium length sentences but can result in vanishing gradients for very long sequences. To help with this, you will be adding an attention mechanism to allow the decoder to access all relevant parts of the input sentence regardless of its length. By completing this assignment, you will:\n",
+    "\n",
+    "- Implement an encoder-decoder system with attention\n",
+    "- Build the NMT model from scratch using Tensorflow\n",
+    "- Generate translations using greedy and Minimum Bayes Risk (MBR) decoding\n",
+    "\n",
+    "## Table of Contents\n",
+    "- [1 - Data Preparation](#1)\n",
+    "- [2 - NMT model with attention](#2)\n",
+    "    - [Exercise 1 - Encoder](#ex1)\n",
+    "    - [Exercise 2 - CrossAttention](#ex2)\n",
+    "    - [Exercise 3 - Decoder](#ex3)    \n",
+    "    - [Exercise 4 - Translator](#ex4)\n",
+    "- [3 - Training](#3)\n",
+    "- [4 - Using the model for inference ](#4)\n",
+    "    - [Exercise 5 - translate](#ex5)\n",
+    "- [5 - Minimum Bayes-Risk Decoding](#5)\n",
+    "    - [Exercise 6 - rouge1_similarity](#ex6)\n",
+    "    - [Exercise 7 - average_overlap](#ex7)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f9ef370d",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # Setting this env variable prevents TF warnings from showing up\n",
+    "\n",
+    "import numpy as np\n",
+    "import tensorflow as tf\n",
+    "from collections import Counter\n",
+    "from utils import (sentences, train_data, val_data, english_vectorizer, portuguese_vectorizer, \n",
+    "                   masked_loss, masked_acc, tokens_to_text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "8adb8fd6",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import w1_unittest"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e76be1dc",
+   "metadata": {},
+   "source": [
+    "<a name=\"1\"></a>\n",
+    "## 1. Data Preparation\n",
+    "\n",
+    "The text pre-processing bits have already been taken care of (if you are interested in this be sure to check the `utils.py` file). The steps performed can be summarized as:\n",
+    "\n",
+    "- Reading the raw data from the text files\n",
+    "- Cleaning the data (using lowercase, adding space around punctuation, trimming whitespaces, etc)\n",
+    "- Splitting it into training and validation sets\n",
+    "- Adding the start-of-sentence and end-of-sentence tokens to every sentence\n",
+    "- Tokenizing the sentences\n",
+    "- Creating a Tensorflow dataset out of the tokenized sentences\n",
+    "\n",
+    "Take a moment to inspect the raw sentences:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "226033a1",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "English (to translate) sentence:\n",
+      "\n",
+      "No matter how much you try to convince people that chocolate is vanilla, it'll still be chocolate, even though you may manage to convince yourself and a few others that it's vanilla.\n",
+      "\n",
+      "Portuguese (translation) sentence:\n",
+      "\n",
+      "Não importa o quanto você tenta convencer os outros de que chocolate é baunilha, ele ainda será chocolate, mesmo que você possa convencer a si mesmo e poucos outros de que é baunilha.\n"
+     ]
+    }
+   ],
+   "source": [
+    "portuguese_sentences, english_sentences = sentences\n",
+    "\n",
+    "print(f\"English (to translate) sentence:\\n\\n{english_sentences[-5]}\\n\")\n",
+    "print(f\"Portuguese (translation) sentence:\\n\\n{portuguese_sentences[-5]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ba90eb9",
+   "metadata": {},
+   "source": [
+    "You don't have much use for the raw sentences so delete them to save memory:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "d9f081b0",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "del portuguese_sentences\n",
+    "del english_sentences\n",
+    "del sentences"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a2ff83d2",
+   "metadata": {},
+   "source": [
+    "Notice that you imported an `english_vectorizer` and a `portuguese_vectorizer` from `utils.py`. These were created using [tf.keras.layers.TextVectorization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization) and they provide interesting features such as ways to visualize the vocabulary and convert text into tokenized ids and vice versa. In fact, you can inspect the first ten words of the vocabularies for both languages:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "2c1cfc17",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "First 10 words of the english vocabulary:\n",
+      "\n",
+      "['', '[UNK]', '[SOS]', '[EOS]', '.', 'tom', 'i', 'to', 'you', 'the']\n",
+      "\n",
+      "First 10 words of the portuguese vocabulary:\n",
+      "\n",
+      "['', '[UNK]', '[SOS]', '[EOS]', '.', 'tom', 'que', 'o', 'nao', 'eu']\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(f\"First 10 words of the english vocabulary:\\n\\n{english_vectorizer.get_vocabulary()[:10]}\\n\")\n",
+    "print(f\"First 10 words of the portuguese vocabulary:\\n\\n{portuguese_vectorizer.get_vocabulary()[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3152b075",
+   "metadata": {},
+   "source": [
+    "Notice that the first 4 words are reserved for special words. In order, these are:\n",
+    "\n",
+    "- the empty string\n",
+    "- a special token to represent an unknown word\n",
+    "- a special token to represent the start of a sentence\n",
+    "- a special token to represent the end of a sentence\n",
+    "\n",
+    "You can see how many words are in a vocabulary by using the `vocabulary_size` method:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "5facaa0c",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Portuguese vocabulary is made up of 12000 words\n",
+      "English vocabulary is made up of 12000 words\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Size of the vocabulary\n",
+    "vocab_size_por = portuguese_vectorizer.vocabulary_size()\n",
+    "vocab_size_eng = english_vectorizer.vocabulary_size()\n",
+    "\n",
+    "print(f\"Portuguese vocabulary is made up of {vocab_size_por} words\")\n",
+    "print(f\"English vocabulary is made up of {vocab_size_eng} words\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53e4b615",
+   "metadata": {
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "You can define [tf.keras.layers.StringLookup](https://www.tensorflow.org/api_docs/python/tf/keras/layers/StringLookup) objects that will help you map from words to ids and vice versa. Do this for the portuguese vocabulary since this will be useful later on when you decode the predictions from your model:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "218f7a36",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# This helps you convert from words to ids\n",
+    "word_to_id = tf.keras.layers.StringLookup(\n",
+    "    vocabulary=portuguese_vectorizer.get_vocabulary(), \n",
+    "    mask_token=\"\", \n",
+    "    oov_token=\"[UNK]\"\n",
+    ")\n",
+    "\n",
+    "# This helps you convert from ids to words\n",
+    "id_to_word = tf.keras.layers.StringLookup(\n",
+    "    vocabulary=portuguese_vectorizer.get_vocabulary(),\n",
+    "    mask_token=\"\",\n",
+    "    oov_token=\"[UNK]\",\n",
+    "    invert=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4af8b623",
+   "metadata": {},
+   "source": [
+    "Try it out for the special tokens and a random word:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "20076b9a",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The id for the [UNK] token is 1\n",
+      "The id for the [SOS] token is 2\n",
+      "The id for the [EOS] token is 3\n",
+      "The id for baunilha (vanilla) is 7079\n"
+     ]
+    }
+   ],
+   "source": [
+    "unk_id = word_to_id(\"[UNK]\")\n",
+    "sos_id = word_to_id(\"[SOS]\")\n",
+    "eos_id = word_to_id(\"[EOS]\")\n",
+    "baunilha_id = word_to_id(\"baunilha\")\n",
+    "\n",
+    "print(f\"The id for the [UNK] token is {unk_id}\")\n",
+    "print(f\"The id for the [SOS] token is {sos_id}\")\n",
+    "print(f\"The id for the [EOS] token is {eos_id}\")\n",
+    "print(f\"The id for baunilha (vanilla) is {baunilha_id}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2f1d744c",
+   "metadata": {},
+   "source": [
+    "Finally take a look at how the data that is going to be fed to the neural network looks like. Both `train_data` and `val_data` are of type `tf.data.Dataset` and are already arranged in batches of 64 examples. To get the first batch out of a tf dataset you can use the `take` method. To get the first example out of the batch you can slice the tensor and use the `numpy` method for nicer printing:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "739777eb",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tokenized english sentence:\n",
+      "[   2  210    9  146  123   38    9 1672    4    3    0    0    0    0]\n",
+      "\n",
+      "\n",
+      "Tokenized portuguese sentence (shifted to the right):\n",
+      "[   2 1085    7  128   11  389   37 2038    4    0    0    0    0    0\n",
+      "    0]\n",
+      "\n",
+      "\n",
+      "Tokenized portuguese sentence:\n",
+      "[1085    7  128   11  389   37 2038    4    3    0    0    0    0    0\n",
+      "    0]\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "for (to_translate, sr_translation), translation in train_data.take(1):\n",
+    "    print(f\"Tokenized english sentence:\\n{to_translate[0, :].numpy()}\\n\\n\")\n",
+    "    print(f\"Tokenized portuguese sentence (shifted to the right):\\n{sr_translation[0, :].numpy()}\\n\\n\")\n",
+    "    print(f\"Tokenized portuguese sentence:\\n{translation[0, :].numpy()}\\n\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bdd9ee3c",
+   "metadata": {
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "There are a couple of important details to notice.\n",
+    "\n",
+    "- Padding has already been applied to the tensors and the value used for this is 0\n",
+    "- Each example consists of 3 different tensors:\n",
+    "    - The sentence to translate\n",
+    "    - The shifted-to-the-right translation\n",
+    "    - The translation\n",
+    "    \n",
+    "The first two can be considered as the features, while the third one as the target. By doing this your model can perform Teacher Forcing as you saw in the lectures.\n",
+    "\n",
+    "Now it is time to begin coding!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dd41cb52",
+   "metadata": {
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "<a name=\"2\"></a>\n",
+    "## 2. NMT model with attention\n",
+    "\n",
+    "The model you will build uses an encoder-decoder architecture. This Recurrent Neural Network (RNN) takes in a tokenized version of a sentence in its encoder, then passes it on to the decoder for translation. As mentioned in the lectures, just using a a regular sequence-to-sequence model with LSTMs will work effectively for short to medium sentences but will start to degrade for longer ones. You can picture it like the figure below where all of the context of the input sentence is compressed into one vector that is passed into the decoder block. You can see how this will be an issue for very long sentences (e.g. 100 tokens or more) because the context of the first parts of the input will have very little effect on the final vector passed to the decoder.\n",
+    "\n",
+    "<img src='images/plain_rnn.png'>\n",
+    "\n",
+    "Adding an attention layer to this model avoids this problem by giving the decoder access to all parts of the input sentence. To illustrate, let's just use a 4-word input sentence as shown below. Remember that a hidden state is produced at each timestep of the encoder (represented by the orange rectangles). These are all passed to the attention layer and each are given a score given the current activation (i.e. hidden state) of the decoder. For instance, let's consider the figure below where the first prediction \"como\" is already made. To produce the next prediction, the attention layer will first receive all the encoder hidden states (i.e. orange rectangles) as well as the decoder hidden state when producing the word \"como\" (i.e. first green rectangle). Given this information, it will score each of the encoder hidden states to know which one the decoder should focus on to produce the next word. As a result of training, the model might have learned that it should align to the second encoder hidden state and subsequently assigns a high probability to the word \"você\". If we are using greedy decoding, we will output the said word as the next symbol, then restart the process to produce the next word until we reach an end-of-sentence prediction.\n",
+    "\n",
+    "<img src='images/attention_overview.png'>\n",
+    "\n",
+    "\n",
+    "There are different ways to implement attention and the one we'll use for this assignment is the Scaled Dot Product Attention which has the form:\n",
+    "\n",
+    "$$Attention(Q, K, V) = softmax(\\frac{QK^T}{\\sqrt{d_k}})V$$\n",
+    "\n",
+    "You will dive deeper into this equation in the next week but for now, you can think of it as computing scores using queries (Q) and keys (K), followed by a multiplication of values (V) to get a context vector at a particular timestep of the decoder. This context vector is fed to the decoder RNN to get a set of probabilities for the next predicted word. The division by square root of the keys dimensionality ($\\sqrt{d_k}$) is for improving model performance and you'll also learn more about it next week. For our machine translation application, the encoder activations (i.e. encoder hidden states) will be the keys and values, while the decoder activations (i.e. decoder hidden states) will be the queries.\n",
+    "\n",
+    "You will see in the upcoming sections that this complex architecture and mechanism can be implemented with just a few lines of code. \n",
+    "\n",
+    "First you will define two important global variables:\n",
+    "\n",
+    "- The size of the vocabulary\n",
+    "- The number of units in the LSTM layers (the same number will be used for all LSTM layers)\n",
+    "\n",
+    "In this assignment, the vocabulary sizes for English and Portuguese are the same. Therefore, we use a single constant VOCAB_SIZE throughout the notebook. While in other settings, vocabulary sizes could differ, that is not the case in our assignment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "2e484abf",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "VOCAB_SIZE = 12000\n",
+    "UNITS = 256"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cc251965",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex1\"></a>\n",
+    "## Exercise 1 - Encoder\n",
+    "\n",
+    "Your first exercise is to code the encoder part of the neural network. For this, complete the `Encoder` class below. Notice that in the constructor (the `__init__` method) you need to define all of the sublayers of the encoder and then use these sublayers during the forward pass (the `call` method).\n",
+    "\n",
+    "The encoder consists of the following layers:\n",
+    "\n",
+    "- [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding). For this layer you need to define the appropriate `input_dim` and `output_dim` and let it know that you are using '0' as padding, which can be done by using the appropriate value for the `mask_zero` parameter.\n",
+    "    \n",
+    "+ [Bidirectional](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Bidirectional) [LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM). In TF you can implement bidirectional behaviour for RNN-like layers. This part is already taken care of but you will need to specify the appropriate type of layer as well as its parameters. In particular you need to set the appropriate number of units and make sure that the LSTM returns the full sequence and not only the last output, which can be done by using the appropriate value for the `return_sequences` parameter.\n",
+    "\n",
+    "\n",
+    "You need to define the forward pass using the syntax of TF's [functional API](https://www.tensorflow.org/guide/keras/functional_api). What this means is that you chain function calls together to define your network like this:\n",
+    "\n",
+    "```python\n",
+    "encoder_input = keras.Input(shape=(28, 28, 1), name=\"original_img\")\n",
+    "x = layers.Conv2D(16, 3, activation=\"relu\")(encoder_input)\n",
+    "x = layers.MaxPooling2D(3)(x)\n",
+    "x = layers.Conv2D(16, 3, activation=\"relu\")(x)\n",
+    "encoder_output = layers.GlobalMaxPooling2D()(x)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "b1db0a1d",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: Encoder\n",
+    "class Encoder(tf.keras.layers.Layer):\n",
+    "    def __init__(self, vocab_size, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            vocab_size (int): Size of the vocabulary\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super(Encoder, self).__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        self.embedding = tf.keras.layers.Embedding(  \n",
+    "            input_dim=vocab_size,\n",
+    "            output_dim=units,\n",
+    "            mask_zero=True\n",
+    "        )  \n",
+    "\n",
+    "        self.rnn = tf.keras.layers.Bidirectional(  \n",
+    "            merge_mode=\"sum\",  \n",
+    "            layer=tf.keras.layers.LSTM(\n",
+    "                units=units,\n",
+    "                return_sequences=True\n",
+    "            ),  \n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "    def call(self, context):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            context (tf.Tensor): The sentence to translate\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: Encoded sentence to translate\n",
+    "        \"\"\"\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Pass the context through the embedding layer\n",
+    "        x = self.embedding(context)\n",
+    "\n",
+    "        # Pass the output of the embedding through the RNN\n",
+    "        x = self.rnn(x)\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        return x"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "65034ffd",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tensor of sentences in english has shape: (64, 14)\n",
+      "\n",
+      "Encoder output has shape: (64, 14, 256)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "encoder = Encoder(VOCAB_SIZE, UNITS)\n",
+    "\n",
+    "# Pass a batch of sentences to translate from english to portuguese\n",
+    "encoder_output = encoder(to_translate)\n",
+    "\n",
+    "print(f'Tensor of sentences in english has shape: {to_translate.shape}\\n')\n",
+    "print(f'Encoder output has shape: {encoder_output.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a909aea1",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of sentences in english has shape: (64, 14)\n",
+    "\n",
+    "Encoder output has shape: (64, 14, 256)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "3031bb14",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[92m All tests passed!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test your code!\n",
+    "\n",
+    "w1_unittest.test_encoder(Encoder)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1afe83f4",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex2\"></a>\n",
+    "## Exercise 2 - CrossAttention\n",
+    "\n",
+    "Your next exercise is to code the layer that will perform cross attention between the original sentences and the translations. For this, complete the `CrossAttention` class below. Notice that in the constructor (the `__init__` method) you need to define all of the sublayers and then use these sublayers during the forward pass (the `call` method). For this particular case some of these bits are already taken care of.\n",
+    "\n",
+    "The cross attention consists of the following layers:\n",
+    "\n",
+    "- [MultiHeadAttention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention). For this layer you need to define the appropriate `key_dim`, which is the size of the key and query tensors. You will also need to set the number of heads to 1 since you aren't implementing multi head attention but attention between two tensors. The reason why this layer is preferred over [Attention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention) is that it allows simpler code during the forward pass.\n",
+    "    \n",
+    "A couple of things to notice:\n",
+    "- You need a way to pass both the output of the attention alongside the shifted-to-the-right translation (since this cross attention happens in the decoder side). For this you will use an [Add](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Add) layer so that the original dimension is preserved, which would not happen if you use something like a [Concatenate](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Concatenate) layer.\n",
+    "\n",
+    "+ Layer normalization is also performed for better stability of the network by using a [LayerNormalization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LayerNormalization) layer.\n",
+    "\n",
+    "- You don't need to worry about these last steps as these are already solved.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "74e71f3d",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: CrossAttention\n",
+    "class CrossAttention(tf.keras.layers.Layer):\n",
+    "    def __init__(self, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super().__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        self.mha = ( \n",
+    "            tf.keras.layers.MultiHeadAttention(\n",
+    "                key_dim=units,\n",
+    "                num_heads=1\n",
+    "            ) \n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        self.layernorm = tf.keras.layers.LayerNormalization()\n",
+    "        self.add = tf.keras.layers.Add()\n",
+    "\n",
+    "    def call(self, context, target):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            context (tf.Tensor): Encoded sentence to translate\n",
+    "            target (tf.Tensor): The embedded shifted-to-the-right translation\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: Cross attention between context and target\n",
+    "        \"\"\"\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Call the MH attention by passing in the query and value\n",
+    "        # For this case the query should be the translation and the value the encoded sentence to translate\n",
+    "        # Hint: Check the call arguments of MultiHeadAttention in the docs\n",
+    "        attn_output =self.mha(\n",
+    "            query=target,\n",
+    "            value=context\n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        x = self.add([target, attn_output])\n",
+    "\n",
+    "        x = self.layernorm(x)\n",
+    "\n",
+    "        return x"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "4c62796f",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tensor of contexts has shape: (64, 14, 256)\n",
+      "Tensor of translations has shape: (64, 15, 256)\n",
+      "Tensor of attention scores has shape: (64, 15, 256)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "attention_layer = CrossAttention(UNITS)\n",
+    "\n",
+    "# The attention layer expects the embedded sr-translation and the context\n",
+    "# The context (encoder_output) is already embedded so you need to do this for sr_translation:\n",
+    "sr_translation_embed = tf.keras.layers.Embedding(VOCAB_SIZE, output_dim=UNITS, mask_zero=True)(sr_translation)\n",
+    "\n",
+    "# Compute the cross attention\n",
+    "attention_result = attention_layer(encoder_output, sr_translation_embed)\n",
+    "\n",
+    "print(f'Tensor of contexts has shape: {encoder_output.shape}')\n",
+    "print(f'Tensor of translations has shape: {sr_translation_embed.shape}')\n",
+    "print(f'Tensor of attention scores has shape: {attention_result.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "41d4f99a",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of contexts has shape: (64, 14, 256)\n",
+    "Tensor of translations has shape: (64, 15, 256)\n",
+    "Tensor of attention scores has shape: (64, 15, 256)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "4f658975",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[92m All tests passed!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test your code!\n",
+    "\n",
+    "w1_unittest.test_cross_attention(CrossAttention)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aa296ee2",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex3\"></a>\n",
+    "## Exercise 3 - Decoder\n",
+    "\n",
+    "\n",
+    "Now you will implement the decoder part of the neural network by completing the `Decoder` class below. Notice that in the constructor (the `__init__` method) you need to define all of the sublayers of the decoder and then use these sublayers during the forward pass (the `call` method).\n",
+    "\n",
+    "The decoder consists of the following layers:\n",
+    "\n",
+    "- [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding). For this layer you need to define the appropriate `input_dim` and `output_dim` and let it know that you are using '0' as padding, which can be done by using the appropriate value for the `mask_zero` parameter.\n",
+    "  \n",
+    "  \n",
+    "+ Pre-attention [LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM). Unlike in the encoder in which you used a Bidirectional LSTM, here you will use a vanilla LSTM. Don't forget to set the appropriate number of units and make sure that the LSTM returns the full sequence and not only the last output, which can be done by using the appropriate value for the `return_sequences` parameter. It is very important that this layer returns the state since this will be needed for inference so make sure to set the `return_state` parameter accordingly. Notice that LSTM layers return state as a tuple of two tensors called `memory_state` and `carry_state`, **however these names have been changed to better reflect what you have seen in the lectures to `hidden_state` and `cell_state` respectively**.\n",
+    "\n",
+    "- The attention layer that performs cross attention between the sentence to translate and the right-shifted translation. Here you need to use the `CrossAttention` layer you defined in the previous exercise.\n",
+    "\n",
+    "+ Post-attention [LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM). Another LSTM layer. For this one you don't need it to return the state.\n",
+    "\n",
+    "- Finally a [Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) layer. This one should have the same number of units as the size of the vocabulary since you expect it to compute the logits for every possible word in the vocabulary. Make sure to use a `logsoftmax` activation function for this one, which you can get as [tf.nn.log_softmax](https://www.tensorflow.org/api_docs/python/tf/nn/log_softmax).\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "e9639bdb",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: Decoder\n",
+    "class Decoder(tf.keras.layers.Layer):\n",
+    "    def __init__(self, vocab_size, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            vocab_size (int): Size of the vocabulary\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super(Decoder, self).__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # The embedding layer\n",
+    "        self.embedding = tf.keras.layers.Embedding(\n",
+    "            input_dim=vocab_size,\n",
+    "            output_dim=units,\n",
+    "            mask_zero=True\n",
+    "        )  \n",
+    "\n",
+    "        # The RNN before attention\n",
+    "        self.pre_attention_rnn = tf.keras.layers.LSTM(\n",
+    "            units=units,\n",
+    "            return_sequences=True,\n",
+    "            return_state=True\n",
+    "        )  \n",
+    "\n",
+    "        # The attention layer\n",
+    "        self.attention = CrossAttention(units)\n",
+    "\n",
+    "        # The RNN after attention\n",
+    "        self.post_attention_rnn = tf.keras.layers.LSTM(\n",
+    "            units=units,\n",
+    "            return_sequences=True\n",
+    "        )  \n",
+    "\n",
+    "        # The dense layer with logsoftmax activation\n",
+    "        self.output_layer = tf.keras.layers.Dense(\n",
+    "            units=vocab_size,\n",
+    "            activation=tf.nn.log_softmax\n",
+    "        )  \n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "    def call(self, context, target, state=None, return_state=False):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            context (tf.Tensor): Encoded sentence to translate\n",
+    "            target (tf.Tensor): The shifted-to-the-right translation\n",
+    "            state (list[tf.Tensor, tf.Tensor], optional): Hidden state of the pre-attention LSTM. Defaults to None.\n",
+    "            return_state (bool, optional): If set to true return the hidden states of the LSTM. Defaults to False.\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: The log_softmax probabilities of predicting a particular token\n",
+    "        \"\"\"\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Get the embedding of the input\n",
+    "        x = self.embedding(target)\n",
+    "\n",
+    "        # Pass the embedded input into the pre attention LSTM\n",
+    "        # Hints:\n",
+    "        # - The LSTM you defined earlier should return the output alongside the state (made up of two tensors)\n",
+    "        # - Pass in the state to the LSTM (needed for inference)\n",
+    "        x, hidden_state, cell_state = self.pre_attention_rnn(x, initial_state=state)\n",
+    "\n",
+    "        # Perform cross attention between the context and the output of the LSTM (in that order)\n",
+    "        x = self.attention(context, x)\n",
+    "\n",
+    "        # Do a pass through the post attention LSTM\n",
+    "        x = self.post_attention_rnn(x)\n",
+    "\n",
+    "        # Compute the logits\n",
+    "        logits = self.output_layer(x)\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        if return_state:\n",
+    "            return logits, [hidden_state, cell_state]\n",
+    "\n",
+    "        return logits"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "id": "f6165cf2",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tensor of contexts has shape: (64, 14, 256)\n",
+      "Tensor of right-shifted translations has shape: (64, 15)\n",
+      "Tensor of logits has shape: (64, 15, 12000)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "decoder = Decoder(VOCAB_SIZE, UNITS)\n",
+    "\n",
+    "# Notice that you don't need the embedded version of sr_translation since this is done inside the class\n",
+    "logits = decoder(encoder_output, sr_translation)\n",
+    "\n",
+    "print(f'Tensor of contexts has shape: {encoder_output.shape}')\n",
+    "print(f'Tensor of right-shifted translations has shape: {sr_translation.shape}')\n",
+    "print(f'Tensor of logits has shape: {logits.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f2b5d7d",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of contexts has shape: (64, 14, 256)\n",
+    "Tensor of right-shifted translations has shape: (64, 15)\n",
+    "Tensor of logits has shape: (64, 15, 12000)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "id": "1b61093a",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[92m All tests passed!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test your code!\n",
+    "\n",
+    "w1_unittest.test_decoder(Decoder, CrossAttention)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9dcce3a7",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex4\"></a>\n",
+    "## Exercise 4 - Translator\n",
+    "\n",
+    "Now you have to put together all of the layers you previously coded into an actual model. For this, complete the `Translator` class below. Notice how unlike the Encoder and Decoder classes inherited from `tf.keras.layers.Layer`, the Translator class inherits from `tf.keras.Model`.\n",
+    "\n",
+    "Remember that `train_data` will yield a tuple with the sentence to translate and the shifted-to-the-right translation, which are the \"features\" of the model. This means that the inputs of your network will be tuples containing context and targets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "205fcf31",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED CLASS: Translator\n",
+    "class Translator(tf.keras.Model):\n",
+    "    def __init__(self, vocab_size, units):\n",
+    "        \"\"\"Initializes an instance of this class\n",
+    "\n",
+    "        Args:\n",
+    "            vocab_size (int): Size of the vocabulary\n",
+    "            units (int): Number of units in the LSTM layer\n",
+    "        \"\"\"\n",
+    "        super().__init__()\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # Define the encoder with the appropriate vocab_size and number of units\n",
+    "        self.encoder = Encoder(vocab_size,units)\n",
+    "\n",
+    "        # Define the decoder with the appropriate vocab_size and number of units\n",
+    "        self.decoder = Decoder(vocab_size,units)\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "    def call(self, inputs):\n",
+    "        \"\"\"Forward pass of this layer\n",
+    "\n",
+    "        Args:\n",
+    "            inputs (tuple(tf.Tensor, tf.Tensor)): Tuple containing the context (sentence to translate) and the target (shifted-to-the-right translation)\n",
+    "\n",
+    "        Returns:\n",
+    "            tf.Tensor: The log_softmax probabilities of predicting a particular token\n",
+    "        \"\"\"\n",
+    "\n",
+    "        ### START CODE HERE ###\n",
+    "\n",
+    "        # In this case inputs is a tuple consisting of the context and the target, unpack it into single variables\n",
+    "        context, target = inputs\n",
+    "\n",
+    "        # Pass the context through the encoder\n",
+    "        encoded_context = self.encoder(context)\n",
+    "\n",
+    "        # Compute the logits by passing the encoded context and the target to the decoder\n",
+    "        logits = self.decoder(target=target,context=encoded_context)\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "\n",
+    "        return logits"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "id": "4d4a231c",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tensor of sentences to translate has shape: (64, 14)\n",
+      "Tensor of right-shifted translations has shape: (64, 15)\n",
+      "Tensor of logits has shape: (64, 15, 12000)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Do a quick check of your implementation\n",
+    "\n",
+    "# Create an instance of your class\n",
+    "translator = Translator(VOCAB_SIZE, UNITS)\n",
+    "\n",
+    "# Compute the logits for every word in the vocabulary\n",
+    "logits = translator((to_translate, sr_translation))\n",
+    "\n",
+    "print(f'Tensor of sentences to translate has shape: {to_translate.shape}')\n",
+    "print(f'Tensor of right-shifted translations has shape: {sr_translation.shape}')\n",
+    "print(f'Tensor of logits has shape: {logits.shape}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e3a162dd",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "Tensor of sentences to translate has shape: (64, 14)\n",
+    "Tensor of right-shifted translations has shape: (64, 15)\n",
+    "Tensor of logits has shape: (64, 15, 12000)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 50,
+   "id": "37009022",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[92m All tests passed!\n"
+     ]
+    }
+   ],
+   "source": [
+    "w1_unittest.test_translator(Translator, Encoder, Decoder)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f81bc228",
+   "metadata": {},
+   "source": [
+    "<a name=\"3\"></a>\n",
+    "## 3. Training\n",
+    "\n",
+    "Now that you have an untrained instance of the NMT model, it is time to train it. You can use the `compile_and_train` function below to achieve this:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 79,
+   "id": "8a61ef65",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def compile_and_train(model, epochs=20, steps_per_epoch=500):\n",
+    "    model.compile(optimizer=\"adam\", loss=masked_loss, metrics=[masked_acc, masked_loss])\n",
+    "\n",
+    "    history = model.fit(\n",
+    "        train_data.repeat(),\n",
+    "        epochs=epochs,\n",
+    "        steps_per_epoch=steps_per_epoch,\n",
+    "        validation_data=val_data,\n",
+    "        validation_steps=50,\n",
+    "        callbacks=[tf.keras.callbacks.EarlyStopping(patience=3)],\n",
+    "    )\n",
+    "\n",
+    "    return model, history"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 80,
+   "id": "87d9bf9f",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Epoch 1/20\n",
+      "500/500 [==============================] - 44s 62ms/step - loss: 0.7642 - masked_acc: 0.8158 - masked_loss: 0.7650 - val_loss: 1.0111 - val_masked_acc: 0.7826 - val_masked_loss: 1.0127\n",
+      "Epoch 2/20\n",
+      "500/500 [==============================] - 17s 33ms/step - loss: 0.7811 - masked_acc: 0.8130 - masked_loss: 0.7816 - val_loss: 0.9974 - val_masked_acc: 0.7832 - val_masked_loss: 0.9979\n",
+      "Epoch 3/20\n",
+      "500/500 [==============================] - 16s 32ms/step - loss: 0.7887 - masked_acc: 0.8119 - masked_loss: 0.7894 - val_loss: 0.9753 - val_masked_acc: 0.7860 - val_masked_loss: 0.9780\n",
+      "Epoch 4/20\n",
+      "500/500 [==============================] - 15s 31ms/step - loss: 0.7835 - masked_acc: 0.8125 - masked_loss: 0.7841 - val_loss: 0.9936 - val_masked_acc: 0.7821 - val_masked_loss: 0.9955\n",
+      "Epoch 5/20\n",
+      "500/500 [==============================] - 16s 32ms/step - loss: 0.7452 - masked_acc: 0.8175 - masked_loss: 0.7461 - val_loss: 0.9643 - val_masked_acc: 0.7896 - val_masked_loss: 0.9638\n",
+      "Epoch 6/20\n",
+      "500/500 [==============================] - 15s 30ms/step - loss: 0.6538 - masked_acc: 0.8322 - masked_loss: 0.6543 - val_loss: 0.9582 - val_masked_acc: 0.7932 - val_masked_loss: 0.9586\n",
+      "Epoch 7/20\n",
+      "500/500 [==============================] - 15s 30ms/step - loss: 0.6596 - masked_acc: 0.8308 - masked_loss: 0.6605 - val_loss: 0.9581 - val_masked_acc: 0.7916 - val_masked_loss: 0.9594\n",
+      "Epoch 8/20\n",
+      "500/500 [==============================] - 15s 30ms/step - loss: 0.6746 - masked_acc: 0.8267 - masked_loss: 0.6754 - val_loss: 0.9448 - val_masked_acc: 0.7925 - val_masked_loss: 0.9467\n",
+      "Epoch 9/20\n",
+      "500/500 [==============================] - 16s 32ms/step - loss: 0.6785 - masked_acc: 0.8266 - masked_loss: 0.6788 - val_loss: 0.9292 - val_masked_acc: 0.7928 - val_masked_loss: 0.9295\n",
+      "Epoch 10/20\n",
+      "500/500 [==============================] - 15s 31ms/step - loss: 0.6287 - masked_acc: 0.8356 - masked_loss: 0.6292 - val_loss: 0.9324 - val_masked_acc: 0.7955 - val_masked_loss: 0.9324\n",
+      "Epoch 11/20\n",
+      "500/500 [==============================] - 15s 30ms/step - loss: 0.5875 - masked_acc: 0.8432 - masked_loss: 0.5880 - val_loss: 0.9407 - val_masked_acc: 0.7978 - val_masked_loss: 0.9401\n",
+      "Epoch 12/20\n",
+      "500/500 [==============================] - 15s 31ms/step - loss: 0.5988 - masked_acc: 0.8398 - masked_loss: 0.5992 - val_loss: 0.9546 - val_masked_acc: 0.7926 - val_masked_loss: 0.9546\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Train the translator (this takes some minutes so feel free to take a break)\n",
+    "\n",
+    "trained_translator, history = compile_and_train(translator)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d23b9301",
+   "metadata": {},
+   "source": [
+    "<a name=\"4\"></a>\n",
+    "## 4. Using the model for inference \n",
+    "\n",
+    "\n",
+    "Now that your model is trained you can use it for inference. To help you with this the `generate_next_token` function is provided. Notice that this function is meant to be used inside a for-loop, so you feed to it the information of the previous step to generate the information of the next step. In particular you need to keep track of the state of the pre-attention LSTM in the decoder and if you are done with the translation. Also notice that a `temperature` variable is introduced which determines how to select the next token given the predicted logits:  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 82,
+   "id": "522f6b6f",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def generate_next_token(decoder, context, next_token, done, state, temperature=0.0):\n",
+    "    \"\"\"Generates the next token in the sequence\n",
+    "\n",
+    "    Args:\n",
+    "        decoder (Decoder): The decoder\n",
+    "        context (tf.Tensor): Encoded sentence to translate\n",
+    "        next_token (tf.Tensor): The predicted next token\n",
+    "        done (bool): True if the translation is complete\n",
+    "        state (list[tf.Tensor, tf.Tensor]): Hidden states of the pre-attention LSTM layer\n",
+    "        temperature (float, optional): The temperature that controls the randomness of the predicted tokens. Defaults to 0.0.\n",
+    "\n",
+    "    Returns:\n",
+    "        tuple(tf.Tensor, np.float, list[tf.Tensor, tf.Tensor], bool): The next token, log prob of said token, hidden state of LSTM and if translation is done\n",
+    "    \"\"\"\n",
+    "    # Get the logits and state from the decoder\n",
+    "    logits, state = decoder(context, next_token, state=state, return_state=True)\n",
+    "    \n",
+    "    # Trim the intermediate dimension \n",
+    "    logits = logits[:, -1, :]\n",
+    "        \n",
+    "    # If temp is 0 then next_token is the argmax of logits\n",
+    "    if temperature == 0.0:\n",
+    "        next_token = tf.argmax(logits, axis=-1)\n",
+    "        \n",
+    "    # If temp is not 0 then next_token is sampled out of logits\n",
+    "    else:\n",
+    "        logits = logits / temperature\n",
+    "        next_token = tf.random.categorical(logits, num_samples=1)\n",
+    "    \n",
+    "    # Trim dimensions of size 1\n",
+    "    logits = tf.squeeze(logits)\n",
+    "    next_token = tf.squeeze(next_token)\n",
+    "    \n",
+    "    # Get the logit of the selected next_token\n",
+    "    logit = logits[next_token].numpy()\n",
+    "    \n",
+    "    # Reshape to (1,1) since this is the expected shape for text encoded as TF tensors\n",
+    "    next_token = tf.reshape(next_token, shape=(1,1))\n",
+    "    \n",
+    "    # If next_token is End-of-Sentence token you are done\n",
+    "    if next_token == eos_id:\n",
+    "        done = True\n",
+    "    \n",
+    "    return next_token, logit, state, done"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "190d2d76",
+   "metadata": {},
+   "source": [
+    "See how it works by running the following cell:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 83,
+   "id": "9937547a",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Next token: [[11484]]\n",
+      "Logit: -18.7833\n",
+      "Done? False\n"
+     ]
+    }
+   ],
+   "source": [
+    "# PROCESS SENTENCE TO TRANSLATE AND ENCODE\n",
+    "\n",
+    "# A sentence you wish to translate\n",
+    "eng_sentence = \"I love languages\"\n",
+    "\n",
+    "# Convert it to a tensor\n",
+    "texts = tf.convert_to_tensor(eng_sentence)[tf.newaxis]\n",
+    "\n",
+    "# Vectorize it and pass it through the encoder\n",
+    "context = english_vectorizer(texts).to_tensor()\n",
+    "context = encoder(context)\n",
+    "\n",
+    "# SET STATE OF THE DECODER\n",
+    "\n",
+    "# Next token is Start-of-Sentence since you are starting fresh\n",
+    "next_token = tf.fill((1,1), sos_id)\n",
+    "\n",
+    "# Hidden and Cell states of the LSTM can be mocked using uniform samples\n",
+    "state = [tf.random.uniform((1, UNITS)), tf.random.uniform((1, UNITS))]\n",
+    "\n",
+    "# You are not done until next token is EOS token\n",
+    "done = False\n",
+    "\n",
+    "# Generate next token\n",
+    "next_token, logit, state, done = generate_next_token(decoder, context, next_token, done, state, temperature=0.5)\n",
+    "print(f\"Next token: {next_token}\\nLogit: {logit:.4f}\\nDone? {done}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "170323dd",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex5\"></a>\n",
+    "## Exercise 5 - translate\n",
+    "\n",
+    "Now you can put everything together to translate a given sentence. For this, complete the `translate` function below. This function will take care of the following steps: \n",
+    "- Process the sentence to translate and encode it\n",
+    "\n",
+    "+ Set the initial state of the decoder\n",
+    "\n",
+    "- Get predictions of the next token (starting with the \\<SOS> token) for a maximum of iterations (in case the \\<EOS> token is never returned)\n",
+    "    \n",
+    "+ Return the translated text (as a string), the logit of the last iteration (this helps measure how certain was that the sequence was translated in its totality) and the translation in token format.\n",
+    "\n",
+    "\n",
+    "Hints: \n",
+    "\n",
+    "- The previous cell provides a lot of insights on how this function should work, so if you get stuck refer to it.\n",
+    "\n",
+    "+ Some useful docs:\n",
+    "    + [tf.newaxis](https://www.tensorflow.org/api_docs/python/tf#newaxis)\n",
+    "\n",
+    "    - [tf.fill](https://www.tensorflow.org/api_docs/python/tf/fill)\n",
+    "\n",
+    "    + [tf.zeros](https://www.tensorflow.org/api_docs/python/tf/zeros)\n",
+    "\n",
+    "\n",
+    "**IMPORTANT NOTE**: Due to randomness processes involving tensorflow training and weight initializing, the results below may vary a lot, even if you retrain your model in the same session. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 84,
+   "id": "42c74f1f",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED FUNCTION: translate\n",
+    "def translate(model, text, max_length=50, temperature=0.0):\n",
+    "    \"\"\"Translate a given sentence from English to Portuguese\n",
+    "\n",
+    "    Args:\n",
+    "        model (tf.keras.Model): The trained translator\n",
+    "        text (string): The sentence to translate\n",
+    "        max_length (int, optional): The maximum length of the translation. Defaults to 50.\n",
+    "        temperature (float, optional): The temperature that controls the randomness of the predicted tokens. Defaults to 0.0.\n",
+    "\n",
+    "    Returns:\n",
+    "        tuple(str, np.float, tf.Tensor): The translation, logit that predicted <EOS> token and the tokenized translation\n",
+    "    \"\"\"\n",
+    "    # Lists to save tokens and logits\n",
+    "    tokens, logits = [], []\n",
+    "\n",
+    "    ### START CODE HERE ###\n",
+    "    \n",
+    "    # PROCESS THE SENTENCE TO TRANSLATE\n",
+    "    \n",
+    "    # Convert the original string into a tensor\n",
+    "    text = tf.convert_to_tensor(text)[tf.newaxis]\n",
+    "    \n",
+    "    # Vectorize the text using the correct vectorizer\n",
+    "    context = english_vectorizer(text).to_tensor()\n",
+    "    \n",
+    "    # Get the encoded context (pass the context through the encoder)\n",
+    "    # Hint: Remember you can get the encoder by using model.encoder\n",
+    "    context = model.encoder(context)\n",
+    "    \n",
+    "    # INITIAL STATE OF THE DECODER\n",
+    "    \n",
+    "    # First token should be SOS token with shape (1,1)\n",
+    "    next_token = tf.fill((1,1), sos_id)\n",
+    "    \n",
+    "    # Initial hidden and cell states should be tensors of zeros with shape (1, UNITS)\n",
+    "    state = [tf.random.uniform((1, UNITS)), tf.random.uniform((1, UNITS))]\n",
+    "    \n",
+    "    # You are done when you draw a EOS token as next token (initial state is False)\n",
+    "    done = False\n",
+    "\n",
+    "    # Iterate for max_length iterations\n",
+    "    for _ in range(max_length):\n",
+    "        # Generate the next token\n",
+    "        try:\n",
+    "            next_token, logit, state, done = generate_next_token(\n",
+    "                decoder=model.decoder,\n",
+    "                context=context,\n",
+    "                next_token=next_token,\n",
+    "                done=done,\n",
+    "                state=state,\n",
+    "                temperature=temperature\n",
+    "            )\n",
+    "        except:\n",
+    "             raise Exception(\"Problem generating the next token\")\n",
+    "        \n",
+    "        # If done then break out of the loop\n",
+    "        if done:\n",
+    "            break\n",
+    "        \n",
+    "        # Add next_token to the list of tokens\n",
+    "        tokens.append(next_token)\n",
+    "        \n",
+    "        # Add logit to the list of logits\n",
+    "        logits.append(logit)\n",
+    "    \n",
+    "    ### END CODE HERE ###\n",
+    "    \n",
+    "    # Concatenate all tokens into a tensor\n",
+    "    tokens = tf.concat(tokens, axis=-1)\n",
+    "    \n",
+    "    # Convert the translated tokens into text\n",
+    "    translation = tf.squeeze(tokens_to_text(tokens, id_to_word))\n",
+    "    translation = translation.numpy().decode()\n",
+    "    \n",
+    "    return translation, logits[-1], tokens"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3525e8ba",
+   "metadata": {},
+   "source": [
+    "Try your function with temperature of 0, which will yield a deterministic output and is equivalent to a greedy decoding:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 85,
+   "id": "daaea8c5",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Temperature: 0.0\n",
+      "\n",
+      "Original sentence: I love languages\n",
+      "Translation: eu adoro idiomas .\n",
+      "Translation tokens:[[  9 564 850   4]]\n",
+      "Logit: -0.074\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Running this cell multiple times should return the same output since temp is 0\n",
+    "\n",
+    "temp = 0.0 \n",
+    "original_sentence = \"I love languages\"\n",
+    "\n",
+    "translation, logit, tokens = translate(trained_translator, original_sentence, temperature=temp)\n",
+    "\n",
+    "print(f\"Temperature: {temp}\\n\\nOriginal sentence: {original_sentence}\\nTranslation: {translation}\\nTranslation tokens:{tokens}\\nLogit: {logit:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7d05129b",
+   "metadata": {},
+   "source": [
+    "Try your function with temperature of 0.7 (stochastic output):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 86,
+   "id": "0e0697db",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Temperature: 0.7\n",
+      "\n",
+      "Original sentence: I love languages\n",
+      "Translation: eu adoro idiomas .\n",
+      "Translation tokens:[[  9 564 850   4]]\n",
+      "Logit: -0.093\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Running this cell multiple times should return different outputs since temp is not 0\n",
+    "# You can try different temperatures\n",
+    "\n",
+    "temp = 0.7\n",
+    "original_sentence = \"I love languages\"\n",
+    "\n",
+    "translation, logit, tokens = translate(trained_translator, original_sentence, temperature=temp)\n",
+    "\n",
+    "print(f\"Temperature: {temp}\\n\\nOriginal sentence: {original_sentence}\\nTranslation: {translation}\\nTranslation tokens:{tokens}\\nLogit: {logit:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 87,
+   "id": "a3a9ea35",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[91mFailed test case: translate didn't return the same translation when using temperature of 0.0.\n",
+      "Expected: o meu nome e [UNK] a [UNK] .\n",
+      "Got: , meu nome e [UNK] a [UNK] .\n",
+      "\n",
+      "\u001b[91mFailed test case: translate didn't return the same logit when using temperature of 0.0.\n",
+      "Expected: -0.5501561164855957\n",
+      "Got: -0.6304512619972229\n",
+      "\n",
+      "\u001b[91mFailed test case: translate didn't return the same tokens when using temperature of 0.0.\n",
+      "Expected: [[  7  43 175  13   1  12   1   4]]\n",
+      "Got: [[ 19  43 175  13   1  12   1   4]]\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "w1_unittest.test_translate(translate, trained_translator)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ba027524",
+   "metadata": {},
+   "source": [
+    "<a name=\"5\"></a>\n",
+    "## 5. Minimum Bayes-Risk Decoding\n",
+    "\n",
+    "As mentioned in the lectures, getting the most probable token at each step may not necessarily produce the best results. Another approach is to do Minimum Bayes Risk Decoding or MBR. The general steps to implement this are:\n",
+    "\n",
+    "- Take several random samples\n",
+    "+ Score each sample against all other samples\n",
+    "- Select the one with the highest score\n",
+    "\n",
+    "You will be building helper functions for these steps in the following sections.\n",
+    "\n",
+    "With the ability to generate different translations by setting different temperature values you can do what you saw in the lectures and generate a bunch of translations and then determine which one is the best candidate. You will now do this by using the provided `generate_samples` function. This function will return any desired number of candidate translations alongside the log-probability for each one:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 88,
+   "id": "62301cd5",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def generate_samples(model, text, n_samples=4, temperature=0.6):\n",
+    "    \n",
+    "    samples, log_probs = [], []\n",
+    "\n",
+    "    # Iterate for n_samples iterations\n",
+    "    for _ in range(n_samples):\n",
+    "        \n",
+    "        # Save the logit and the translated tensor\n",
+    "        _, logp, sample = translate(model, text, temperature=temperature)\n",
+    "        \n",
+    "        # Save the translated tensors\n",
+    "        samples.append(np.squeeze(sample.numpy()).tolist())\n",
+    "        \n",
+    "        # Save the logits\n",
+    "        log_probs.append(logp)\n",
+    "                \n",
+    "    return samples, log_probs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 89,
+   "id": "06bd792c",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Translated tensor: [9, 81, 850, 4] has logit: -0.080\n",
+      "Translated tensor: 4 has logit: -0.677\n",
+      "Translated tensor: [9, 98, 11, 850, 4] has logit: -0.063\n",
+      "Translated tensor: [9, 564, 850, 4] has logit: -0.110\n"
+     ]
+    }
+   ],
+   "source": [
+    "samples, log_probs = generate_samples(trained_translator, 'I love languages')\n",
+    "\n",
+    "for s, l in zip(samples, log_probs):\n",
+    "    print(f\"Translated tensor: {s} has logit: {l:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "29b10677",
+   "metadata": {},
+   "source": [
+    "## Comparing overlaps\n",
+    "\n",
+    "Now that you can generate multiple translations it is time to come up with a method to measure the goodness of each one. As you saw in the lectures, one way to achieve this is by comparing each sample against the others. \n",
+    "\n",
+    "There are several metrics you can use for this purpose, as shown in the lectures and you can try experimenting with any one of these. For this assignment, you will be calculating scores for **unigram overlaps**. \n",
+    "\n",
+    "One of these metrics is the widely used yet simple [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index) which gets the intersection over union of two sets. The `jaccard_similarity` function returns this metric for any pair of candidate and reference translations:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 90,
+   "id": "edb54a71",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def jaccard_similarity(candidate, reference):\n",
+    "        \n",
+    "    # Convert the lists to sets to get the unique tokens\n",
+    "    candidate_set = set(candidate)\n",
+    "    reference_set = set(reference)\n",
+    "    \n",
+    "    # Get the set of tokens common to both candidate and reference\n",
+    "    common_tokens = candidate_set.intersection(reference_set)\n",
+    "    \n",
+    "    # Get the set of all tokens found in either candidate or reference\n",
+    "    all_tokens = candidate_set.union(reference_set)\n",
+    "    \n",
+    "    # Compute the percentage of overlap (divide the number of common tokens by the number of all tokens)\n",
+    "    overlap = len(common_tokens) / len(all_tokens)\n",
+    "        \n",
+    "    return overlap"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 91,
+   "id": "fc3384bf",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "jaccard similarity between lists: [1, 2, 3] and [1, 2, 3, 4] is 0.750\n"
+     ]
+    }
+   ],
+   "source": [
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 3, 4]\n",
+    "\n",
+    "js = jaccard_similarity(l1, l2)\n",
+    "\n",
+    "print(f\"jaccard similarity between lists: {l1} and {l2} is {js:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a6997662",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "jaccard similarity between tensors: [1, 2, 3] and [1, 2, 3, 4] is 0.750\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b2510e3d",
+   "metadata": {},
+   "source": [
+    "<a name=\"ex6\"></a>\n",
+    "## Exercise 6 - rouge1_similarity\n",
+    "\n",
+    "Jaccard similarity is good but a more commonly used metric in machine translation is the ROUGE score. For unigrams, this is called ROUGE-1 and as shown in the lectures, you can output the scores for both precision and recall when comparing two samples. To get the final score, you will want to compute the F1-score as given by:\n",
+    "\n",
+    "$$score = 2* \\frac{(precision * recall)}{(precision + recall)}$$\n",
+    "\n",
+    "For the implementation of the `rouge1_similarity` function you want to use the [Counter](https://docs.python.org/3/library/collections.html#collections.Counter) class from the Python standard library:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 92,
+   "id": "fb2e0a00",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED FUNCTION: rouge1_similarity\n",
+    "def rouge1_similarity(candidate, reference):\n",
+    "    \"\"\"Computes the ROUGE 1 score between two token lists\n",
+    "\n",
+    "    Args:\n",
+    "        candidate (list[int]): Tokenized candidate translation\n",
+    "        reference (list[int]): Tokenized reference translation\n",
+    "\n",
+    "    Returns:\n",
+    "        float: Overlap between the two token lists\n",
+    "    \"\"\"\n",
+    "    ### START CODE HERE ###\n",
+    "    \n",
+    "    # Make a frequency table of the candidate and reference tokens\n",
+    "    # Hint: use the Counter class (already imported)\n",
+    "    candidate_word_counts = Counter(candidate)\n",
+    "    reference_word_counts = Counter(reference)\n",
+    "    \n",
+    "    # Initialize overlap at 0\n",
+    "    overlap = 0\n",
+    "    \n",
+    "    # Iterate over the tokens in the candidate frequency table\n",
+    "    # Hint: Counter is a subclass of dict and you can get the keys \n",
+    "    #       out of a dict using the keys method like this: dict.keys()\n",
+    "    for token in candidate_word_counts.keys():\n",
+    "        \n",
+    "        # Get the count of the current token in the candidate frequency table\n",
+    "        # Hint: You can access the counts of a token as you would access values of a dictionary\n",
+    "        token_count_candidate = candidate_word_counts[token]\n",
+    "        \n",
+    "        # Get the count of the current token in the reference frequency table\n",
+    "        # Hint: You can access the counts of a token as you would access values of a dictionary\n",
+    "        token_count_reference = reference_word_counts.get(token, 0)\n",
+    "        \n",
+    "        # Update the overlap by getting the minimum between the two token counts above\n",
+    "        overlap += min(token_count_candidate, token_count_reference)\n",
+    "    \n",
+    "    # Compute the precision\n",
+    "    # Hint: precision = overlap / (number of tokens in candidate list) \n",
+    "    precision = overlap / len(candidate) if len(candidate) > 0 else 0\n",
+    "    \n",
+    "    # Compute the recall\n",
+    "    # Hint: recall = overlap / (number of tokens in reference list) \n",
+    "    recall = overlap / len(reference) if len(reference) > 0 else 0\n",
+    "    \n",
+    "    if precision + recall != 0:\n",
+    "        # Compute the Rouge1 Score\n",
+    "        # Hint: This is equivalent to the F1 score\n",
+    "        f1_score = 2 * (precision * recall) / (precision + recall)\n",
+    "        \n",
+    "        return f1_score\n",
+    "    \n",
+    "    ### END CODE HERE ###\n",
+    "        \n",
+    "    return 0 # If precision + recall = 0 then return 0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 93,
+   "id": "14bb5295",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "rouge 1 similarity between lists: [1, 2, 3] and [1, 2, 3, 4] is 0.857\n"
+     ]
+    }
+   ],
+   "source": [
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 3, 4]\n",
+    "\n",
+    "r1s = rouge1_similarity(l1, l2)\n",
+    "\n",
+    "print(f\"rouge 1 similarity between lists: {l1} and {l2} is {r1s:.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "afb8c61a",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "rouge 1 similarity between lists: [1, 2, 3] and [1, 2, 3, 4] is 0.857\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 94,
+   "id": "a680132e",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[92m All tests passed!\n"
+     ]
+    }
+   ],
+   "source": [
+    "w1_unittest.test_rouge1_similarity(rouge1_similarity)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aaf8a058",
+   "metadata": {},
+   "source": [
+    "## Computing the Overall Score\n",
+    "\n",
+    "\n",
+    "You will now build a function to generate the overall score for a particular sample. As mentioned in the lectures, you need to compare each sample with all other samples. For instance, if we generated 30 sentences, we will need to compare sentence 1 to sentences 2 through 30. Then, we compare sentence 2 to sentences 1 and 3 through 30, and so forth. At each step, we get the average score of all comparisons to get the overall score for a particular sample. To illustrate, these will be the steps to generate the scores of a 4-sample list.\n",
+    "\n",
+    "- Get similarity score between sample 1 and sample 2\n",
+    "+ Get similarity score between sample 1 and sample 3\n",
+    "- Get similarity score between sample 1 and sample 4\n",
+    "+ Get average score of the first 3 steps. This will be the overall score of sample 1\n",
+    "- Iterate and repeat until samples 1 to 4 have overall scores.\n",
+    "\n",
+    "\n",
+    "The results will be stored in a dictionary for easy lookups.\n",
+    "\n",
+    "<a name=\"ex7\"></a>\n",
+    "## Exercise 7 - average_overlap\n",
+    "\n",
+    "Complete the `average_overlap` function below which should implement the process described above:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 95,
+   "id": "142264ff",
+   "metadata": {
+    "deletable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# GRADED FUNCTION: average_overlap\n",
+    "def average_overlap(samples, similarity_fn):\n",
+    "    \"\"\"Computes the arithmetic mean of each candidate sentence in the samples\n",
+    "\n",
+    "    Args:\n",
+    "        samples (list[list[int]]): Tokenized version of translated sentences\n",
+    "        similarity_fn (Function): Similarity function used to compute the overlap\n",
+    "\n",
+    "    Returns:\n",
+    "        dict[int, float]: A dictionary mapping the index of each translation to its score\n",
+    "    \"\"\"\n",
+    "    # Initialize dictionary\n",
+    "    scores = {}\n",
+    "    \n",
+    "    # Iterate through all samples (enumerate helps keep track of indexes)\n",
+    "    for index_candidate, candidate in enumerate(samples):    \n",
+    "        \n",
+    "        ### START CODE HERE ###\n",
+    "                \n",
+    "        # Initially overlap is zero\n",
+    "        overlap = 0.0\n",
+    "        \n",
+    "        # Iterate through all samples (enumerate helps keep track of indexes)\n",
+    "        for index_sample, sample in enumerate(samples):\n",
+    "\n",
+    "            # Skip if the candidate index is the same as the sample index\n",
+    "            if index_candidate == index_sample:\n",
+    "                continue\n",
+    "                \n",
+    "            # Get the overlap between candidate and sample using the similarity function\n",
+    "            sample_overlap = similarity_fn(candidate, sample)\n",
+    "            \n",
+    "            # Add the sample overlap to the total overlap\n",
+    "            overlap += sample_overlap\n",
+    "\n",
+    "        ### END CODE HERE ###\n",
+    "        \n",
+    "        # Get the score for the candidate by computing the average\n",
+    "        score = overlap / (len(samples) - 1)\n",
+    "\n",
+    "        # Only use 3 decimal points\n",
+    "        score = round(score, 3)\n",
+    "        \n",
+    "        # Save the score in the dictionary. use index as the key.\n",
+    "        scores[index_candidate] = score\n",
+    "        \n",
+    "    return scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 96,
+   "id": "f36cf403",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "average overlap between lists: [1, 2, 3], [1, 2, 4] and [1, 2, 4, 5] using Jaccard similarity is:\n",
+      "\n",
+      "{0: 0.45, 1: 0.625, 2: 0.575}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test with Jaccard similarity\n",
+    "\n",
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 4]\n",
+    "l3 = [1, 2, 4, 5]\n",
+    "\n",
+    "avg_ovlp = average_overlap([l1, l2, l3], jaccard_similarity)\n",
+    "\n",
+    "print(f\"average overlap between lists: {l1}, {l2} and {l3} using Jaccard similarity is:\\n\\n{avg_ovlp}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e277aed2-a5c9-4ed0-9ee2-614939f2df7b",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "average overlap between lists: [1, 2, 3], [1, 2, 4] and [1, 2, 4, 5] using Jaccard similarity is:\n",
+    "\n",
+    "{0: 0.45, 1: 0.625, 2: 0.575}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 97,
+   "id": "d961a304-7c03-4ecb-ba5f-c8747ed3ec39",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "average overlap between lists: [1, 2, 3], [1, 4], [1, 2, 4, 5] and [5, 6] using Rouge1 similarity is:\n",
+      "\n",
+      "{0: 0.324, 1: 0.356, 2: 0.524, 3: 0.111}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test with Rouge1 similarity\n",
+    "\n",
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 4]\n",
+    "l3 = [1, 2, 4, 5]\n",
+    "l4 = [5,6]\n",
+    "\n",
+    "avg_ovlp = average_overlap([l1, l2, l3, l4], rouge1_similarity)\n",
+    "\n",
+    "print(f\"average overlap between lists: {l1}, {l2}, {l3} and {l4} using Rouge1 similarity is:\\n\\n{avg_ovlp}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30adc749-ffcb-4e82-a8f0-c04a7e39da0a",
+   "metadata": {},
+   "source": [
+    "##### __Expected Output__\n",
+    "\n",
+    "```\n",
+    "average overlap between lists: [1, 2, 3], [1, 4], [1, 2, 4, 5] and [5, 6] using Rouge1 similarity is:\n",
+    "\n",
+    "{0: 0.324, 1: 0.356, 2: 0.524, 3: 0.111}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 98,
+   "id": "c41b1fba-fd0f-41e6-9b07-746f64030fe3",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[92m All tests passed!\n"
+     ]
+    }
+   ],
+   "source": [
+    "w1_unittest.test_average_overlap(average_overlap)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4482249",
+   "metadata": {},
+   "source": [
+    "In practice, it is also common to see the weighted mean being used to calculate the overall score instead of just the arithmetic mean. This is implemented in the `weighted_avg_overlap` function below and you can use it in your experiments to see which one will give better results:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 99,
+   "id": "398714be",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def weighted_avg_overlap(samples, log_probs, similarity_fn):\n",
+    "    \n",
+    "    # Scores dictionary\n",
+    "    scores = {}\n",
+    "    \n",
+    "    # Iterate over the samples\n",
+    "    for index_candidate, candidate in enumerate(samples):    \n",
+    "        \n",
+    "        # Initialize overlap and weighted sum\n",
+    "        overlap, weight_sum = 0.0, 0.0\n",
+    "        \n",
+    "        # Iterate over all samples and log probabilities\n",
+    "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):\n",
+    "\n",
+    "            # Skip if the candidate index is the same as the sample index            \n",
+    "            if index_candidate == index_sample:\n",
+    "                continue\n",
+    "                \n",
+    "            # Convert log probability to linear scale\n",
+    "            sample_p = float(np.exp(logp))\n",
+    "\n",
+    "            # Update the weighted sum\n",
+    "            weight_sum += sample_p\n",
+    "\n",
+    "            # Get the unigram overlap between candidate and sample\n",
+    "            sample_overlap = similarity_fn(candidate, sample)\n",
+    "            \n",
+    "            # Update the overlap\n",
+    "            overlap += sample_p * sample_overlap\n",
+    "            \n",
+    "        # Compute the score for the candidate\n",
+    "        score = overlap / weight_sum\n",
+    "\n",
+    "        # Only use 3 decimal points\n",
+    "        score = round(score, 3)\n",
+    "        \n",
+    "        # Save the score in the dictionary. use index as the key.\n",
+    "        scores[index_candidate] = score\n",
+    "    \n",
+    "    return scores"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 100,
+   "id": "e3dfd6d3",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "weighted average overlap using Jaccard similarity is:\n",
+      "\n",
+      "{0: 0.443, 1: 0.631, 2: 0.558}\n"
+     ]
+    }
+   ],
+   "source": [
+    "l1 = [1, 2, 3]\n",
+    "l2 = [1, 2, 4]\n",
+    "l3 = [1, 2, 4, 5]\n",
+    "log_probs = [0.4, 0.2, 0.5]\n",
+    "\n",
+    "w_avg_ovlp = weighted_avg_overlap([l1, l2, l3], log_probs, jaccard_similarity)\n",
+    "\n",
+    "print(f\"weighted average overlap using Jaccard similarity is:\\n\\n{w_avg_ovlp}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cdb0b4db",
+   "metadata": {},
+   "source": [
+    "## mbr_decode\n",
+    "\n",
+    "You will now put everything together in the the `mbr_decode` function below. This final step is not graded as this function is just a wrapper around all the cool stuff you have coded so far! \n",
+    "\n",
+    "You can use it to play around, trying different numbers of samples, temperatures and similarity functions!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 101,
+   "id": "6fcfa640",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": [
+     "graded"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "def mbr_decode(model, text, n_samples=5, temperature=0.6, similarity_fn=jaccard_similarity):\n",
+    "    \n",
+    "    # Generate samples\n",
+    "    samples, log_probs = generate_samples(model, text, n_samples=n_samples, temperature=temperature)\n",
+    "    \n",
+    "    # Compute the overlap scores\n",
+    "    scores = weighted_avg_overlap(samples, log_probs, similarity_fn)\n",
+    "\n",
+    "    # Decode samples\n",
+    "    decoded_translations = [tokens_to_text(s, id_to_word).numpy().decode('utf-8') for s in samples]\n",
+    "    \n",
+    "    # Find the key with the highest score\n",
+    "    max_score_key = max(scores, key=lambda k: scores[k])\n",
+    "    \n",
+    "    # Get the translation \n",
+    "    translation = decoded_translations[max_score_key]\n",
+    "    \n",
+    "    return translation, decoded_translations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 102,
+   "id": "99507fcc-7727-45e7-933b-d3a08034f731",
+   "metadata": {
+    "deletable": false,
+    "editable": false,
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Translation candidates:\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu sinto idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "eu adoro idiomas .\n",
+      "\n",
+      "Selected translation: eu adoro idiomas .\n"
+     ]
+    }
+   ],
+   "source": [
+    "english_sentence = \"I love languages\"\n",
+    "\n",
+    "translation, candidates = mbr_decode(trained_translator, english_sentence, n_samples=10, temperature=0.6)\n",
+    "\n",
+    "print(\"Translation candidates:\")\n",
+    "for c in candidates:\n",
+    "    print(c)\n",
+    "\n",
+    "print(f\"\\nSelected translation: {translation}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "801b193f-4ea6-4ca1-ae29-a506cce656d9",
+   "metadata": {},
+   "source": [
+    "**Congratulations!** Next week, you'll dive deeper into attention models and study the Transformer architecture. You will build another network but without the recurrent part. It will show that attention is all you need! It should be fun!\n",
+    "\n",
+    "**Keep up the good work!**"
+   ]
+  }
+ ],
+ "metadata": {
+  "grader_version": "1",
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

NLP with Attention Models/NMT_with_Attention/NMT with MBR/Files/tf/por-eng/por.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5307326410edc8f65c3d0213cf3e5544e41c400efed7fe9ba557c8847a6ee803
+size 27856622

NLP with Attention Models/NMT_with_Attention/NMT with MBR/Files/tf/utils.py

@@ -0,0 +1,114 @@
+import numpy as np
+import tensorflow as tf
+import tensorflow_text as tf_text
+import pathlib
+
+path_to_file = pathlib.Path("por-eng/por.txt")
+
+np.random.seed(1234)
+tf.random.set_seed(1234)
+
+def load_data(path):
+    text = path.read_text(encoding="utf-8")
+
+    lines = text.splitlines()
+    pairs = [line.split("\t") for line in lines]
+
+    context = np.array([context for target, context, _ in pairs])
+    target = np.array([target for target, context, _ in pairs])
+
+    return context, target
+
+
+portuguese_sentences, english_sentences = load_data(path_to_file)
+
+sentences = (portuguese_sentences, english_sentences)
+
+BUFFER_SIZE = len(english_sentences)
+BATCH_SIZE = 64
+
+is_train = np.random.uniform(size=(len(portuguese_sentences),)) < 0.8
+
+train_raw = (
+    tf.data.Dataset.from_tensor_slices(
+        (english_sentences[is_train], portuguese_sentences[is_train])
+    )
+    .shuffle(BUFFER_SIZE)
+    .batch(BATCH_SIZE)
+)
+val_raw = (
+    tf.data.Dataset.from_tensor_slices(
+        (english_sentences[~is_train], portuguese_sentences[~is_train])
+    )
+    .shuffle(BUFFER_SIZE)
+    .batch(BATCH_SIZE)
+)
+
+
+def tf_lower_and_split_punct(text):
+    text = tf_text.normalize_utf8(text, "NFKD")
+    text = tf.strings.lower(text)
+    text = tf.strings.regex_replace(text, "[^ a-z.?!,¿]", "")
+    text = tf.strings.regex_replace(text, "[.?!,¿]", r" \0 ")
+    text = tf.strings.strip(text)
+    text = tf.strings.join(["[SOS]", text, "[EOS]"], separator=" ")
+    return text
+
+
+max_vocab_size = 12000
+
+english_vectorizer = tf.keras.layers.TextVectorization(
+    standardize=tf_lower_and_split_punct, max_tokens=max_vocab_size, ragged=True
+)
+
+english_vectorizer.adapt(train_raw.map(lambda context, target: context))
+
+portuguese_vectorizer = tf.keras.layers.TextVectorization(
+    standardize=tf_lower_and_split_punct, max_tokens=max_vocab_size, ragged=True
+)
+
+portuguese_vectorizer.adapt(train_raw.map(lambda context, target: target))
+
+
+def process_text(context, target):
+    context = english_vectorizer(context).to_tensor()
+    target = portuguese_vectorizer(target)
+    targ_in = target[:, :-1].to_tensor()
+    targ_out = target[:, 1:].to_tensor()
+    return (context, targ_in), targ_out
+
+
+train_data = train_raw.map(process_text, tf.data.AUTOTUNE)
+val_data = val_raw.map(process_text, tf.data.AUTOTUNE)
+
+del train_raw
+del val_raw
+
+
+def masked_loss(y_true, y_pred):
+    
+    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction='none')
+    loss = loss_fn(y_true, y_pred)
+    
+    # Check which elements of y_true are padding
+    mask = tf.cast(y_true != 0, loss.dtype)
+    
+    loss *= mask
+    # Return the total.
+    return tf.reduce_sum(loss)/tf.reduce_sum(mask)
+
+
+def masked_acc(y_true, y_pred):
+    y_pred = tf.argmax(y_pred, axis=-1)
+    y_pred = tf.cast(y_pred, y_true.dtype)
+    match = tf.cast(y_true == y_pred, tf.float32)
+    mask = tf.cast(y_true != 0, tf.float32)
+    match*= mask
+
+    return tf.reduce_sum(match)/tf.reduce_sum(mask)
+
+
+def tokens_to_text(tokens, id_to_word):
+    words = id_to_word(tokens)
+    result = tf.strings.reduce_join(words, axis=-1, separator=" ")
+    return result

NLP with Attention Models/NMT_with_Attention/NMT with MBR/Files/tf/w1_unittest.py

@@ -0,0 +1,702 @@
+import math
+from itertools import combinations
+import tensorflow as tf
+import numpy as np
+from dlai_grader.grading import test_case, print_feedback
+from utils import train_data
+
+VOCAB_SIZE = 12000
+UNITS = 256
+
+
+def test_encoder(encoder_to_test):
+    def g():
+        vocab_sizes = [5, 20, 1000, 15000]
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        encoder = encoder_to_test(vocab_sizes[0], units[0])
+
+        t = test_case()
+        if encoder.embedding.mask_zero != True:
+            t.failed = True
+            t.msg = "Embedding layer has incorrect value for 'mask_zero' attribute"
+            t.want = True
+            t.got = encoder.embedding.mask_zero
+        cases.append(t)
+
+        for vs, u in zip(vocab_sizes, units):
+            encoder = encoder_to_test(vs, u)
+
+            t = test_case()
+            if encoder.embedding.input_dim != vs:
+                t.failed = True
+                t.msg = "Incorrect input dim of embedding layer"
+                t.want = vs
+                t.got = encoder.embedding.input_dim
+            cases.append(t)
+
+            t = test_case()
+            if encoder.embedding.output_dim != u:
+                t.failed = True
+                t.msg = "Incorrect output dim of embedding layer"
+                t.want = u
+                t.got = encoder.embedding.output_dim
+            cases.append(t)
+
+        t = test_case()
+        if not isinstance(encoder.rnn.layer, tf.keras.layers.LSTM):
+            t.failed = True
+            t.msg = "Incorrect type of layer inside Bidirectional"
+            t.want = tf.keras.layers.LSTM
+            t.got = type(encoder.rnn.layer)
+            return [t]
+
+        for u in units:
+            encoder = encoder_to_test(vocab_sizes[1], u)
+            t = test_case()
+            if encoder.rnn.layer.units != u:
+                t.failed = True
+                t.msg = "Incorrect number of units in LSTM layer"
+                t.want = u
+                t.got = encoder.rnn.layer.units
+            cases.append(t)
+
+        t = test_case()
+        if encoder.rnn.layer.return_sequences != True:
+            t.failed = True
+            t.msg = "LSTM layer has incorrect value for 'return_sequences' attribute"
+            t.want = True
+            t.got = encoder.rnn.layer.return_sequences
+        cases.append(t)
+
+        vocab_size = 16
+        n_units = 8
+        encoder = encoder_to_test(vocab_size, n_units)
+        to_translate = np.array([[1, 2, 3, 4, 5, 6, 14, 0, 0, 0],
+                               [2, 1, 1, 1, 1, 1, 8, 0, 0, 0],
+                               [5, 4, 2, 3, 3, 15, 11, 0, 0, 0]])
+        #for (to_translate, _), _ in train_data.take(3):
+            
+        first_dim_in, second_dim_in = to_translate.shape
+        encoder_output = encoder(to_translate)
+        t = test_case()
+        if len(encoder_output.shape) != 3:
+            t.failed = True
+            t.msg = "Incorrect shape of encoder output"
+            t.want = "a shape with 3 dimensions"
+            t.got = encoder_output.shape
+            return [t]
+
+        first_dim_out, second_dim_out, third_dim_out = encoder_output.shape
+
+        t = test_case()
+        if first_dim_in != first_dim_out:
+            t.failed = True
+            t.msg = "Incorrect first dimension of encoder output"
+            t.want = first_dim_in
+            t.got = first_dim_out
+        cases.append(t)
+
+        t = test_case()
+        if second_dim_in != second_dim_out:
+            t.failed = True
+            t.msg = "Incorrect second dimension of encoder output"
+            t.want = second_dim_in
+            t.got = second_dim_out
+        cases.append(t)
+
+        t = test_case()
+        if third_dim_out != n_units:
+            t.failed = True
+            t.msg = "Incorrect third dimension of encoder output"
+            t.want = units
+            t.got = third_dim_out
+        cases.append(t)
+
+        return cases
+
+    cases = g()
+    print_feedback(cases)
+
+
+def test_cross_attention(cross_attention_to_test):
+    def g():
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        n_units = 512
+        cross_attention = cross_attention_to_test(n_units)
+
+        t = test_case()
+        if not isinstance(cross_attention.mha, tf.keras.layers.MultiHeadAttention):
+            t.failed = True
+            t.msg = "Incorrect type of layer for Multi Head Attention"
+            t.want = tf.keras.layers.MultiHeadAttention
+            t.got = type(cross_attention.mha)
+            return [t]
+
+        #         for u in units:
+        #             cross_attention = cross_attention_to_test(u)
+
+        #             t = test_case()
+        #             if cross_attention.mha.key_dim != u:
+        #                 t.failed = True
+        #                 t.msg = "Incorrect key dim of Multi Head Attention layer"
+        #                 t.want = u
+        #                 t.got = cross_attention.mha.key_dim
+        #             cases.append(t)
+
+        cross_attention = cross_attention_to_test(n_units)
+        embed = tf.keras.layers.Embedding(VOCAB_SIZE, output_dim=UNITS, mask_zero=True)
+
+        for (to_translate, sr_translation), _ in train_data.take(3):
+            sr_translation_embed = embed(sr_translation)
+            first_dim_in, second_dim_in, third_dim_in = sr_translation_embed.shape
+            dummy_encoder_output = np.random.rand(64, 14, 512)
+            cross_attention_output = cross_attention(
+                dummy_encoder_output, sr_translation_embed
+            )
+            #             print(cross_attention_output.shape)
+
+            t = test_case()
+            if len(cross_attention_output.shape) != 3:
+                t.failed = True
+                t.msg = "Incorrect shape of cross_attention output"
+                t.want = "a shape with 3 dimensions"
+                t.got = cross_attention_output.shape
+                return [t]
+
+            first_dim_out, second_dim_out, third_dim_out = cross_attention_output.shape
+
+            t = test_case()
+            if first_dim_in != first_dim_out:
+                t.failed = True
+                t.msg = "Incorrect first dimension of cross_attention output"
+                t.want = first_dim_in
+                t.got = first_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if second_dim_in != second_dim_out:
+                t.failed = True
+                t.msg = "Incorrect second dimension of cross_attention output"
+                t.want = second_dim_in
+                t.got = second_dim_out
+            cases.append(t)
+
+            t = test_case()
+            if third_dim_in != third_dim_out:
+                t.failed = True
+                t.msg = "Incorrect third dimension of cross_attention output"
+                t.want = third_dim_in
+                t.got = third_dim_out
+            cases.append(t)
+
+        _, n_heads, key_dim = cross_attention.mha.get_weights()[0].shape
+
+        t = test_case()
+        if n_heads != 1:
+            t.failed = True
+            t.msg = "Incorrect number of attention heads"
+            t.want = 1
+            t.got = n_heads
+        cases.append(t)
+
+        t = test_case()
+        if key_dim != n_units:
+            t.failed = True
+            t.msg = f"Incorrect size of query and key for every attention head when passing {n_units} units to the constructor"
+            t.want = n_units
+            t.got = key_dim
+        cases.append(t)
+
+        return cases
+
+    cases = g()
+    print_feedback(cases)
+
+
+def test_decoder(decoder_to_test, CrossAttention):
+    def g():
+        vocab_sizes = [5, 20, 1000, 15000]
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        vocab_size = 10000
+        n_units = 512
+        decoder = decoder_to_test(vocab_size, n_units)
+
+        t = test_case()
+        if not isinstance(decoder.embedding, tf.keras.layers.Embedding):
+            t.failed = True
+            t.msg = "Incorrect type of embedding layer"
+            t.want = tf.keras.layers.Embedding
+            t.got = type(decoder.embedding)
+            return [t]
+
+        t = test_case()
+        if decoder.embedding.mask_zero != True:
+            t.failed = True
+            t.msg = "Embedding layer has incorrect value for 'mask_zero' attribute"
+            t.want = True
+            t.got = decoder.embedding.mask_zero
+        cases.append(t)
+
+        for vs, u in zip(vocab_sizes, units):
+            decoder = decoder_to_test(vs, u)
+
+            t = test_case()
+            if decoder.embedding.input_dim != vs:
+                t.failed = True
+                t.msg = "Incorrect input dim of embedding layer"
+                t.want = vs
+                t.got = decoder.embedding.input_dim
+            cases.append(t)
+
+            t = test_case()
+            if decoder.embedding.output_dim != u:
+                t.failed = True
+                t.msg = "Incorrect output dim of embedding layer"
+                t.want = u
+                t.got = decoder.embedding.output_dim
+            cases.append(t)
+
+        t = test_case()
+        if not isinstance(decoder.pre_attention_rnn, tf.keras.layers.LSTM):
+            t.failed = True
+            t.msg = "Incorrect type of pre_attention_rnn layer"
+            t.want = tf.keras.layers.LSTM
+            t.got = type(decoder.pre_attention_rnn)
+            return [t]
+
+        for u in units:
+            decoder = decoder_to_test(vocab_size, u)
+            t = test_case()
+            if decoder.pre_attention_rnn.units != u:
+                t.failed = True
+                t.msg = "Incorrect number of units in pre_attention_rnn layer"
+                t.want = u
+                t.got = decoder.pre_attention_rnn.units
+            cases.append(t)
+
+            #             t = test_case()
+            #             if decoder.attention.units != u:
+            #                 t.failed = True
+            #                 t.msg = "Incorrect number of units in attention layer"
+            #                 t.want = u
+            #                 t.got = decoder.attention.units
+            #             cases.append(t)
+
+            t = test_case()
+            if decoder.post_attention_rnn.units != u:
+                t.failed = True
+                t.msg = "Incorrect number of units in post_attention_rnn layer"
+                t.want = u
+                t.got = decoder.post_attention_rnn.units
+            cases.append(t)
+
+        t = test_case()
+        if decoder.pre_attention_rnn.return_sequences != True:
+            t.failed = True
+            t.msg = "pre_attention_rnn layer has incorrect value for 'return_sequences' attribute"
+            t.want = True
+            t.got = decoder.pre_attention_rnn.return_sequences
+        cases.append(t)
+
+        t = test_case()
+        if decoder.pre_attention_rnn.return_state != True:
+            t.failed = True
+            t.msg = "pre_attention_rnn layer has incorrect value for 'return_state' attribute"
+            t.want = True
+            t.got = decoder.pre_attention_rnn.return_state
+        cases.append(t)
+
+        t = test_case()
+        if not isinstance(decoder.attention, CrossAttention):
+            t.failed = True
+            t.msg = "Incorrect type of attention layer"
+            t.want = CrossAttention
+            t.got = type(decoder.attention)
+            return [t]
+
+        t = test_case()
+        if decoder.post_attention_rnn.return_sequences != True:
+            t.failed = True
+            t.msg = "post_attention_rnn layer has incorrect value for 'return_sequences' attribute"
+            t.want = True
+            t.got = decoder.post_attention_rnn.return_sequences
+        cases.append(t)
+
+        t = test_case()
+        if not isinstance(decoder.post_attention_rnn, tf.keras.layers.LSTM):
+            t.failed = True
+            t.msg = "Incorrect type of pre_attention_rnn layer"
+            t.want = tf.keras.layers.LSTM
+            t.got = type(decoder.post_attention_rnn)
+            return [t]
+
+        t = test_case()
+        if not isinstance(decoder.output_layer, tf.keras.layers.Dense):
+            t.failed = True
+            t.msg = "Incorrect type of output_layer layer"
+            t.want = tf.keras.layers.Dense
+            t.got = type(decoder.output_layer)
+            return [t]
+
+        t = test_case()
+        if (
+            "log" not in decoder.output_layer.activation.__name__
+            or "softmax" not in decoder.output_layer.activation.__name__
+        ):
+            t.failed = True
+            t.msg = "output_layer layer has incorrect activation function"
+            t.want = "a log softmax activation function such as 'log_softmax_v2'"
+            t.got = decoder.output_layer.activation.__name__
+        cases.append(t)
+
+        vocab_size = 6
+        n_units = 4
+        decoder = decoder_to_test(vocab_size, n_units)
+        sr_translation = np.array([[3, 4, 5, 3, 3, 3, 5, 1, 1, 1, 1, 1], 
+                                    [1, 2, 3, 4, 5, 1, 1, 0, 0, 0, 0, 0]])
+        encoder_output = np.random.rand(2, 10, n_units)
+        decoder_output = decoder(encoder_output, sr_translation)
+
+        first_dim_in, second_dim_in = sr_translation.shape
+
+        t = test_case()
+        if len(decoder_output.shape) != 3:
+            t.failed = True
+            t.msg = "Incorrect shape of decoder output"
+            t.want = "a shape with 3 dimensions"
+            t.got = decoder_output.shape
+            return [t]
+
+        first_dim_out, second_dim_out, third_dim_out = decoder_output.shape
+
+        t = test_case()
+        if first_dim_in != first_dim_out:
+            t.failed = True
+            t.msg = "Incorrect first dimension of decoder output"
+            t.want = first_dim_in
+            t.got = first_dim_out
+        cases.append(t)
+
+        t = test_case()
+        if second_dim_in != second_dim_out:
+            t.failed = True
+            t.msg = "Incorrect second dimension of decoder output"
+            t.want = second_dim_in
+            t.got = second_dim_out
+        cases.append(t)
+
+        t = test_case()
+        if third_dim_out != vocab_size:
+            t.failed = True
+            t.msg = "Incorrect third dimension of decoder output"
+            t.want = vocab_size
+            t.got = third_dim_out
+        cases.append(t)
+        
+        return cases
+
+    cases = g()
+    print_feedback(cases)
+
+
+def test_translator(translator_to_test, Encoder, Decoder):
+    def g():
+        vocab_sizes = [5, 20, 1000, 15000]
+        units = [32, 64, 256, 512]
+
+        cases = []
+
+        vocab_size = 10000
+        n_units = 512
+        translator = translator_to_test(vocab_size, n_units)
+
+        t = test_case()
+        if not isinstance(translator.encoder, Encoder):
+            t.failed = True
+            t.msg = "Incorrect type of encoder layer"
+            t.want = Encoder
+            t.got = type(translator.encoder)
+            return [t]
+
+        t = test_case()
+        if not isinstance(translator.decoder, Decoder):
+            t.failed = True
+            t.msg = "Incorrect type of encoder layer"
+            t.want = Decoder
+            t.got = type(translator.decoder)
+            return [t]
+
+        vocab_size = 16
+        n_units = 8
+        translator = translator_to_test(vocab_size, n_units)
+
+        to_translate = np. array([[1, 2, 3, 4, 5, 0, 0],
+                                 [5, 2, 3, 4, 5, 6, 0],
+                                 [6, 3, 3, 4, 5, 3, 3],
+                                 [7, 9, 9, 6, 5, 3, 3]])
+
+        sr_translation = np. array([[8, 1, 2, 3, 4, 5, 0, 0],
+                                 [9, 5, 2, 3, 4, 5, 6, 0],
+                                 [10, 6, 3, 3, 4, 5, 3, 3],
+                                 [11, 7, 9, 9, 6, 5, 3, 3]])
+
+        #for (to_translate, sr_translation), _ in train_data.take(3):
+        first_dim_in, second_dim_in = sr_translation.shape
+        translator_output = translator((to_translate, sr_translation))
+        t = test_case()
+        if len(translator_output.shape) != 3:
+            t.failed = True
+            t.msg = "Incorrect shape of translator output"
+            t.want = "a shape with 3 dimensions"
+            t.got = translator_output.shape
+            return [t]
+
+        first_dim_out, second_dim_out, third_dim_out = translator_output.shape
+
+        t = test_case()
+        if first_dim_in != first_dim_out:
+            t.failed = True
+            t.msg = "Incorrect first dimension of translator output"
+            t.want = first_dim_in
+            t.got = first_dim_out
+        cases.append(t)
+
+        t = test_case()
+        if second_dim_in != second_dim_out:
+            t.failed = True
+            t.msg = "Incorrect second dimension of translator output"
+            t.want = second_dim_in
+            t.got = second_dim_out
+        cases.append(t)
+
+        t = test_case()
+        if third_dim_out != vocab_size:
+            t.failed = True
+            t.msg = "Incorrect third dimension of translator output"
+            t.want = vocab_size
+            t.got = third_dim_out
+        cases.append(t)
+
+        return cases
+
+    cases = g()
+    print_feedback(cases)
+
+    
+
+def test_translate(learner_func, model):
+    def g():
+        
+        cases = []
+        
+        txt = "Hi, my name is Younes"
+        try:
+            translation, logit, tokens = learner_func(model, txt, temperature=0.9)
+        except Exception as e:
+            t = test_case()
+            t.failed = True
+            t.msg = "There was an exception when running your function"
+            t.want = "No exceptions"
+            t.got = f"{str(e)}"
+            return [t]
+        
+        txt = "Hi, my name is Alejandra"
+        translation, logit, tokens = learner_func(model, txt, temperature=0.0)
+        
+        t = test_case()
+        
+        if not isinstance(translation, str):
+            t.failed = True
+            t.msg = "'translation' has incorrect type"
+            t.want = str
+            t.got = type(translation)
+        cases.append(t)
+        
+        if not isinstance(logit, np.number):
+            t.failed = True
+            t.msg = "'logit' has incorrect type"
+            t.want = np.number
+            t.got = type(logit)
+        cases.append(t)
+        
+        if not isinstance(tokens, tf.Tensor):
+            t.failed = True
+            t.msg = "'tokens' has incorrect type"
+            t.want = tf.Tensor
+            t.got = type(tokens)
+        cases.append(t)
+        
+        translation2, logit2, tokens2 = learner_func(model, txt, temperature=0.0)
+        
+        t = test_case()
+        if translation != translation2:
+            t.failed = True
+            t.msg = "translate didn't return the same translation when using temperature of 0.0"
+            t.want = translation
+            t.got = translation2
+        cases.append(t)
+        
+        t = test_case()
+        if logit != logit2:
+            t.failed = True
+            t.msg = "translate didn't return the same logit when using temperature of 0.0"
+            t.want = logit
+            t.got = logit2
+        cases.append(t)
+        
+        t = test_case()
+        if not np.allclose(tokens, tokens2):
+            t.failed = True
+            t.msg = "translate didn't return the same tokens when using temperature of 0.0"
+            t.want = tokens
+            t.got = tokens2
+        cases.append(t)
+
+        # Check that function uses the model.decoder and model.enconder functions
+        inputs = tf.keras.Input(shape=(37,))
+        outputs = tf.keras.layers.Dense(5, activation="softmax")(inputs)
+        model_fake = tf.keras.Model(inputs = inputs, outputs = outputs)
+        
+        model_fake.encoder = model.encoder
+        model_fake.decoder = None
+        t = test_case()
+        try:
+            ff = learner_func(model_fake, "Hello world", temperature=0.0)
+            t.failed = True
+            t.msg = "The translator is not using the internal model.decoder. You are probably using a global variable"
+            t.want = "Fail translation"
+            t.got = "Succeed translation with wrong decoder"
+        except:
+            None
+            
+        cases.append(t)
+        
+        model_fake.encoder = None
+        model_fake.decoder = model.decoder
+        t = test_case()
+        try:
+            ff = learner_func(model_fake, "Hello world", temperature=0.0)
+            t.failed = True
+            t.msg = "The translator is not using the internal model.encoder. You are probably using a global variable"
+            t.want = "Fail translation"
+            t.got = "Succeed translation with wrong encoder"
+        except:
+            None
+
+        cases.append(t)
+        
+        return cases
+    
+    cases = g()
+    print_feedback(cases)
+
+    
+
+
+def test_rouge1_similarity(learner_func):
+    
+    def g():
+        
+        tensors = [
+            [0],
+            [0, 1],
+            [0, 1, 2],
+            [1, 2, 4, 5],
+            [5, 5, 7, 0, 232]
+        ]
+        
+        expected = [0.6666666666666666, 0.5, 0, 0.33333333333333337, 0.8, 0.3333333333333333, 0.28571428571428575, 0.5714285714285715, 0.25]
+
+        cases = []
+        pairs = list(combinations(tensors, 2))
+        
+        for (candidate, reference), solution in zip(pairs, expected):
+            answer = learner_func(candidate, reference)
+            t = test_case()
+            if not math.isclose(answer, solution):
+                t.failed = True
+                t.msg = f"Incorrect similarity for candidate={candidate} and reference={reference}"
+                t.want = solution
+                t.got = answer
+            cases.append(t)
+
+        return cases
+    
+    cases = g()
+    print_feedback(cases)
+
+
+def test_average_overlap(learner_func):
+
+    def jaccard_similarity(candidate, reference):
+        
+        # Convert the lists to sets to get the unique tokens
+        candidate_set = set(candidate)
+        reference_set = set(reference)
+        
+        # Get the set of tokens common to both candidate and reference
+        common_tokens = candidate_set.intersection(reference_set)
+        
+        # Get the set of all tokens found in either candidate or reference
+        all_tokens = candidate_set.union(reference_set)
+        
+        # Compute the percentage of overlap (divide the number of common tokens by the number of all tokens)
+        overlap = len(common_tokens) / len(all_tokens)
+            
+        return overlap
+    
+    def g():
+        
+        l1 = [1, 2, 3]
+        l2 = [1, 2, 4]
+        l3 = [1, 2, 4, 5]
+        l4 = [5,6]
+
+        elements = [l1, l2, l3, l4]
+
+        all_combinations = []
+
+        for r in range(2, len(elements) + 1):
+            # Generate combinations of length r
+            combinations_r = combinations(elements, r)
+            
+            # Append the combinations to the result list
+            all_combinations.extend(combinations_r)
+        
+        expected = [{0: 0.5, 1: 0.5},
+                     {0: 0.4, 1: 0.4},
+                     {0: 0.0, 1: 0.0},
+                     {0: 0.75, 1: 0.75},
+                     {0: 0.0, 1: 0.0},
+                     {0: 0.2, 1: 0.2},
+                     {0: 0.45, 1: 0.625, 2: 0.575},
+                     {0: 0.25, 1: 0.25, 2: 0.0},
+                     {0: 0.2, 1: 0.3, 2: 0.1},
+                     {0: 0.375, 1: 0.475, 2: 0.1},
+                     {0: 0.3, 1: 0.417, 2: 0.45, 3: 0.067}]
+
+        cases = []
+        
+        for combination, solution in zip(all_combinations, expected):
+            answer = learner_func(combination, jaccard_similarity)
+            t = test_case()
+            if answer != solution:
+                t.failed = True
+                t.msg = f"Incorrect overlap for lists={combination}"
+                t.want = solution
+                t.got = answer
+            cases.append(t)
+
+        return cases
+    
+    cases = g()
+    print_feedback(cases)

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/.ipynb_checkpoints/C4W3_SentencePiece_and_BPE-checkpoint.ipynb

@@ -0,0 +1,633 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SentencePiece and BPE "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introduction to Tokenization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In order to process text in neural network models it is first required to **encode** text as numbers with ids, since the tensor operations act on numbers. Finally, if the output of the network is to be words, it is required to **decode** the predicted tokens ids back to text.\n",
+    "\n",
+    "To encode text, the first decision that has to be made is to what level of granularity are we going to consider the text? Because ultimately, from these **tokens**, features are going to be created about them. Many different experiments have been carried out using *words*, *morphological units*, *phonemic units* or *characters* as tokens. For example, \n",
+    "\n",
+    "- Tokens are tricky. (raw text)\n",
+    "- Tokens are tricky . ([words](https://arxiv.org/pdf/1301.3781))\n",
+    "- Token s _ are _ trick _ y . ([morphemes](https://arxiv.org/pdf/1907.02423.pdf))\n",
+    "- t oʊ k ə n z _ ɑː _ ˈt r ɪ k i. ([phonemes](https://www.aclweb.org/anthology/W18-5812.pdf), for STT)\n",
+    "- T o k e n s _ a r e _ t r i c k y . ([character](https://www.aclweb.org/anthology/C18-1139/))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "But how to identify these units, such as words, is largely determined by the language they come from. For example, in many European languages a space is used to separate words, while in some Asian languages there are no spaces between words. Compare English and Mandarin.\n",
+    "\n",
+    "- Tokens are tricky. (original sentence)\n",
+    "- 标记很棘手 (Mandarin)\n",
+    "- Biāojì hěn jíshǒu (pinyin)\n",
+    "- 标记 很 棘手 (Mandarin with spaces)\n",
+    "\n",
+    "\n",
+    "So, the ability to **tokenize**, i.e. split text into meaningful fundamental units, is not always straight-forward.\n",
+    "\n",
+    "Also, there are practical issues of how large our *vocabulary* of words, `vocab_size`, should be, considering memory limitations vs. coverage. A compromise may be need to be made between: \n",
+    "* the finest-grained models employing characters which can be memory intensive and \n",
+    "* more computationally efficient *subword* units such as [n-grams](https://arxiv.org/pdf/1712.09405) or larger units.\n",
+    "\n",
+    "In [SentencePiece](https://www.aclweb.org/anthology/D18-2012.pdf) unicode characters are grouped together using either a [unigram language model](https://www.aclweb.org/anthology/P18-1007.pdf) (used in this week's assignment) or [BPE](https://arxiv.org/pdf/1508.07909.pdf), **byte-pair encoding**. We will discuss BPE, since BERT and many of its variants use a modified version of BPE and its pseudocode is easy to implement and understand... hopefully!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## SentencePiece Preprocessing\n",
+    "### NFKC Normalization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Unsurprisingly, even using unicode to initially tokenize text can be ambiguous, e.g., "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "é = é : False\n"
+     ]
+    }
+   ],
+   "source": [
+    "eaccent = '\\u00E9'\n",
+    "e_accent = '\\u0065\\u0301'\n",
+    "print(f'{eaccent} = {e_accent} : {eaccent == e_accent}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SentencePiece uses the Unicode standard normalization form, [NFKC](https://en.wikipedia.org/wiki/Unicode_equivalence), so this isn't an issue. Looking at the example from above but with normalization:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "é = é : True\n"
+     ]
+    }
+   ],
+   "source": [
+    "from unicodedata import normalize\n",
+    "\n",
+    "norm_eaccent = normalize('NFKC', '\\u00E9')\n",
+    "norm_e_accent = normalize('NFKC', '\\u0065\\u0301')\n",
+    "print(f'{norm_eaccent} = {norm_e_accent} : {norm_eaccent == norm_e_accent}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Normalization has actually changed the unicode code point (unicode unique id) for one of these two characters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_hex_encoding(s):\n",
+    "    return ' '.join(hex(ord(c)) for c in s)\n",
+    "\n",
+    "def print_string_and_encoding(s):\n",
+    "    print(f'{s} : {get_hex_encoding(s)}') "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "é : 0xe9\n",
+      "é : 0x65 0x301\n",
+      "é : 0xe9\n",
+      "é : 0xe9\n"
+     ]
+    }
+   ],
+   "source": [
+    "for s in [eaccent, e_accent, norm_eaccent, norm_e_accent]:\n",
+    "    print_string_and_encoding(s)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This normalization has other side effects which may be considered useful such as converting curly quotes &ldquo; to \" their ASCII equivalent. (<sup>*</sup>Although we *now* lose directionality of the quote...)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Lossless Tokenization\n",
+    "\n",
+    "SentencePiece also ensures that when you tokenize your data and detokenize your data the original position of white space is preserved. However, tabs and newlines are converted to spaces.\n",
+    "\n",
+    "To ensure this **lossless tokenization**, SentencePiece replaces white space with _ (U+2581). So that a simple join of the tokens by replacing underscores with spaces can restore the white space, even if there are consecutive symbols. But remember first to normalize and then replace spaces with _ (U+2581)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "s = 'Tokenization is hard.'\n",
+    "sn = normalize('NFKC', s)\n",
+    "sn_ = sn.replace(' ', '\\u2581')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "0x54 0x6f 0x6b 0x65 0x6e 0x69 0x7a 0x61 0x74 0x69 0x6f 0x6e 0x20 0x69 0x73 0x20 0x68 0x61 0x72 0x64 0x2e\n",
+      "0x54 0x6f 0x6b 0x65 0x6e 0x69 0x7a 0x61 0x74 0x69 0x6f 0x6e 0x20 0x69 0x73 0x20 0x68 0x61 0x72 0x64 0x2e\n",
+      "0x54 0x6f 0x6b 0x65 0x6e 0x69 0x7a 0x61 0x74 0x69 0x6f 0x6e 0x2581 0x69 0x73 0x2581 0x68 0x61 0x72 0x64 0x2e\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(get_hex_encoding(s))\n",
+    "print(get_hex_encoding(sn))\n",
+    "print(get_hex_encoding(sn_))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## BPE Algorithm\n",
+    "\n",
+    "After discussing the preprocessing that SentencePiece performs, you will get the data, preprocess it, and apply the BPE algorithm. You will see how this reproduces the tokenization produced by training SentencePiece on the example dataset (from this week's assignment).\n",
+    "\n",
+    "### Preparing our Data\n",
+    "First, you get the Squad data and process it as above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import ast\n",
+    "\n",
+    "def convert_json_examples_to_text(filepath):\n",
+    "    example_jsons = list(map(ast.literal_eval, open(filepath))) # Read in the json from the example file\n",
+    "    texts = [example_json['text'].decode('utf-8') for example_json in example_jsons] # Decode the byte sequences\n",
+    "    text = '\\n\\n'.join(texts)       # Separate different articles by two newlines\n",
+    "    text = normalize('NFKC', text)  # Normalize the text\n",
+    "\n",
+    "    with open('example.txt', 'w') as fw:\n",
+    "        fw.write(text)\n",
+    "    \n",
+    "    return text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = convert_json_examples_to_text('./data/data.txt')\n",
+    "print(text[:900])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the algorithm the `vocab` variable is actually a frequency dictionary of the words. Those words have been prepended with an *underscore* to indicate that they are the beginning of a word. Finally, the characters have been delimited by spaces so that the BPE algorithm can group the most common characters together in the dictionary in a greedy fashion. You will see how that is done shortly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "vocab = Counter(['\\u2581' + word for word in text.split()])\n",
+    "vocab = {' '.join([l for l in word]): freq for word, freq in vocab.items()}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def show_vocab(vocab, end='\\n', limit=20):\n",
+    "    \"\"\"Show word frequencys in vocab up to the limit number of words\"\"\"\n",
+    "    shown = 0\n",
+    "    for word, freq in vocab.items():\n",
+    "        print(f'{word}: {freq}', end=end)\n",
+    "        shown +=1\n",
+    "        if shown > limit:\n",
+    "            break"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "show_vocab(vocab)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You check the size of the vocabulary (frequency dictionary) because this is the one hyperparameter that BPE depends on crucially on how far it breaks up a word into SentencePieces. It turns out that for your trained model on the small dataset that 60% of 455 merges of the most frequent characters need to be done to reproduce the upperlimit of a 32K `vocab_size` over the entire corpus of examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f'Total number of unique words: {len(vocab)}')\n",
+    "print(f'Number of merges required to reproduce SentencePiece training on the whole corpus: {int(0.60*len(vocab))}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### BPE Algorithm\n",
+    "Directly from the BPE paper you have the following algorithm. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re, collections\n",
+    "\n",
+    "def get_stats(vocab):\n",
+    "    pairs = collections.defaultdict(int)\n",
+    "    for word, freq in vocab.items():\n",
+    "        symbols = word.split()\n",
+    "        for i in range(len(symbols) - 1):\n",
+    "            pairs[symbols[i], symbols[i+1]] += freq\n",
+    "    return pairs\n",
+    "\n",
+    "def merge_vocab(pair, v_in):\n",
+    "    v_out = {}\n",
+    "    bigram = re.escape(' '.join(pair))\n",
+    "    p = re.compile(r'(?<!\\S)' + bigram + r'(?!\\S)')\n",
+    "    for word in v_in:\n",
+    "        w_out = p.sub(''.join(pair), word)\n",
+    "        v_out[w_out] = v_in[word]\n",
+    "    return v_out\n",
+    "\n",
+    "def get_sentence_piece_vocab(vocab, frac_merges=0.60):\n",
+    "    sp_vocab = vocab.copy()\n",
+    "    num_merges = int(len(sp_vocab)*frac_merges)\n",
+    "    \n",
+    "    for i in range(num_merges):\n",
+    "        pairs = get_stats(sp_vocab)\n",
+    "        best = max(pairs, key=pairs.get)\n",
+    "        sp_vocab = merge_vocab(best, sp_vocab)\n",
+    "\n",
+    "    return sp_vocab"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To understand what's going on first take a look at the third function `get_sentence_piece_vocab`. It takes in the current `vocab` word-frequency dictionary and the fraction, `frac_merges`, of the total `vocab_size` to merge characters in the words of the dictionary, `num_merges` times. Then for each *merge* operation it `get_stats` on how many of each pair of character sequences there are. It gets the most frequent *pair* of symbols as the `best` pair. Then it merges that pair of symbols (removes the space between them) in each word in the `vocab` that contains this `best` (= `pair`). Consequently, `merge_vocab` creates a new `vocab`, `v_out`. This process is repeated `num_merges` times and the result is the set of SentencePieces (keys of the final `sp_vocab`)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Additional Discussion of BPE Algorithm"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please feel free to skip the below if the above description was enough.\n",
+    "\n",
+    "In a little more detail you can see in `get_stats` you initially create a list of bigram (two character sequence) frequencies from the vocabulary. Later, this may include trigrams, quadgrams, etc. Note that the key of the `pairs` frequency dictionary is actually a 2-tuple, which is just shorthand notation for a pair.\n",
+    "\n",
+    "In `merge_vocab` you take in an individual `pair` (of character sequences, note this is the most frequency `best` pair) and the current `vocab` as `v_in`. You create a new `vocab`, `v_out`, from the old by joining together the characters in the pair (removing the space), if they are present in a word of the dictionary.\n",
+    "\n",
+    "[Warning](https://regex101.com/): the expression `(?<!\\S)` means that either a whitespace character follows before the `bigram` or there is nothing before the bigram (it is the beginning of the word), similarly for `(?!\\S)` for preceding whitespace or the end of the word. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sp_vocab = get_sentence_piece_vocab(vocab)\n",
+    "show_vocab(sp_vocab) "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Train SentencePiece BPE Tokenizer on Example Data\n",
+    "### Explore SentencePiece Model\n",
+    "First, explore the SentencePiece model provided with this week's assignment. Remember you can always use Python's built in `help` command to see the documentation for any object or method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sentencepiece as spm\n",
+    "sp = spm.SentencePieceProcessor(model_file='./data/sentencepiece.model')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# help(sp)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Try it out on the first sentence of the example text."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "s0 = 'Beginners BBQ Class Taking Place in Missoula!'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# encode: text => id\n",
+    "print(sp.encode_as_pieces(s0))\n",
+    "print(sp.encode_as_ids(s0))\n",
+    "\n",
+    "# decode: id => text\n",
+    "print(sp.decode_pieces(sp.encode_as_pieces(s0)))\n",
+    "print(sp.decode_ids([12847, 277]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Notice how SentencePiece breaks the words into seemingly odd parts, but you have seen something similar with BPE. But how close was the model trained on the whole corpus of examples with a `vocab_size` of 32,000 instead of 455? Here you can also test what happens to white space, like '\\n'. \n",
+    "\n",
+    "But first note that SentencePiece encodes the SentencePieces, the tokens, and has reserved some of the ids as can be seen in this week's assignment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "uid = 15068\n",
+    "spiece = \"\\u2581BBQ\"\n",
+    "unknown = \"__MUST_BE_UNKNOWN__\"\n",
+    "\n",
+    "# id <=> piece conversion\n",
+    "print(f'SentencePiece for ID {uid}: {sp.id_to_piece(uid)}')\n",
+    "print(f'ID for Sentence Piece {spiece}: {sp.piece_to_id(spiece)}')\n",
+    "\n",
+    "# returns 0 for unknown tokens (we can change the id for UNK)\n",
+    "print(f'ID for unknown text {unknown}: {sp.piece_to_id(unknown)}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f'Beginning of sentence id: {sp.bos_id()}')\n",
+    "print(f'Pad id: {sp.pad_id()}')\n",
+    "print(f'End of sentence id: {sp.eos_id()}')\n",
+    "print(f'Unknown id: {sp.unk_id()}')\n",
+    "print(f'Vocab size: {sp.vocab_size()}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also check what are the ids for the first part and last part of the vocabulary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('\\nId\\tSentP\\tControl?')\n",
+    "print('------------------------')\n",
+    "# <unk>, <s>, </s> are defined by default. Their ids are (0, 1, 2)\n",
+    "# <s> and </s> are defined as 'control' symbol.\n",
+    "for uid in range(10):\n",
+    "    print(uid, sp.id_to_piece(uid), sp.is_control(uid), sep='\\t')\n",
+    "    \n",
+    "# for uid in range(sp.vocab_size()-10,sp.vocab_size()):\n",
+    "#     print(uid, sp.id_to_piece(uid), sp.is_control(uid), sep='\\t')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Train SentencePiece BPE model with our example.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, train your own BPE model directly from the SentencePiece library and compare it to the results of the implemention of the algorithm from the BPE paper itself."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "spm.SentencePieceTrainer.train('--input=example.txt --model_prefix=example_bpe --vocab_size=450 --model_type=bpe')\n",
+    "sp_bpe = spm.SentencePieceProcessor()\n",
+    "sp_bpe.load('example_bpe.model')\n",
+    "\n",
+    "print('*** BPE ***')\n",
+    "print(sp_bpe.encode_as_pieces(s0))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "show_vocab(sp_vocab, end = ', ')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The implementation of BPE's code from the paper matches up pretty well with the library itself! The differences are probably accounted for by the `vocab_size`. There is also another technical difference in that in the SentencePiece implementation of BPE a priority queue is used to more efficiently keep track of the *best pairs*. Actually, there is a priority queue in the Python standard library called `heapq` if you would like to give that a try below! "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Optionally try to implement BPE using a priority queue below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from heapq import heappush, heappop"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def heapsort(iterable):\n",
+    "    h = []\n",
+    "    for value in iterable:\n",
+    "        heappush(h, value)\n",
+    "    return [heappop(h) for i in range(len(h))]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "a = [1,4,3,1,3,2,1,4,2]\n",
+    "heapsort(a)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For a more extensive example consider looking at the [SentencePiece repo](https://github.com/google/sentencepiece/blob/master/python/sentencepiece_python_module_example.ipynb). The last few sections of this code were repurposed from that tutorial. Thanks for your participation! Next stop BERT and T5!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/C4W3_SentencePiece_and_BPE.ipynb

@@ -0,0 +1,724 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SentencePiece and BPE "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introduction to Tokenization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In order to process text in neural network models it is first required to **encode** text as numbers with ids, since the tensor operations act on numbers. Finally, if the output of the network is to be words, it is required to **decode** the predicted tokens ids back to text.\n",
+    "\n",
+    "To encode text, the first decision that has to be made is to what level of granularity are we going to consider the text? Because ultimately, from these **tokens**, features are going to be created about them. Many different experiments have been carried out using *words*, *morphological units*, *phonemic units* or *characters* as tokens. For example, \n",
+    "\n",
+    "- Tokens are tricky. (raw text)\n",
+    "- Tokens are tricky . ([words](https://arxiv.org/pdf/1301.3781))\n",
+    "- Token s _ are _ trick _ y . ([morphemes](https://arxiv.org/pdf/1907.02423.pdf))\n",
+    "- t oʊ k ə n z _ ɑː _ ˈt r ɪ k i. ([phonemes](https://www.aclweb.org/anthology/W18-5812.pdf), for STT)\n",
+    "- T o k e n s _ a r e _ t r i c k y . ([character](https://www.aclweb.org/anthology/C18-1139/))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "But how to identify these units, such as words, is largely determined by the language they come from. For example, in many European languages a space is used to separate words, while in some Asian languages there are no spaces between words. Compare English and Mandarin.\n",
+    "\n",
+    "- Tokens are tricky. (original sentence)\n",
+    "- 标记很棘手 (Mandarin)\n",
+    "- Biāojì hěn jíshǒu (pinyin)\n",
+    "- 标记 很 棘手 (Mandarin with spaces)\n",
+    "\n",
+    "\n",
+    "So, the ability to **tokenize**, i.e. split text into meaningful fundamental units, is not always straight-forward.\n",
+    "\n",
+    "Also, there are practical issues of how large our *vocabulary* of words, `vocab_size`, should be, considering memory limitations vs. coverage. A compromise may be need to be made between: \n",
+    "* the finest-grained models employing characters which can be memory intensive and \n",
+    "* more computationally efficient *subword* units such as [n-grams](https://arxiv.org/pdf/1712.09405) or larger units.\n",
+    "\n",
+    "In [SentencePiece](https://www.aclweb.org/anthology/D18-2012.pdf) unicode characters are grouped together using either a [unigram language model](https://www.aclweb.org/anthology/P18-1007.pdf) (used in this week's assignment) or [BPE](https://arxiv.org/pdf/1508.07909.pdf), **byte-pair encoding**. We will discuss BPE, since BERT and many of its variants use a modified version of BPE and its pseudocode is easy to implement and understand... hopefully!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## SentencePiece Preprocessing\n",
+    "### NFKC Normalization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Unsurprisingly, even using unicode to initially tokenize text can be ambiguous, e.g., "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "é = é : False\n"
+     ]
+    }
+   ],
+   "source": [
+    "eaccent = '\\u00E9'\n",
+    "e_accent = '\\u0065\\u0301'\n",
+    "print(f'{eaccent} = {e_accent} : {eaccent == e_accent}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SentencePiece uses the Unicode standard normalization form, [NFKC](https://en.wikipedia.org/wiki/Unicode_equivalence), so this isn't an issue. Looking at the example from above but with normalization:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "é = é : True\n"
+     ]
+    }
+   ],
+   "source": [
+    "from unicodedata import normalize\n",
+    "\n",
+    "norm_eaccent = normalize('NFKC', '\\u00E9')\n",
+    "norm_e_accent = normalize('NFKC', '\\u0065\\u0301')\n",
+    "print(f'{norm_eaccent} = {norm_e_accent} : {norm_eaccent == norm_e_accent}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Normalization has actually changed the unicode code point (unicode unique id) for one of these two characters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_hex_encoding(s):\n",
+    "    return ' '.join(hex(ord(c)) for c in s)\n",
+    "\n",
+    "def print_string_and_encoding(s):\n",
+    "    print(f'{s} : {get_hex_encoding(s)}') "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "é : 0xe9\n",
+      "é : 0x65 0x301\n",
+      "é : 0xe9\n",
+      "é : 0xe9\n"
+     ]
+    }
+   ],
+   "source": [
+    "for s in [eaccent, e_accent, norm_eaccent, norm_e_accent]:\n",
+    "    print_string_and_encoding(s)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This normalization has other side effects which may be considered useful such as converting curly quotes &ldquo; to \" their ASCII equivalent. (<sup>*</sup>Although we *now* lose directionality of the quote...)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Lossless Tokenization\n",
+    "\n",
+    "SentencePiece also ensures that when you tokenize your data and detokenize your data the original position of white space is preserved. However, tabs and newlines are converted to spaces.\n",
+    "\n",
+    "To ensure this **lossless tokenization**, SentencePiece replaces white space with _ (U+2581). So that a simple join of the tokens by replacing underscores with spaces can restore the white space, even if there are consecutive symbols. But remember first to normalize and then replace spaces with _ (U+2581)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "s = 'Tokenization is hard.'\n",
+    "sn = normalize('NFKC', s)\n",
+    "sn_ = sn.replace(' ', '\\u2581')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "0x54 0x6f 0x6b 0x65 0x6e 0x69 0x7a 0x61 0x74 0x69 0x6f 0x6e 0x20 0x69 0x73 0x20 0x68 0x61 0x72 0x64 0x2e\n",
+      "0x54 0x6f 0x6b 0x65 0x6e 0x69 0x7a 0x61 0x74 0x69 0x6f 0x6e 0x20 0x69 0x73 0x20 0x68 0x61 0x72 0x64 0x2e\n",
+      "0x54 0x6f 0x6b 0x65 0x6e 0x69 0x7a 0x61 0x74 0x69 0x6f 0x6e 0x2581 0x69 0x73 0x2581 0x68 0x61 0x72 0x64 0x2e\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(get_hex_encoding(s))\n",
+    "print(get_hex_encoding(sn))\n",
+    "print(get_hex_encoding(sn_))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## BPE Algorithm\n",
+    "\n",
+    "After discussing the preprocessing that SentencePiece performs, you will get the data, preprocess it, and apply the BPE algorithm. You will see how this reproduces the tokenization produced by training SentencePiece on the example dataset (from this week's assignment).\n",
+    "\n",
+    "### Preparing our Data\n",
+    "First, you get the Squad data and process it as above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import ast\n",
+    "\n",
+    "def convert_json_examples_to_text(filepath):\n",
+    "    example_jsons = list(map(ast.literal_eval, open(filepath))) # Read in the json from the example file\n",
+    "    texts = [example_json['text'].decode('utf-8') for example_json in example_jsons] # Decode the byte sequences\n",
+    "    text = '\\n\\n'.join(texts)       # Separate different articles by two newlines\n",
+    "    text = normalize('NFKC', text)  # Normalize the text\n",
+    "\n",
+    "    with open('example.txt', 'w') as fw:\n",
+    "        fw.write(text)\n",
+    "    \n",
+    "    return text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Beginners BBQ Class Taking Place in Missoula!\n",
+      "Do you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\n",
+      "He will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\n",
+      "The cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.\n",
+      "\n",
+      "Discussion in 'Mac OS X Lion (10.7)' started by axboi87, Jan 20, 2012.\n",
+      "I've got a 500gb internal drive and a 240gb SSD.\n",
+      "When trying to restore using di\n"
+     ]
+    }
+   ],
+   "source": [
+    "text = convert_json_examples_to_text('./data/data.txt')\n",
+    "print(text[:900])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the algorithm the `vocab` variable is actually a frequency dictionary of the words. Those words have been prepended with an *underscore* to indicate that they are the beginning of a word. Finally, the characters have been delimited by spaces so that the BPE algorithm can group the most common characters together in the dictionary in a greedy fashion. You will see how that is done shortly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import Counter\n",
+    "\n",
+    "vocab = Counter(['\\u2581' + word for word in text.split()])\n",
+    "vocab = {' '.join([l for l in word]): freq for word, freq in vocab.items()}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def show_vocab(vocab, end='\\n', limit=20):\n",
+    "    \"\"\"Show word frequencys in vocab up to the limit number of words\"\"\"\n",
+    "    shown = 0\n",
+    "    for word, freq in vocab.items():\n",
+    "        print(f'{word}: {freq}', end=end)\n",
+    "        shown +=1\n",
+    "        if shown > limit:\n",
+    "            break"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "▁ B e g i n n e r s: 1\n",
+      "▁ B B Q: 3\n",
+      "▁ C l a s s: 2\n",
+      "▁ T a k i n g: 1\n",
+      "▁ P l a c e: 1\n",
+      "▁ i n: 15\n",
+      "▁ M i s s o u l a !: 1\n",
+      "▁ D o: 1\n",
+      "▁ y o u: 13\n",
+      "▁ w a n t: 1\n",
+      "▁ t o: 33\n",
+      "▁ g e t: 2\n",
+      "▁ b e t t e r: 2\n",
+      "▁ a t: 1\n",
+      "▁ m a k i n g: 2\n",
+      "▁ d e l i c i o u s: 1\n",
+      "▁ B B Q ?: 1\n",
+      "▁ Y o u: 1\n",
+      "▁ w i l l: 6\n",
+      "▁ h a v e: 4\n",
+      "▁ t h e: 31\n"
+     ]
+    }
+   ],
+   "source": [
+    "show_vocab(vocab)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You check the size of the vocabulary (frequency dictionary) because this is the one hyperparameter that BPE depends on crucially on how far it breaks up a word into SentencePieces. It turns out that for your trained model on the small dataset that 60% of 455 merges of the most frequent characters need to be done to reproduce the upperlimit of a 32K `vocab_size` over the entire corpus of examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total number of unique words: 455\n",
+      "Number of merges required to reproduce SentencePiece training on the whole corpus: 273\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(f'Total number of unique words: {len(vocab)}')\n",
+    "print(f'Number of merges required to reproduce SentencePiece training on the whole corpus: {int(0.60*len(vocab))}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### BPE Algorithm\n",
+    "Directly from the BPE paper you have the following algorithm. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re, collections\n",
+    "\n",
+    "def get_stats(vocab):\n",
+    "    pairs = collections.defaultdict(int)\n",
+    "    for word, freq in vocab.items():\n",
+    "        symbols = word.split()\n",
+    "        for i in range(len(symbols) - 1):\n",
+    "            pairs[symbols[i], symbols[i+1]] += freq\n",
+    "    return pairs\n",
+    "\n",
+    "def merge_vocab(pair, v_in):\n",
+    "    v_out = {}\n",
+    "    bigram = re.escape(' '.join(pair))\n",
+    "    p = re.compile(r'(?<!\\S)' + bigram + r'(?!\\S)')\n",
+    "    for word in v_in:\n",
+    "        w_out = p.sub(''.join(pair), word)\n",
+    "        v_out[w_out] = v_in[word]\n",
+    "    return v_out\n",
+    "\n",
+    "def get_sentence_piece_vocab(vocab, frac_merges=0.60):\n",
+    "    sp_vocab = vocab.copy()\n",
+    "    num_merges = int(len(sp_vocab)*frac_merges)\n",
+    "    \n",
+    "    for i in range(num_merges):\n",
+    "        pairs = get_stats(sp_vocab)\n",
+    "        best = max(pairs, key=pairs.get)\n",
+    "        sp_vocab = merge_vocab(best, sp_vocab)\n",
+    "\n",
+    "    return sp_vocab"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To understand what's going on first take a look at the third function `get_sentence_piece_vocab`. It takes in the current `vocab` word-frequency dictionary and the fraction, `frac_merges`, of the total `vocab_size` to merge characters in the words of the dictionary, `num_merges` times. Then for each *merge* operation it `get_stats` on how many of each pair of character sequences there are. It gets the most frequent *pair* of symbols as the `best` pair. Then it merges that pair of symbols (removes the space between them) in each word in the `vocab` that contains this `best` (= `pair`). Consequently, `merge_vocab` creates a new `vocab`, `v_out`. This process is repeated `num_merges` times and the result is the set of SentencePieces (keys of the final `sp_vocab`)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Additional Discussion of BPE Algorithm"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please feel free to skip the below if the above description was enough.\n",
+    "\n",
+    "In a little more detail you can see in `get_stats` you initially create a list of bigram (two character sequence) frequencies from the vocabulary. Later, this may include trigrams, quadgrams, etc. Note that the key of the `pairs` frequency dictionary is actually a 2-tuple, which is just shorthand notation for a pair.\n",
+    "\n",
+    "In `merge_vocab` you take in an individual `pair` (of character sequences, note this is the most frequency `best` pair) and the current `vocab` as `v_in`. You create a new `vocab`, `v_out`, from the old by joining together the characters in the pair (removing the space), if they are present in a word of the dictionary.\n",
+    "\n",
+    "[Warning](https://regex101.com/): the expression `(?<!\\S)` means that either a whitespace character follows before the `bigram` or there is nothing before the bigram (it is the beginning of the word), similarly for `(?!\\S)` for preceding whitespace or the end of the word. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "▁B e g in n ers: 1\n",
+      "▁BBQ: 3\n",
+      "▁Cl ass: 2\n",
+      "▁T ak ing: 1\n",
+      "▁P la ce: 1\n",
+      "▁in: 15\n",
+      "▁M is s ou la !: 1\n",
+      "▁D o: 1\n",
+      "▁you: 13\n",
+      "▁w an t: 1\n",
+      "▁to: 33\n",
+      "▁g et: 2\n",
+      "▁be t ter: 2\n",
+      "▁a t: 1\n",
+      "▁mak ing: 2\n",
+      "▁d e l ic i ou s: 1\n",
+      "▁BBQ ?: 1\n",
+      "▁ Y ou: 1\n",
+      "▁will: 6\n",
+      "▁have: 4\n",
+      "▁the: 31\n"
+     ]
+    }
+   ],
+   "source": [
+    "sp_vocab = get_sentence_piece_vocab(vocab)\n",
+    "show_vocab(sp_vocab) "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Train SentencePiece BPE Tokenizer on Example Data\n",
+    "### Explore SentencePiece Model\n",
+    "First, explore the SentencePiece model provided with this week's assignment. Remember you can always use Python's built in `help` command to see the documentation for any object or method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sentencepiece as spm\n",
+    "sp = spm.SentencePieceProcessor(model_file='./data/sentencepiece.model')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# help(sp)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Try it out on the first sentence of the example text."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "s0 = 'Beginners BBQ Class Taking Place in Missoula!'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['▁Beginn', 'ers', '▁BBQ', '▁Class', '▁', 'Taking', '▁Place', '▁in', '▁Miss', 'oul', 'a', '!']\n",
+      "[12847, 277, 15068, 4501, 3, 12297, 3399, 16, 5964, 7115, 9, 55]\n",
+      "Beginners BBQ Class Taking Place in Missoula!\n",
+      "Beginners\n"
+     ]
+    }
+   ],
+   "source": [
+    "# encode: text => id\n",
+    "print(sp.encode_as_pieces(s0))\n",
+    "print(sp.encode_as_ids(s0))\n",
+    "\n",
+    "# decode: id => text\n",
+    "print(sp.decode_pieces(sp.encode_as_pieces(s0)))\n",
+    "print(sp.decode_ids([12847, 277]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Notice how SentencePiece breaks the words into seemingly odd parts, but you have seen something similar with BPE. But how close was the model trained on the whole corpus of examples with a `vocab_size` of 32,000 instead of 455? Here you can also test what happens to white space, like '\\n'. \n",
+    "\n",
+    "But first note that SentencePiece encodes the SentencePieces, the tokens, and has reserved some of the ids as can be seen in this week's assignment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "uid = 15068\n",
+    "spiece = \"\\u2581BBQ\"\n",
+    "unknown = \"__MUST_BE_UNKNOWN__\"\n",
+    "\n",
+    "# id <=> piece conversion\n",
+    "print(f'SentencePiece for ID {uid}: {sp.id_to_piece(uid)}')\n",
+    "print(f'ID for Sentence Piece {spiece}: {sp.piece_to_id(spiece)}')\n",
+    "\n",
+    "# returns 0 for unknown tokens (we can change the id for UNK)\n",
+    "print(f'ID for unknown text {unknown}: {sp.piece_to_id(unknown)}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f'Beginning of sentence id: {sp.bos_id()}')\n",
+    "print(f'Pad id: {sp.pad_id()}')\n",
+    "print(f'End of sentence id: {sp.eos_id()}')\n",
+    "print(f'Unknown id: {sp.unk_id()}')\n",
+    "print(f'Vocab size: {sp.vocab_size()}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also check what are the ids for the first part and last part of the vocabulary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('\\nId\\tSentP\\tControl?')\n",
+    "print('------------------------')\n",
+    "# <unk>, <s>, </s> are defined by default. Their ids are (0, 1, 2)\n",
+    "# <s> and </s> are defined as 'control' symbol.\n",
+    "for uid in range(10):\n",
+    "    print(uid, sp.id_to_piece(uid), sp.is_control(uid), sep='\\t')\n",
+    "    \n",
+    "# for uid in range(sp.vocab_size()-10,sp.vocab_size()):\n",
+    "#     print(uid, sp.id_to_piece(uid), sp.is_control(uid), sep='\\t')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Train SentencePiece BPE model with our example.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, train your own BPE model directly from the SentencePiece library and compare it to the results of the implemention of the algorithm from the BPE paper itself."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "spm.SentencePieceTrainer.train('--input=example.txt --model_prefix=example_bpe --vocab_size=450 --model_type=bpe')\n",
+    "sp_bpe = spm.SentencePieceProcessor()\n",
+    "sp_bpe.load('example_bpe.model')\n",
+    "\n",
+    "print('*** BPE ***')\n",
+    "print(sp_bpe.encode_as_pieces(s0))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "show_vocab(sp_vocab, end = ', ')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The implementation of BPE's code from the paper matches up pretty well with the library itself! The differences are probably accounted for by the `vocab_size`. There is also another technical difference in that in the SentencePiece implementation of BPE a priority queue is used to more efficiently keep track of the *best pairs*. Actually, there is a priority queue in the Python standard library called `heapq` if you would like to give that a try below! "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Optionally try to implement BPE using a priority queue below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from heapq import heappush, heappop"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def heapsort(iterable):\n",
+    "    h = []\n",
+    "    for value in iterable:\n",
+    "        heappush(h, value)\n",
+    "    return [heappop(h) for i in range(len(h))]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "a = [1,4,3,1,3,2,1,4,2]\n",
+    "heapsort(a)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For a more extensive example consider looking at the [SentencePiece repo](https://github.com/google/sentencepiece/blob/master/python/sentencepiece_python_module_example.ipynb). The last few sections of this code were repurposed from that tutorial. Thanks for your participation! Next stop BERT and T5!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/data/data.txt

@@ -0,0 +1,5 @@
+{'content-length': b'1970', 'content-type': b'text/plain', 'text': b'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': b'2019-04-25T12:57:54Z', 'url': b'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
+{'content-length': b'12064', 'content-type': b'text/plain', 'text': b'Discussion in \'Mac OS X Lion (10.7)\' started by axboi87, Jan 20, 2012.\nI\'ve got a 500gb internal drive and a 240gb SSD.\nWhen trying to restore using disk utility i\'m given the error "Not enough space on disk ____ to restore"\nBut I shouldn\'t have to do that!!!\nAny ideas or workarounds before resorting to the above?\nUse Carbon Copy Cloner to copy one drive to the other. I\'ve done this several times going from larger HDD to smaller SSD and I wound up with a bootable SSD drive. One step you have to remember not to skip is to use Disk Utility to partition the SSD as GUID partition scheme HFS+ before doing the clone. If it came Apple Partition Scheme, even if you let CCC do the clone, the resulting drive won\'t be bootable. CCC usually works in "file mode" and it can easily copy a larger drive (that\'s mostly empty) onto a smaller drive. If you tell CCC to clone a drive you did NOT boot from, it can work in block copy mode where the destination drive must be the same size or larger than the drive you are cloning from (if I recall).\nI\'ve actually done this somehow on Disk Utility several times (booting from a different drive (or even the dvd) so not running disk utility from the drive your cloning) and had it work just fine from larger to smaller bootable clone. Definitely format the drive cloning to first, as bootable Apple etc..\nThanks for pointing this out. My only experience using DU to go larger to smaller was when I was trying to make a Lion install stick and I was unable to restore InstallESD.dmg to a 4 GB USB stick but of course the reason that wouldn\'t fit is there was slightly more than 4 GB of data.', 'timestamp': b'2019-04-21T10:07:13Z', 'url':b'https://forums.macrumors.com/threads/restore-from-larger-disk-to-smaller-disk.1311329/'}
+{'content-length': b'5235', 'content-type': b'text/plain', 'text': b'Foil plaid lycra and spandex shortall with metallic slinky insets. Attached metallic elastic belt with O-ring. Headband included. Great hip hop or jazz dance costume. Made in the USA.', 'timestamp': b'2019-04-25T10:40:23Z', 'url': b'https://awishcometrue.com/Catalogs/Clearance/Tweens/V1960-Find-A-Way'}
+{'content-length': b'4967', 'content-type': b'text/plain', 'text': b"How many backlinks per day for new site?\nDiscussion in 'Black Hat SEO' started by Omoplata, Dec 3, 2010.\n1) for a newly created site, what's the max # backlinks per day I should do to be safe?\n2) how long do I have to let my site age before I can start making more blinks?\nI did about 6000 forum profiles every 24 hours for 10 days for one of my sites which had a brand new domain.\nThere is three backlinks for every of these forum profile so thats 18 000 backlinks every 24 hours and nothing happened in terms of being penalized or sandboxed. This is now maybe 3 months ago and the site is ranking on first page for a lot of my targeted keywords.\nbuild more you can in starting but do manual submission and not spammy type means manual + relevant to the post.. then after 1 month you can make a big blast..\nWow, dude, you built 18k backlinks a day on a brand new site? How quickly did you rank up? What kind of competition/searches did those keywords have?", 'timestamp': b'2019-04-21T12:46:19Z', 'url': b'https://www.blackhatworld.com/seo/how-many-backlinks-per-day-for-new-site.258615/'}
+{'content-length': b'4499', 'content-type': b'text/plain', 'text': b'The Denver Board of Education opened the 2017-18 school year with an update on projects that include new construction, upgrades, heat mitigation and quality learning environments.\nWe are excited that Denver students will be the beneficiaries of a four year, $572 million General Obligation Bond. Since the passage of the bond, our construction team has worked to schedule the projects over the four-year term of the bond.\nDenver voters on Tuesday approved bond and mill funding measures for students in Denver Public Schools, agreeing to invest $572 million in bond funding to build and improve schools and $56.6 million in operating dollars to support proven initiatives, such as early literacy.\nDenver voters say yes to bond and mill levy funding support for DPS students and schools. Click to learn more about the details of the voter-approved bond measure.\nDenver voters on Nov. 8 approved bond and mill funding measures for DPS students and schools. Learn more about what\xe2\x80\x99s included in the mill levy measure.', 'timestamp': b'2019-04-20T14:33:21Z', 'url': b'http://bond.dpsk12.org/category/news/'}

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/data/example.txt

@@ -0,0 +1,30 @@
+Beginners BBQ Class Taking Place in Missoula!
+Do you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.
+He will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.
+The cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.
+
+Discussion in 'Mac OS X Lion (10.7)' started by axboi87, Jan 20, 2012.
+I've got a 500gb internal drive and a 240gb SSD.
+When trying to restore using disk utility i'm given the error "Not enough space on disk ____ to restore"
+But I shouldn't have to do that!!!
+Any ideas or workarounds before resorting to the above?
+Use Carbon Copy Cloner to copy one drive to the other. I've done this several times going from larger HDD to smaller SSD and I wound up with a bootable SSD drive. One step you have to remember not to skip is to use Disk Utility to partition the SSD as GUID partition scheme HFS+ before doing the clone. If it came Apple Partition Scheme, even if you let CCC do the clone, the resulting drive won't be bootable. CCC usually works in "file mode" and it can easily copy a larger drive (that's mostly empty) onto a smaller drive. If you tell CCC to clone a drive you did NOT boot from, it can work in block copy mode where the destination drive must be the same size or larger than the drive you are cloning from (if I recall).
+I've actually done this somehow on Disk Utility several times (booting from a different drive (or even the dvd) so not running disk utility from the drive your cloning) and had it work just fine from larger to smaller bootable clone. Definitely format the drive cloning to first, as bootable Apple etc..
+Thanks for pointing this out. My only experience using DU to go larger to smaller was when I was trying to make a Lion install stick and I was unable to restore InstallESD.dmg to a 4 GB USB stick but of course the reason that wouldn't fit is there was slightly more than 4 GB of data.
+
+Foil plaid lycra and spandex shortall with metallic slinky insets. Attached metallic elastic belt with O-ring. Headband included. Great hip hop or jazz dance costume. Made in the USA.
+
+How many backlinks per day for new site?
+Discussion in 'Black Hat SEO' started by Omoplata, Dec 3, 2010.
+1) for a newly created site, what's the max # backlinks per day I should do to be safe?
+2) how long do I have to let my site age before I can start making more blinks?
+I did about 6000 forum profiles every 24 hours for 10 days for one of my sites which had a brand new domain.
+There is three backlinks for every of these forum profile so thats 18 000 backlinks every 24 hours and nothing happened in terms of being penalized or sandboxed. This is now maybe 3 months ago and the site is ranking on first page for a lot of my targeted keywords.
+build more you can in starting but do manual submission and not spammy type means manual + relevant to the post.. then after 1 month you can make a big blast..
+Wow, dude, you built 18k backlinks a day on a brand new site? How quickly did you rank up? What kind of competition/searches did those keywords have?
+
+The Denver Board of Education opened the 2017-18 school year with an update on projects that include new construction, upgrades, heat mitigation and quality learning environments.
+We are excited that Denver students will be the beneficiaries of a four year, $572 million General Obligation Bond. Since the passage of the bond, our construction team has worked to schedule the projects over the four-year term of the bond.
+Denver voters on Tuesday approved bond and mill funding measures for students in Denver Public Schools, agreeing to invest $572 million in bond funding to build and improve schools and $56.6 million in operating dollars to support proven initiatives, such as early literacy.
+Denver voters say yes to bond and mill levy funding support for DPS students and schools. Click to learn more about the details of the voter-approved bond measure.
+Denver voters on Nov. 8 approved bond and mill funding measures for DPS students and schools. Learn more about what’s included in the mill levy measure.

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/data/example_bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72964ffad404e80f7c7450e333730318ecb18b36c94cd8191b23b4461283610b
+size 243359

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/data/example_bpe.vocab

@@ -0,0 +1,450 @@
+<unk>	0
+<s>	0
+</s>	0
+▁t	-0
+in	-1
+on	-2
+▁a	-3
+er	-4
+▁s	-5
+▁th	-6
+or	-7
+▁b	-8
+▁d	-9
+▁m	-10
+it	-11
+nd	-12
+ou	-13
+▁f	-14
+ing	-15
+▁the	-16
+▁to	-17
+ve	-18
+▁w	-19
+ar	-20
+▁c	-21
+at	-22
+ll	-23
+▁in	-24
+re	-25
+en	-26
+is	-27
+le	-28
+st	-29
+ion	-30
+▁and	-31
+an	-32
+▁p	-33
+ot	-34
+▁y	-35
+as	-36
+ed	-37
+▁o	-38
+ch	-39
+ro	-40
+▁D	-41
+▁I	-42
+▁e	-43
+▁be	-44
+▁h	-45
+▁for	-46
+▁you	-47
+ill	-48
+ive	-49
+ver	-50
+▁of	-51
+▁n	-52
+all	-53
+▁dr	-54
+▁on	-55
+▁drive	-56
+ck	-57
+es	-58
+▁u	-59
+ore	-60
+▁st	-61
+et	-62
+il	-63
+ud	-64
+▁C	-65
+▁S	-66
+▁re	-67
+al	-68
+ay	-69
+pp	-70
+▁2	-71
+▁B	-72
+▁T	-73
+▁l	-74
+lin	-75
+▁cl	-76
+▁co	-77
+ks	-78
+me	-79
+ow	-80
+ts	-81
+▁H	-82
+ond	-83
+one	-84
+▁do	-85
+▁ha	-86
+▁is	-87
+ly	-88
+mp	-89
+art	-90
+rom	-91
+▁le	-92
+▁me	-93
+▁bond	-94
+▁from	-95
+▁mill	-96
+ic	-97
+id	-98
+la	-99
+se	-100
+▁g	-101
+arg	-102
+ers	-103
+ite	-104
+ith	-105
+ity	-106
+oot	-107
+our	-108
+▁Th	-109
+▁ne	-110
+▁wh	-111
+▁Den	-112
+▁sch	-113
+links	-114
+▁that	-115
+▁will	-116
+▁Denver	-117
+SD	-118
+ab	-119
+ak	-120
+ce	-121
+cl	-122
+ct	-123
+ir	-124
+ol	-125
+▁(	-126
+▁1	-127
+▁G	-128
+▁O	-129
+▁U	-130
+▁W	-131
+ack	-132
+and	-133
+ass	-134
+isk	-135
+ool	-136
+ort	-137
+▁bu	-138
+▁it	-139
+▁or	-140
+▁sm	-141
+▁te	-142
+able	-143
+clud	-144
+ents	-145
+rove	-146
+very	-147
+▁can	-148
+▁new	-149
+▁wor	-150
+arger	-151
+ation	-152
+ition	-153
+▁back	-154
+▁boot	-155
+▁have	-156
+▁more	-157
+▁site	-158
+▁with	-159
+▁every	-160
+▁larger	-161
+▁backlinks	-162
+BQ	-163
+ig	-164
+ld	-165
+py	-166
+th	-167
+▁$	-168
+▁A	-169
+▁L	-170
+▁k	-171
+▁v	-172
+ach	-173
+asu	-174
+ear	-175
+ick	-176
+out	-177
+ter	-178
+til	-179
+und	-180
+▁20	-181
+▁Cl	-182
+▁ab	-183
+▁sp	-184
+▁su	-185
+▁up	-186
+ools	-187
+▁BBQ	-188
+▁SSD	-189
+▁day	-190
+▁did	-191
+▁mak	-192
+▁not	-193
+▁pro	-194
+▁vot	-195
+▁was	-196
+aller	-197
+asure	-198
+▁fund	-199
+▁stud	-200
+▁this	-201
+▁work	-202
+tility	-203
+▁clone	-204
+▁start	-205
+▁includ	-206
+▁funding	-207
+▁measure	-208
+▁smaller	-209
+▁bootable	-210
+▁students	-211
+..	-212
+00	-213
+ad	-214
+ec	-215
+fi	-216
+ge	-217
+if	-218
+im	-219
+ip	-220
+qu	-221
+ru	-222
+us	-223
+▁M	-224
+▁P	-225
+▁j	-226
+ere	-227
+ree	-228
+▁$5	-229
+▁24	-230
+▁CC	-231
+▁He	-232
+▁as	-233
+▁mo	-234
+▁my	-235
+▁sa	-236
+▁se	-237
+▁sh	-238
+▁so	-239
+▁tr	-240
+▁us	-241
+file	-242
+fore	-243
+mpet	-244
+ould	-245
+sion	-246
+▁201	-247
+▁CCC	-248
+▁man	-249
+▁per	-250
+ction	-251
+oning	-252
+pport	-253
+roved	-254
+store	-255
+▁buil	-256
+▁copy	-257
+▁cost	-258
+▁disk	-259
+▁about	-260
+pproved	-261
+▁before	-262
+▁compet	-263
+▁voters	-264
+artition	-265
+▁cloning	-266
+▁million	-267
+▁restore	-268
+▁schools	-269
+0.	-270
+72	-271
+PS	-272
+__	-273
+ac	-274
+am	-275
+bl	-276
+bo	-277
+de	-278
+ds	-279
+ef	-280
+ep	-281
+ey	-282
+gb	-283
+iz	-284
+lt	-285
+mb	-286
+mo	-287
+um	-288
+ut	-289
+vy	-290
+▁"	-291
+▁'	-292
+▁3	-293
+▁4	-294
+▁N	-295
+▁i	-296
+▁r	-297
+000	-298
+age	-299
+ank	-300
+ant	-301
+arn	-302
+ata	-303
+cus	-304
+day	-305
+eme	-306
+erm	-307
+eyw	-308
+gin	-309
+ici	-310
+jec	-311
+oin	-312
+per	-313
+ual	-314
+ust	-315
+ven	-316
+▁18	-317
+▁GB	-318
+▁If	-319
+▁In	-320
+▁US	-321
+▁Wh	-322
+▁ag	-323
+▁br	-324
+▁by	-325
+▁ca	-326
+▁de	-327
+▁en	-328
+▁ex	-329
+▁go	-330
+▁qu	-331
+▁sk	-332
+ally	-333
+ened	-334
+ginn	-335
+imes	-336
+irst	-337
+last	-338
+mber	-339
+onst	-340
+onth	-341
+ords	-342
+ound	-343
+ours	-344
+pple	-345
+reat	-346
+tter	-347
+ying	-348
+▁DPS	-349
+▁Dis	-350
+▁How	-351
+▁Sch	-352
+▁The	-353
+▁are	-354
+▁but	-355
+▁get	-356
+▁had	-357
+▁let	-358
+▁met	-359
+▁now	-360
+▁one	-361
+▁rec	-362
+▁res	-363
+allic	-364
+jects	-365
+ouldn	-366
+stall	-367
+ually	-368
+veral	-369
+▁$572	-370
+▁Disk	-371
+▁Lion	-372
+▁done	-373
+▁	-374
+e	-375
+o	-376
+t	-377
+n	-378
+a	-379
+i	-380
+r	-381
+s	-382
+l	-383
+d	-384
+h	-385
+u	-386
+c	-387
+m	-388
+y	-389
+p	-390
+b	-391
+f	-392
+g	-393
+v	-394
+w	-395
+k	-396
+.	-397
+,	-398
+D	-399
+S	-400
+B	-401
+C	-402
+I	-403
+0	-404
+'	-405
+2	-406
+1	-407
+T	-408
+?	-409
+H	-410
+)	-411
+O	-412
+U	-413
+x	-414
+(	-415
+-	-416
+4	-417
+5	-418
+7	-419
+8	-420
+A	-421
+G	-422
+P	-423
+W	-424
+j	-425
+!	-426
+"	-427
+$	-428
+L	-429
+M	-430
+Q	-431
+_	-432
+z	-433
+3	-434
+6	-435
+E	-436
+N	-437
+q	-438
++	-439
+F	-440
+#	-441
+/	-442
+J	-443
+K	-444
+R	-445
+X	-446

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/data/sentencepiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
+size 791656

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/example.txt

@@ -0,0 +1,30 @@
+Beginners BBQ Class Taking Place in Missoula!
+Do you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.
+He will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.
+The cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.
+
+Discussion in 'Mac OS X Lion (10.7)' started by axboi87, Jan 20, 2012.
+I've got a 500gb internal drive and a 240gb SSD.
+When trying to restore using disk utility i'm given the error "Not enough space on disk ____ to restore"
+But I shouldn't have to do that!!!
+Any ideas or workarounds before resorting to the above?
+Use Carbon Copy Cloner to copy one drive to the other. I've done this several times going from larger HDD to smaller SSD and I wound up with a bootable SSD drive. One step you have to remember not to skip is to use Disk Utility to partition the SSD as GUID partition scheme HFS+ before doing the clone. If it came Apple Partition Scheme, even if you let CCC do the clone, the resulting drive won't be bootable. CCC usually works in "file mode" and it can easily copy a larger drive (that's mostly empty) onto a smaller drive. If you tell CCC to clone a drive you did NOT boot from, it can work in block copy mode where the destination drive must be the same size or larger than the drive you are cloning from (if I recall).
+I've actually done this somehow on Disk Utility several times (booting from a different drive (or even the dvd) so not running disk utility from the drive your cloning) and had it work just fine from larger to smaller bootable clone. Definitely format the drive cloning to first, as bootable Apple etc..
+Thanks for pointing this out. My only experience using DU to go larger to smaller was when I was trying to make a Lion install stick and I was unable to restore InstallESD.dmg to a 4 GB USB stick but of course the reason that wouldn't fit is there was slightly more than 4 GB of data.
+
+Foil plaid lycra and spandex shortall with metallic slinky insets. Attached metallic elastic belt with O-ring. Headband included. Great hip hop or jazz dance costume. Made in the USA.
+
+How many backlinks per day for new site?
+Discussion in 'Black Hat SEO' started by Omoplata, Dec 3, 2010.
+1) for a newly created site, what's the max # backlinks per day I should do to be safe?
+2) how long do I have to let my site age before I can start making more blinks?
+I did about 6000 forum profiles every 24 hours for 10 days for one of my sites which had a brand new domain.
+There is three backlinks for every of these forum profile so thats 18 000 backlinks every 24 hours and nothing happened in terms of being penalized or sandboxed. This is now maybe 3 months ago and the site is ranking on first page for a lot of my targeted keywords.
+build more you can in starting but do manual submission and not spammy type means manual + relevant to the post.. then after 1 month you can make a big blast..
+Wow, dude, you built 18k backlinks a day on a brand new site? How quickly did you rank up? What kind of competition/searches did those keywords have?
+
+The Denver Board of Education opened the 2017-18 school year with an update on projects that include new construction, upgrades, heat mitigation and quality learning environments.
+We are excited that Denver students will be the beneficiaries of a four year, $572 million General Obligation Bond. Since the passage of the bond, our construction team has worked to schedule the projects over the four-year term of the bond.
+Denver voters on Tuesday approved bond and mill funding measures for students in Denver Public Schools, agreeing to invest $572 million in bond funding to build and improve schools and $56.6 million in operating dollars to support proven initiatives, such as early literacy.
+Denver voters say yes to bond and mill levy funding support for DPS students and schools. Click to learn more about the details of the voter-approved bond measure.
+Denver voters on Nov. 8 approved bond and mill funding measures for DPS students and schools. Learn more about what’s included in the mill levy measure.

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/example_bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3c6c33b38015133abc0b477cfc51ef435a4741e2f907ae26760f61ce8ae85cb
+size 243359

NLP with Attention Models/QA/BPE_algorithm/home/jovyan/work/example_bpe.vocab

@@ -0,0 +1,450 @@
+<unk>	0
+<s>	0
+</s>	0
+▁t	-0
+in	-1
+on	-2
+▁a	-3
+er	-4
+▁s	-5
+▁th	-6
+or	-7
+▁b	-8
+▁d	-9
+▁m	-10
+it	-11
+nd	-12
+ou	-13
+▁f	-14
+ing	-15
+▁the	-16
+▁to	-17
+ve	-18
+▁w	-19
+ar	-20
+▁c	-21
+at	-22
+ll	-23
+▁in	-24
+re	-25
+en	-26
+is	-27
+le	-28
+st	-29
+ion	-30
+▁and	-31
+an	-32
+▁p	-33
+ot	-34
+▁y	-35
+as	-36
+ed	-37
+▁o	-38
+ch	-39
+ro	-40
+▁D	-41
+▁I	-42
+▁e	-43
+▁be	-44
+▁h	-45
+▁for	-46
+▁you	-47
+ill	-48
+ive	-49
+ver	-50
+▁of	-51
+▁n	-52
+all	-53
+▁dr	-54
+▁on	-55
+▁drive	-56
+ck	-57
+es	-58
+▁u	-59
+ore	-60
+▁st	-61
+et	-62
+il	-63
+ud	-64
+▁C	-65
+▁S	-66
+▁re	-67
+al	-68
+ay	-69
+pp	-70
+▁2	-71
+▁B	-72
+▁T	-73
+▁l	-74
+lin	-75
+▁cl	-76
+▁co	-77
+ks	-78
+me	-79
+ow	-80
+ts	-81
+▁H	-82
+ond	-83
+one	-84
+▁do	-85
+▁ha	-86
+▁is	-87
+ly	-88
+mp	-89
+art	-90
+rom	-91
+▁le	-92
+▁me	-93
+▁bond	-94
+▁from	-95
+▁mill	-96
+ic	-97
+id	-98
+la	-99
+se	-100
+▁g	-101
+arg	-102
+ers	-103
+ite	-104
+ith	-105
+ity	-106
+oot	-107
+our	-108
+▁Th	-109
+▁ne	-110
+▁wh	-111
+▁Den	-112
+▁sch	-113
+links	-114
+▁that	-115
+▁will	-116
+▁Denver	-117
+00	-118
+SD	-119
+ab	-120
+ak	-121
+ce	-122
+cl	-123
+ct	-124
+ir	-125
+ol	-126
+▁(	-127
+▁1	-128
+▁G	-129
+▁O	-130
+▁U	-131
+▁W	-132
+ack	-133
+and	-134
+ass	-135
+isk	-136
+ool	-137
+ort	-138
+▁bu	-139
+▁it	-140
+▁or	-141
+▁sm	-142
+▁te	-143
+able	-144
+clud	-145
+ents	-146
+rove	-147
+very	-148
+▁can	-149
+▁new	-150
+▁wor	-151
+arger	-152
+ation	-153
+ition	-154
+▁back	-155
+▁boot	-156
+▁have	-157
+▁more	-158
+▁site	-159
+▁with	-160
+▁every	-161
+▁larger	-162
+▁backlinks	-163
+BQ	-164
+ig	-165
+ld	-166
+py	-167
+th	-168
+▁$	-169
+▁A	-170
+▁L	-171
+▁k	-172
+▁v	-173
+ach	-174
+asu	-175
+ear	-176
+ick	-177
+out	-178
+ter	-179
+til	-180
+und	-181
+▁20	-182
+▁Cl	-183
+▁ab	-184
+▁sp	-185
+▁su	-186
+▁up	-187
+ools	-188
+▁BBQ	-189
+▁SSD	-190
+▁day	-191
+▁did	-192
+▁mak	-193
+▁not	-194
+▁pro	-195
+▁vot	-196
+▁was	-197
+aller	-198
+asure	-199
+▁fund	-200
+▁stud	-201
+▁this	-202
+▁work	-203
+tility	-204
+▁clone	-205
+▁start	-206
+▁includ	-207
+▁funding	-208
+▁measure	-209
+▁smaller	-210
+▁bootable	-211
+▁students	-212
+..	-213
+CC	-214
+__	-215
+ad	-216
+ec	-217
+fi	-218
+ge	-219
+if	-220
+im	-221
+ip	-222
+qu	-223
+ru	-224
+us	-225
+▁M	-226
+▁P	-227
+▁j	-228
+ere	-229
+ree	-230
+▁$5	-231
+▁24	-232
+▁He	-233
+▁as	-234
+▁mo	-235
+▁my	-236
+▁sa	-237
+▁se	-238
+▁sh	-239
+▁so	-240
+▁tr	-241
+▁us	-242
+file	-243
+fore	-244
+mpet	-245
+ould	-246
+sion	-247
+▁201	-248
+▁CCC	-249
+▁man	-250
+▁per	-251
+ction	-252
+oning	-253
+pport	-254
+roved	-255
+store	-256
+▁buil	-257
+▁copy	-258
+▁cost	-259
+▁disk	-260
+▁about	-261
+pproved	-262
+▁before	-263
+▁compet	-264
+▁voters	-265
+artition	-266
+▁cloning	-267
+▁million	-268
+▁restore	-269
+▁schools	-270
+!!	-271
+0.	-272
+72	-273
+PS	-274
+ac	-275
+am	-276
+bl	-277
+bo	-278
+de	-279
+ds	-280
+ef	-281
+ep	-282
+ey	-283
+gb	-284
+iz	-285
+lt	-286
+mb	-287
+mo	-288
+um	-289
+ut	-290
+vy	-291
+▁"	-292
+▁'	-293
+▁3	-294
+▁4	-295
+▁N	-296
+▁i	-297
+▁r	-298
+000	-299
+age	-300
+ank	-301
+ant	-302
+arn	-303
+ata	-304
+cus	-305
+day	-306
+eme	-307
+erm	-308
+eyw	-309
+gin	-310
+ici	-311
+jec	-312
+oin	-313
+per	-314
+ual	-315
+ust	-316
+ven	-317
+▁18	-318
+▁GB	-319
+▁If	-320
+▁In	-321
+▁US	-322
+▁Wh	-323
+▁ag	-324
+▁br	-325
+▁by	-326
+▁ca	-327
+▁de	-328
+▁en	-329
+▁ex	-330
+▁go	-331
+▁qu	-332
+▁sk	-333
+ally	-334
+ened	-335
+ginn	-336
+imes	-337
+irst	-338
+last	-339
+mber	-340
+onst	-341
+onth	-342
+ords	-343
+ound	-344
+ours	-345
+pple	-346
+reat	-347
+tter	-348
+ying	-349
+▁DPS	-350
+▁Dis	-351
+▁How	-352
+▁Sch	-353
+▁The	-354
+▁are	-355
+▁but	-356
+▁get	-357
+▁had	-358
+▁let	-359
+▁met	-360
+▁now	-361
+▁one	-362
+▁rec	-363
+▁res	-364
+allic	-365
+jects	-366
+ouldn	-367
+stall	-368
+ually	-369
+veral	-370
+▁$572	-371
+▁Disk	-372
+▁Lion	-373
+▁	-374
+e	-375
+o	-376
+t	-377
+n	-378
+a	-379
+i	-380
+r	-381
+s	-382
+l	-383
+d	-384
+h	-385
+u	-386
+c	-387
+m	-388
+y	-389
+p	-390
+b	-391
+f	-392
+g	-393
+v	-394
+w	-395
+k	-396
+.	-397
+,	-398
+D	-399
+S	-400
+B	-401
+C	-402
+I	-403
+0	-404
+'	-405
+2	-406
+1	-407
+T	-408
+?	-409
+H	-410
+)	-411
+O	-412
+U	-413
+x	-414
+(	-415
+-	-416
+4	-417
+5	-418
+7	-419
+8	-420
+A	-421
+G	-422
+P	-423
+W	-424
+j	-425
+!	-426
+"	-427
+$	-428
+L	-429
+M	-430
+Q	-431
+_	-432
+z	-433
+3	-434
+6	-435
+E	-436
+N	-437
+q	-438
++	-439
+F	-440
+#	-441
+/	-442
+J	-443
+K	-444
+R	-445
+X	-446

NLP with Attention Models/QA/QA_DistilBERT_pipline_FT/Files/tf/.ipynb_checkpoints/C4W3_HF_Lab1_QA_BERT-checkpoint.ipynb

@@ -0,0 +1,2110 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "u2UXutvEvpUj"
+   },
+   "source": [
+    "# Question Answering with BERT and HuggingFace\n",
+    "\n",
+    "You've seen how to use BERT and other transformer models for a wide range of natural language tasks, including machine translation, summarization, and question answering. Transformers have become the standard model for NLP, similar to convolutional models in computer vision. And all started with Attention!\n",
+    "\n",
+    "In practice, you'll rarely train a transformer model from scratch.  Transformers tend to be very large, so they take time, money, and lots of data to train fully. Instead, you'll want to start with a pre-trained model and fine-tune it with your dataset if you need to.\n",
+    "\n",
+    "[Hugging Face](https://huggingface.co/) (🤗) is the best resource for pre-trained transformers. Their open-source libraries simplify downloading and using transformer models like BERT, T5, and GPT-2. And the best part, you can use them alongside either TensorFlow, PyTorch or Flax.\n",
+    "\n",
+    "In this notebook, you'll use 🤗  transformers to use the DistilBERT model for question answering."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "tm675LmQvpUm"
+   },
+   "source": [
+    "## Pipelines\n",
+    "\n",
+    "Before fine-tuning a model, you will look at the pipelines from Hugging Face to use pre-trained transformer models for specific tasks. The `transformers` library provides pipelines for popular tasks like sentiment analysis, summarization, and text generation. A pipeline consists of a tokenizer, a model, and the model configuration. All these are packaged together into an easy-to-use object. Hugging Face makes life easier.\n",
+    "\n",
+    "Pipelines are intended to be used without fine-tuning and will often be immediately helpful in your projects. For example, `transformers` provides a pipeline for [question answering](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.QuestionAnsweringPipeline) that you can directly use to answer your questions if you give some context. Let's see how to do just that.\n",
+    "\n",
+    "You will import `pipeline` from `transformers` for creating pipelines."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "uNJGGbRWvpUm"
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
+    "\n",
+    "from transformers import pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_CeFTIr7P3QR"
+   },
+   "source": [
+    "Now, you will create the pipeline for question-answering, which uses the [DistilBert](https://hf.co/distilbert-base-cased-distilled-squad) model for extractive question answering (i.e., answering questions with the exact wording provided in the context)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 177,
+     "referenced_widgets": [
+      "d7e158e614f44983b229d6dd0d8960f9",
+      "69aad11dbc914410b95f6c3cb17a2457",
+      "0302e718c6084fb0a96d92fd976738dc",
+      "72b47b116b0b4125a35d47e060f46807",
+      "6f5fbb8f0f5a4374a7bde870c64f1fa4",
+      "331c23df507e4d679e3aaf81af39cd22",
+      "e33617a01b03437986c143c9a69ba14f",
+      "b1271eb1b7e74250bd9273f229b49cd8",
+      "6b20c9e39d36404fb761b4d83954a278",
+      "322cc6ef697945ccbbc2b3029dfdf0e3",
+      "f9456ff5134242bc9541d9d60c753384",
+      "5b917388b6624637ad8d8f60516d4001",
+      "13e05f2f64a54245a2478393b1f6b409",
+      "3c63301478f54f95ba0f0f8c853a7266",
+      "bb455638d3ac451096fd7cce4cb0d82c",
+      "c4a6b418089147f6b50eab097ace0342",
+      "9f4a770ae6b84593ac7de85e15c305a9",
+      "65d019ca643045a2b0933411d059c920",
+      "794c088a92ba4f6798fa94cded51d0bd",
+      "deb9a0d8d3e1430bacab12c8b4ce7573",
+      "c53ce49d9b7c4a3c87ad7f7c75dce1f5",
+      "b099d3bb966a4e9b8d02710329030ff3",
+      "6d42d468b1d04c4a94fb2eed75e3c238",
+      "a371ed9c75184b78a80facda31086426",
+      "c363fb4238e0464cb3a4ab16250e554a",
+      "86dce8a2c404469dab0dc1a466788c0a",
+      "f6a334e9b5da4c82a1c4f1fbc1fe3c7e",
+      "e9f1a9476de147c3bc98c1c36960dad6",
+      "2f8a7ddfcee64b978ff23f8f43911e01",
+      "099a1ac5af9b4e38abb6afb9c333d37a",
+      "340a7a5171ba4b8aa9b70e945df1618c",
+      "b0f31194b8f24b5ab601f8edd5332e04",
+      "cbeb9e9dfdf6420d91ec1126fedd8e48",
+      "da1553ec3e044a4fb7bb9b0c2a84bfe0",
+      "b440f3b3937549d69a4f188bb8415531",
+      "1b937b91e8ee46c4a764a8091f365291",
+      "1369029047864cc68100a44ffaad35ca",
+      "80cb9869ae694ecda95aab298598c7a2",
+      "2aaa61bf4df248be970940e103af0276",
+      "3a7f8b6302034a0e889976ba0fbb4531",
+      "8cc02da6124448598923899b144afd6d",
+      "7361467d667d49ee85c432eec884d882",
+      "39c0c457e3bd46c1971ae2913fd66429",
+      "2985df1d86904019a8fdf69a356ade6d",
+      "6c291edce30b4cefb329113d5ecbe640",
+      "1249849d670f4824a0a21aa61d187b56",
+      "0e7b7a12422d49d4a33538e638bfc1c9",
+      "ec90909f38e0448bab37c735ea9b9ebe",
+      "f4274a98f06945a6a1e4a56b680c1790",
+      "c04562a21d36405890c11e74915839c7",
+      "e6b91c2208e44524a9883309ae431277",
+      "9adf994d114141f98aeea509a73e9c59",
+      "520e9bf4c0164e60a2c3288bd97ef93e",
+      "dd420d29751341faa84c025afa743bb5",
+      "4d56ed704097453a930baa9ecdfb1156"
+     ]
+    },
+    "id": "nKy4AAhLvpUo",
+    "outputId": "0419ab21-4237-4ad2-b076-9ed83377ed34"
+   },
+   "outputs": [],
+   "source": [
+    "# The task \"question-answering\" will return a QuestionAnsweringPipeline object\n",
+    "question_answerer = pipeline(task=\"question-answering\", model=\"distilbert-base-cased-distilled-squad\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4ltQLVWgvpUo"
+   },
+   "source": [
+    "Notice that this environment already has the model stored in the directory `distilbert-base-cased-distilled-squad`. However if you were to run that exact code on your local computer, Huggingface will download the model for you, which is a great feature!\n",
+    "\n",
+    "\n",
+    "After running the last cell, you have a pipeline for performing question answering given a context string. The pipeline `question_answerer` you just created needs you to pass the question and context as strings. It returns an answer to the question from the context you provided. For example, here are the first few paragraphs from the [Wikipedia entry for tea](https://en.wikipedia.org/wiki/Tea) that you will use as the context.\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "D_-MzZNJvpUp"
+   },
+   "outputs": [],
+   "source": [
+    "context = \"\"\"\n",
+    "Tea is an aromatic beverage prepared by pouring hot or boiling water over cured or fresh leaves of Camellia sinensis,\n",
+    "an evergreen shrub native to China and East Asia. After water, it is the most widely consumed drink in the world.\n",
+    "There are many different types of tea; some, like Chinese greens and Darjeeling, have a cooling, slightly bitter,\n",
+    "and astringent flavour, while others have vastly different profiles that include sweet, nutty, floral, or grassy\n",
+    "notes. Tea has a stimulating effect in humans primarily due to its caffeine content.\n",
+    "\n",
+    "The tea plant originated in the region encompassing today's Southwest China, Tibet, north Myanmar and Northeast India,\n",
+    "where it was used as a medicinal drink by various ethnic groups. An early credible record of tea drinking dates to\n",
+    "the 3rd century AD, in a medical text written by Hua Tuo. It was popularised as a recreational drink during the\n",
+    "Chinese Tang dynasty, and tea drinking spread to other East Asian countries. Portuguese priests and merchants\n",
+    "introduced it to Europe during the 16th century. During the 17th century, drinking tea became fashionable among the\n",
+    "English, who started to plant tea on a large scale in India.\n",
+    "\n",
+    "The term herbal tea refers to drinks not made from Camellia sinensis: infusions of fruit, leaves, or other plant\n",
+    "parts, such as steeps of rosehip, chamomile, or rooibos. These may be called tisanes or herbal infusions to prevent\n",
+    "confusion with 'tea' made from the tea plant.\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "HyR3o2mrvpUq"
+   },
+   "source": [
+    "Now, you can ask your model anything related to that passage. For instance, \"Where is tea native to?\"."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "eiRohAWWvpUq",
+    "outputId": "a1ddfca3-3723-4d43-cbda-0509337b60d6",
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "result = question_answerer(question=\"Where is tea native to?\", context=context)\n",
+    "\n",
+    "print(result['answer'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "cRXzFlZ5vpUr"
+   },
+   "source": [
+    "You can also pass multiple questions to your pipeline within a list so that you can ask:\n",
+    "\n",
+    "*   \"Where is tea native to?\"\n",
+    "*   \"When was tea discovered?\"\n",
+    "*   \"What is the species name for tea?\"\n",
+    "\n",
+    "at the same time, and your `question-answerer` will return all the answers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "IMLyXeMZvpUr",
+    "outputId": "ac9badb1-083d-4234-9474-f112c1f2f20f"
+   },
+   "outputs": [],
+   "source": [
+    "questions = [\"Where is tea native to?\",\n",
+    "             \"When was tea discovered?\",\n",
+    "             \"What is the species name for tea?\"]\n",
+    "\n",
+    "results = question_answerer(question=questions, context=context)\n",
+    "\n",
+    "for q, r in zip(questions, results):\n",
+    "    print(f\"{q} \\n>> {r['answer']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "XXf18tVu8p70"
+   },
+   "source": [
+    "Although the models used in the Hugging Face pipelines generally give outstanding results, sometimes you will have particular examples where they don't perform so well. Let's use the following example with a context string about the Golden Age of Comic Books:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "0v9C0TAqwinw"
+   },
+   "outputs": [],
+   "source": [
+    "context = \"\"\"\n",
+    "The Golden Age of Comic Books describes an era of American comic books from the\n",
+    "late 1930s to circa 1950. During this time, modern comic books were first published\n",
+    "and rapidly increased in popularity. The superhero archetype was created and many\n",
+    "well-known characters were introduced, including Superman, Batman, Captain Marvel\n",
+    "(later known as SHAZAM!), Captain America, and Wonder Woman.\n",
+    "Between 1939 and 1941 Detective Comics and its sister company, All-American Publications,\n",
+    "introduced popular superheroes such as Batman and Robin, Wonder Woman, the Flash,\n",
+    "Green Lantern, Doctor Fate, the Atom, Hawkman, Green Arrow and Aquaman.[7] Timely Comics,\n",
+    "the 1940s predecessor of Marvel Comics, had million-selling titles featuring the Human Torch,\n",
+    "the Sub-Mariner, and Captain America.[8]\n",
+    "As comic books grew in popularity, publishers began launching titles that expanded\n",
+    "into a variety of genres. Dell Comics' non-superhero characters (particularly the\n",
+    "licensed Walt Disney animated-character comics) outsold the superhero comics of the day.[12]\n",
+    "The publisher featured licensed movie and literary characters such as Mickey Mouse, Donald Duck,\n",
+    "Roy Rogers and Tarzan.[13] It was during this era that noted Donald Duck writer-artist\n",
+    "Carl Barks rose to prominence.[14] Additionally, MLJ's introduction of Archie Andrews\n",
+    "in Pep Comics #22 (December 1941) gave rise to teen humor comics,[15] with the Archie\n",
+    "Andrews character remaining in print well into the 21st century.[16]\n",
+    "At the same time in Canada, American comic books were prohibited importation under\n",
+    "the War Exchange Conservation Act[17] which restricted the importation of non-essential\n",
+    "goods. As a result, a domestic publishing industry flourished during the duration\n",
+    "of the war which were collectively informally called the Canadian Whites.\n",
+    "The educational comic book Dagwood Splits the Atom used characters from the comic\n",
+    "strip Blondie.[18] According to historian Michael A. Amundson, appealing comic-book\n",
+    "characters helped ease young readers' fear of nuclear war and neutralize anxiety\n",
+    "about the questions posed by atomic power.[19] It was during this period that long-running\n",
+    "humor comics debuted, including EC's Mad and Carl Barks' Uncle Scrooge in Dell's Four\n",
+    "Color Comics (both in 1952).[20][21]\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "fYbERLKQbhyH"
+   },
+   "source": [
+    "Let's ask the following question: \"What popular superheroes were introduced between 1939 and 1941?\" The answer is in the fourth paragraph of the context string."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "SEmAbSSGbg0J",
+    "outputId": "35b5e3c4-2fd2-4f37-b674-014681ece042"
+   },
+   "outputs": [],
+   "source": [
+    "question = \"What popular superheroes were introduced between 1939 and 1941?\"\n",
+    "\n",
+    "result = question_answerer(question=question, context=context)\n",
+    "print(result['answer'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "LGx_BHkN-ejY"
+   },
+   "source": [
+    "Here, the answer should be:\n",
+    "\"Batman and Robin, Wonder Woman, the Flash,\n",
+    "Green Lantern, Doctor Fate, the Atom, Hawkman, Green Arrow, and Aquaman\". Instead, the pipeline returned a different answer.  You can even try different question wordings:\n",
+    "\n",
+    "*   \"What superheroes were introduced between 1939 and 1941?\"\n",
+    "*   \"What comic book characters were created between 1939 and 1941?\"\n",
+    "*   \"What well-known characters were created between 1939 and 1941?\"\n",
+    "*   \"What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\"\n",
+    "\n",
+    "and you will only get incorrect answers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "f91kLn9VcRzK",
+    "outputId": "bb3942b6-321a-4466-ac18-9f173b115600"
+   },
+   "outputs": [],
+   "source": [
+    "questions = [\"What popular superheroes were introduced between 1939 and 1941?\",\n",
+    "             \"What superheroes were introduced between 1939 and 1941 by Detective Comics and its sister company?\",\n",
+    "             \"What comic book characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\"]\n",
+    "\n",
+    "results = question_answerer(question=questions, context=context)\n",
+    "\n",
+    "for q, r in zip(questions, results):\n",
+    "    print(f\"{q} \\n>> {r['answer']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "QCkLhf27cEsH"
+   },
+   "source": [
+    "It seems like this model is a **huge fan** of Archie Andrews. It even considers him a superhero!\n",
+    "\n",
+    "The example that fooled your `question_answerer` belongs to the [TyDi QA dataset](https://ai.google.com/research/tydiqa), a dataset from Google for question/answering in diverse languages. To achieve better results when you know that the pipeline isn't working as it should, you need to consider fine-tuning your model.\n",
+    "\n",
+    "In the next ungraded lab, you will get the chance to fine-tune the DistilBert model using the TyDi QA dataset.\n",
+    "\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "0302e718c6084fb0a96d92fd976738dc": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_b1271eb1b7e74250bd9273f229b49cd8",
+      "max": 473,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_6b20c9e39d36404fb761b4d83954a278",
+      "value": 473
+     }
+    },
+    "099a1ac5af9b4e38abb6afb9c333d37a": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "0e7b7a12422d49d4a33538e638bfc1c9": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_9adf994d114141f98aeea509a73e9c59",
+      "max": 435797,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_520e9bf4c0164e60a2c3288bd97ef93e",
+      "value": 435797
+     }
+    },
+    "1249849d670f4824a0a21aa61d187b56": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_c04562a21d36405890c11e74915839c7",
+      "placeholder": "​",
+      "style": "IPY_MODEL_e6b91c2208e44524a9883309ae431277",
+      "value": "Downloading: 100%"
+     }
+    },
+    "1369029047864cc68100a44ffaad35ca": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_39c0c457e3bd46c1971ae2913fd66429",
+      "placeholder": "​",
+      "style": "IPY_MODEL_2985df1d86904019a8fdf69a356ade6d",
+      "value": " 213k/213k [00:00<00:00, 170kB/s]"
+     }
+    },
+    "13e05f2f64a54245a2478393b1f6b409": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_9f4a770ae6b84593ac7de85e15c305a9",
+      "placeholder": "​",
+      "style": "IPY_MODEL_65d019ca643045a2b0933411d059c920",
+      "value": "Downloading: 100%"
+     }
+    },
+    "1b937b91e8ee46c4a764a8091f365291": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_8cc02da6124448598923899b144afd6d",
+      "max": 213450,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_7361467d667d49ee85c432eec884d882",
+      "value": 213450
+     }
+    },
+    "2985df1d86904019a8fdf69a356ade6d": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "2aaa61bf4df248be970940e103af0276": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "2f8a7ddfcee64b978ff23f8f43911e01": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "322cc6ef697945ccbbc2b3029dfdf0e3": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "331c23df507e4d679e3aaf81af39cd22": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "340a7a5171ba4b8aa9b70e945df1618c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "39c0c457e3bd46c1971ae2913fd66429": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "3a7f8b6302034a0e889976ba0fbb4531": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "3c63301478f54f95ba0f0f8c853a7266": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_794c088a92ba4f6798fa94cded51d0bd",
+      "max": 260793700,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_deb9a0d8d3e1430bacab12c8b4ce7573",
+      "value": 260793700
+     }
+    },
+    "4d56ed704097453a930baa9ecdfb1156": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "520e9bf4c0164e60a2c3288bd97ef93e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "5b917388b6624637ad8d8f60516d4001": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_13e05f2f64a54245a2478393b1f6b409",
+       "IPY_MODEL_3c63301478f54f95ba0f0f8c853a7266",
+       "IPY_MODEL_bb455638d3ac451096fd7cce4cb0d82c"
+      ],
+      "layout": "IPY_MODEL_c4a6b418089147f6b50eab097ace0342"
+     }
+    },
+    "65d019ca643045a2b0933411d059c920": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "69aad11dbc914410b95f6c3cb17a2457": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_331c23df507e4d679e3aaf81af39cd22",
+      "placeholder": "​",
+      "style": "IPY_MODEL_e33617a01b03437986c143c9a69ba14f",
+      "value": "Downloading: 100%"
+     }
+    },
+    "6b20c9e39d36404fb761b4d83954a278": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "6c291edce30b4cefb329113d5ecbe640": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_1249849d670f4824a0a21aa61d187b56",
+       "IPY_MODEL_0e7b7a12422d49d4a33538e638bfc1c9",
+       "IPY_MODEL_ec90909f38e0448bab37c735ea9b9ebe"
+      ],
+      "layout": "IPY_MODEL_f4274a98f06945a6a1e4a56b680c1790"
+     }
+    },
+    "6d42d468b1d04c4a94fb2eed75e3c238": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_a371ed9c75184b78a80facda31086426",
+       "IPY_MODEL_c363fb4238e0464cb3a4ab16250e554a",
+       "IPY_MODEL_86dce8a2c404469dab0dc1a466788c0a"
+      ],
+      "layout": "IPY_MODEL_f6a334e9b5da4c82a1c4f1fbc1fe3c7e"
+     }
+    },
+    "6f5fbb8f0f5a4374a7bde870c64f1fa4": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "72b47b116b0b4125a35d47e060f46807": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_322cc6ef697945ccbbc2b3029dfdf0e3",
+      "placeholder": "​",
+      "style": "IPY_MODEL_f9456ff5134242bc9541d9d60c753384",
+      "value": " 473/473 [00:00<00:00, 13.5kB/s]"
+     }
+    },
+    "7361467d667d49ee85c432eec884d882": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "794c088a92ba4f6798fa94cded51d0bd": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "80cb9869ae694ecda95aab298598c7a2": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "86dce8a2c404469dab0dc1a466788c0a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_b0f31194b8f24b5ab601f8edd5332e04",
+      "placeholder": "​",
+      "style": "IPY_MODEL_cbeb9e9dfdf6420d91ec1126fedd8e48",
+      "value": " 29.0/29.0 [00:00<00:00, 321B/s]"
+     }
+    },
+    "8cc02da6124448598923899b144afd6d": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "9adf994d114141f98aeea509a73e9c59": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "9f4a770ae6b84593ac7de85e15c305a9": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "a371ed9c75184b78a80facda31086426": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_e9f1a9476de147c3bc98c1c36960dad6",
+      "placeholder": "​",
+      "style": "IPY_MODEL_2f8a7ddfcee64b978ff23f8f43911e01",
+      "value": "Downloading: 100%"
+     }
+    },
+    "b099d3bb966a4e9b8d02710329030ff3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "b0f31194b8f24b5ab601f8edd5332e04": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b1271eb1b7e74250bd9273f229b49cd8": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b440f3b3937549d69a4f188bb8415531": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_2aaa61bf4df248be970940e103af0276",
+      "placeholder": "​",
+      "style": "IPY_MODEL_3a7f8b6302034a0e889976ba0fbb4531",
+      "value": "Downloading: 100%"
+     }
+    },
+    "bb455638d3ac451096fd7cce4cb0d82c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_c53ce49d9b7c4a3c87ad7f7c75dce1f5",
+      "placeholder": "​",
+      "style": "IPY_MODEL_b099d3bb966a4e9b8d02710329030ff3",
+      "value": " 261M/261M [00:04<00:00, 53.4MB/s]"
+     }
+    },
+    "c04562a21d36405890c11e74915839c7": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "c363fb4238e0464cb3a4ab16250e554a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_099a1ac5af9b4e38abb6afb9c333d37a",
+      "max": 29,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_340a7a5171ba4b8aa9b70e945df1618c",
+      "value": 29
+     }
+    },
+    "c4a6b418089147f6b50eab097ace0342": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "c53ce49d9b7c4a3c87ad7f7c75dce1f5": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "cbeb9e9dfdf6420d91ec1126fedd8e48": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "d7e158e614f44983b229d6dd0d8960f9": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_69aad11dbc914410b95f6c3cb17a2457",
+       "IPY_MODEL_0302e718c6084fb0a96d92fd976738dc",
+       "IPY_MODEL_72b47b116b0b4125a35d47e060f46807"
+      ],
+      "layout": "IPY_MODEL_6f5fbb8f0f5a4374a7bde870c64f1fa4"
+     }
+    },
+    "da1553ec3e044a4fb7bb9b0c2a84bfe0": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_b440f3b3937549d69a4f188bb8415531",
+       "IPY_MODEL_1b937b91e8ee46c4a764a8091f365291",
+       "IPY_MODEL_1369029047864cc68100a44ffaad35ca"
+      ],
+      "layout": "IPY_MODEL_80cb9869ae694ecda95aab298598c7a2"
+     }
+    },
+    "dd420d29751341faa84c025afa743bb5": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "deb9a0d8d3e1430bacab12c8b4ce7573": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "e33617a01b03437986c143c9a69ba14f": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "e6b91c2208e44524a9883309ae431277": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "e9f1a9476de147c3bc98c1c36960dad6": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "ec90909f38e0448bab37c735ea9b9ebe": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_dd420d29751341faa84c025afa743bb5",
+      "placeholder": "​",
+      "style": "IPY_MODEL_4d56ed704097453a930baa9ecdfb1156",
+      "value": " 436k/436k [00:01<00:00, 470kB/s]"
+     }
+    },
+    "f4274a98f06945a6a1e4a56b680c1790": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "f6a334e9b5da4c82a1c4f1fbc1fe3c7e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "f9456ff5134242bc9541d9d60c753384": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

NLP with Attention Models/QA/QA_DistilBERT_pipline_FT/Files/tf/.ipynb_checkpoints/C4W3_HF_Lab2_QA_BERT-checkpoint.ipynb

@@ -0,0 +1,644 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "u2UXutvEvpUj"
+   },
+   "source": [
+    "# Question Answering with BERT and HuggingFace 🤗 (Fine-tuning)\n",
+    "\n",
+    "In the previous Hugging Face ungraded lab, you saw how to use the pipeline objects to use transformer models for NLP tasks. In that lab, the model didn't output the desired answers to a series of precise questions for a context related to the history of comic books.\n",
+    "\n",
+    "In this lab, you will fine-tune the model from that lab to give better answers for that type of context. To do that, you'll be using the [TyDi QA dataset](https://ai.google.com/research/tydiqa) but on a filtered version with only English examples. Additionally, you will use a lot of the tools that Hugging Face has to offer.\n",
+    "\n",
+    "You have to note that, in general, you will fine-tune general-purpose transformer models to work for specific tasks. However, fine-tuning a general-purpose model can take a lot of time. That's why you will be using the model from the question answering pipeline in this lab.\n",
+    "\n",
+    "Begin by importing some libraries and/or objects you will use throughout the lab:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
+    "\n",
+    "import numpy as np\n",
+    "\n",
+    "from datasets import load_from_disk\n",
+    "from transformers import AutoTokenizer, AutoModelForQuestionAnswering, Trainer, TrainingArguments\n",
+    "\n",
+    "from sklearn.metrics import f1_score"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "FrEglXPmvpUr"
+   },
+   "source": [
+    "## Fine-tuning a BERT model\n",
+    "\n",
+    "As you saw in the previous lab, you can use these pipelines as they are. But sometimes, you'll need something more specific to your problem, or maybe you need it to perform better on your production data. In these cases, you'll need to fine-tune a model.\n",
+    "\n",
+    "Here, you'll fine-tune a pre-trained DistilBERT model on the TyDi QA dataset.\n",
+    "\n",
+    "To fine-tune your model, you will leverage three components provided by Hugging Face:\n",
+    "\n",
+    "* Datasets: Library that contains some datasets and different metrics to evaluate the performance of your models.\n",
+    "* Tokenizer: Object in charge of preprocessing your text to be given as input for the transformer models.\n",
+    "* Transformers: Library with the pre-trained model checkpoints and the trainer object.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "g0Rg-e4jBFFs"
+   },
+   "source": [
+    "### Datasets\n",
+    "\n",
+    "To get the dataset to fine-tune your model, you will use [🤗 Datasets](https://huggingface.co/docs/datasets/), a lightweight and extensible library to share and access datasets and evaluation metrics for NLP easily. You can download Hugging Face datasets directly using the `load_dataset` function from the `datasets` library. \n",
+    "\n",
+    "Hugging Face `datasets` allows to load data in several formats, such as CSV, JSON, text files and even parquet. You can see more about the supported formats in the [documentation](https://huggingface.co/docs/datasets/loading)\n",
+    "\n",
+    "A common approach is to use `load_dataset` and get the full dataset but **for this lab you will use a filtered version containing only the English examples**, which is already saved in this environment. Since this filtered dataset is saved using the Apache Arrow format, you can read it by using the `load_from_disk` function.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "x68dqaoXg5Ra"
+   },
+   "outputs": [],
+   "source": [
+    "#The path where the dataset is stored\n",
+    "path = './tydiqa_data/'\n",
+    "\n",
+    "#Load Dataset\n",
+    "tydiqa_data = load_from_disk(path)\n",
+    "\n",
+    "tydiqa_data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "1hfzBZU3T47O"
+   },
+   "source": [
+    "<a id='datasets_type'></a>\n",
+    "You can check below that the type of the loaded dataset is a `datasets.arrow_dataset.Dataset`. This object type corresponds to an Apache Arrow Table that allows creating a hash table that contains the position in memory where data is stored instead of loading the complete dataset into memory. But you don't have to worry too much about that. It is just an efficient way to work with lots of data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "gkeppC3GQiW6"
+   },
+   "outputs": [],
+   "source": [
+    "# Checking the object type for one of the elements in the dataset\n",
+    "type(tydiqa_data['train'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "q_HLaNtQaFlR"
+   },
+   "source": [
+    "You can also check the structure of the dataset:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "2l9ANJTrbP-U"
+   },
+   "outputs": [],
+   "source": [
+    "tydiqa_data['train']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "2xRO1yIkvpUt"
+   },
+   "source": [
+    "You can see that each example is like a dictionary object. This dataset consists of questions, contexts, and indices that point to the start and end position of the answer inside the context. You can access the index using the `annotations` key, which is a kind of dictionary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "KNVpW6lADk92"
+   },
+   "outputs": [],
+   "source": [
+    "idx = 600\n",
+    "\n",
+    "# start index\n",
+    "start_index = tydiqa_data['train'][idx]['annotations']['minimal_answers_start_byte'][0]\n",
+    "\n",
+    "# end index\n",
+    "end_index = tydiqa_data['train'][idx]['annotations']['minimal_answers_end_byte'][0]\n",
+    "\n",
+    "print(f\"Question: {tydiqa_data['train'][idx]['question_text']}\")\n",
+    "print(f\"\\nContext (truncated): {tydiqa_data['train'][idx]['document_plaintext'][0:512]} ...\")\n",
+    "print(f\"\\nAnswer: {tydiqa_data['train'][idx]['document_plaintext'][start_index:end_index]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Z-lZgDTEYm74"
+   },
+   "source": [
+    "The question answering model predicts a start and endpoint in the context to extract as the answer. That's why this NLP task is known as extractive question answering.\n",
+    "\n",
+    "To train your model, you need to pass start and endpoints as labels. So, you need to implement a function that extracts the start and end positions from the dataset.\n",
+    "\n",
+    "The dataset contains unanswerable questions. For these, the start and end indices for the answer are equal to `-1`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "Ty_QDcdKYw9a"
+   },
+   "outputs": [],
+   "source": [
+    "tydiqa_data['train'][0]['annotations']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "lHWcNMudcAuO"
+   },
+   "source": [
+    "Now, you have to flatten the dataset to work with an object with a table structure instead of a dictionary structure. This step facilitates the pre-processing steps."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "xDCAQQtoCs_r"
+   },
+   "outputs": [],
+   "source": [
+    "# Flattening the datasets\n",
+    "flattened_train_data = tydiqa_data['train'].flatten()\n",
+    "flattened_test_data =  tydiqa_data['validation'].flatten()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "q5wUa5xED0fK"
+   },
+   "source": [
+    "Also, to make the training more straightforward and faster, we will extract a subset of the train and test datasets. For that purpose, we will use the Hugging Face Dataset object's method called `select()`. This method allows you to take some data points by their index. Here, you will select the first 3000 rows but you can play with the number of data points, however, consider that this will increase the training time."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "BkcIhpEnDHSJ"
+   },
+   "outputs": [],
+   "source": [
+    "# Selecting a subset of the train dataset\n",
+    "flattened_train_data = flattened_train_data.select(range(3000))\n",
+    "\n",
+    "# Selecting a subset of the test dataset\n",
+    "flattened_test_data = flattened_test_data.select(range(1000))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "fBXrmwXhc13M"
+   },
+   "source": [
+    "### Tokenizers\n",
+    "\n",
+    "Now, you will use the [tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html) object from Hugging Face. You can load a tokenizer using different methods. Here, you will retrieve it from the pipeline object you created in the previous Hugging Face lab. With this tokenizer, you can ensure that the tokens you get for the dataset will match the tokens used in the original DistilBERT implementation.\n",
+    "\n",
+    "When loading a tokenizer with any method, you must pass the model checkpoint that you want to fine-tune. Here, you are using the`'distilbert-base-cased-distilled-squad'` checkpoint.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "LInV3b_HyAIF"
+   },
+   "outputs": [],
+   "source": [
+    "# Import the AutoTokenizer from the transformers library\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-cased-distilled-squad\")\n",
+    "\n",
+    "# Define max length of sequences in the tokenizer\n",
+    "tokenizer.model_max_length = 512"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "qz6YtVcOh3qP"
+   },
+   "source": [
+    "Given the characteristics of the dataset and the question-answering task, you will need to add some steps to pre-process the data after the tokenization:\n",
+    "\n",
+    "1. When there is no answer to a question given a context, you will use the `CLS` token, a unique token used to represent the start of the sequence.\n",
+    "\n",
+    "2.  Tokenizers can split a given string into substrings, resulting in a subtoken for each substring, creating misalignment between the list of dataset tags and the labels generated by the tokenizer. Therefore, you will need to align the start and end indices with the tokens associated with the target answer word.\n",
+    "\n",
+    "3. Finally, a tokenizer can truncate a very long sequence. So, if the start/end position of an answer is `None`, you will assume that it was truncated and assign the maximum length of the tokenizer to those positions.\n",
+    "\n",
+    "Those three steps are done within the `process_samples` function defined below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "3l-r4wI06LU7"
+   },
+   "outputs": [],
+   "source": [
+    "# Processing samples using the 3 steps described above\n",
+    "def process_samples(sample):\n",
+    "    tokenized_data = tokenizer(sample['document_plaintext'], sample['question_text'], truncation=\"only_first\", padding=\"max_length\")\n",
+    "\n",
+    "    input_ids = tokenized_data[\"input_ids\"]\n",
+    "\n",
+    "    # We will label impossible answers with the index of the CLS token.\n",
+    "    cls_index = input_ids.index(tokenizer.cls_token_id)\n",
+    "\n",
+    "    # If no answers are given, set the cls_index as answer.\n",
+    "    if sample[\"annotations.minimal_answers_start_byte\"][0] == -1:\n",
+    "        start_position = cls_index\n",
+    "        end_position = cls_index\n",
+    "    else:\n",
+    "        # Start/end character index of the answer in the text.\n",
+    "        gold_text = sample[\"document_plaintext\"][sample['annotations.minimal_answers_start_byte'][0]:sample['annotations.minimal_answers_end_byte'][0]]\n",
+    "        start_char = sample[\"annotations.minimal_answers_start_byte\"][0]\n",
+    "        end_char = sample['annotations.minimal_answers_end_byte'][0] #start_char + len(gold_text)\n",
+    "\n",
+    "        # sometimes answers are off by a character or two – fix this\n",
+    "        if sample['document_plaintext'][start_char-1:end_char-1] == gold_text:\n",
+    "            start_char = start_char - 1\n",
+    "            end_char = end_char - 1     # When the gold label is off by one character\n",
+    "        elif sample['document_plaintext'][start_char-2:end_char-2] == gold_text:\n",
+    "            start_char = start_char - 2\n",
+    "            end_char = end_char - 2     # When the gold label is off by two characters\n",
+    "\n",
+    "        start_token = tokenized_data.char_to_token(start_char)\n",
+    "        end_token = tokenized_data.char_to_token(end_char - 1)\n",
+    "\n",
+    "        # if start position is None, the answer passage has been truncated\n",
+    "        if start_token is None:\n",
+    "            start_token = tokenizer.model_max_length\n",
+    "        if end_token is None:\n",
+    "            end_token = tokenizer.model_max_length\n",
+    "\n",
+    "        start_position = start_token\n",
+    "        end_position = end_token\n",
+    "\n",
+    "    return {'input_ids': tokenized_data['input_ids'],\n",
+    "          'attention_mask': tokenized_data['attention_mask'],\n",
+    "          'start_positions': start_position,\n",
+    "          'end_positions': end_position}\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Q3LAsWSyk_Rm"
+   },
+   "source": [
+    "To apply the `process_samples` function defined above to the whole  dataset, you can use the `map` method as follows:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "rGbYd7QnFetG"
+   },
+   "outputs": [],
+   "source": [
+    "# Tokenizing and processing the flattened dataset\n",
+    "processed_train_data = flattened_train_data.map(process_samples)\n",
+    "processed_test_data = flattened_test_data.map(process_samples)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "wCpPhYKJluMA"
+   },
+   "source": [
+    "# Transformers\n",
+    "\n",
+    "The last component of Hugging Face that is useful for fine-tuning a transformer corresponds to the pre-trained models you can access in multiple ways.\n",
+    "\n",
+    "For this lab, you will use the same model from the question-answering pipeline that you loaded in the previous lab."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "jR3VqjNc1Vb3"
+   },
+   "outputs": [],
+   "source": [
+    "# Import the AutoModelForQuestionAnswering for the pre-trained model. You will only fine tune the head of the model\n",
+    "model = AutoModelForQuestionAnswering.from_pretrained(\"distilbert-base-cased-distilled-squad\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "K29BYtnsm1yH"
+   },
+   "source": [
+    "Now, you can take the necessary columns from the datasets to train/test and return them as Pytorch Tensors."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "0X14G89noLfW"
+   },
+   "outputs": [],
+   "source": [
+    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+    "\n",
+    "processed_train_data.set_format(type='pt', columns=columns_to_return)\n",
+    "processed_test_data.set_format(type='pt', columns=columns_to_return)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "yjoUFWu_nLRq"
+   },
+   "source": [
+    "Here, we give you the F1 score as a metric to evaluate your model's performance. We will use this metric for simplicity, although it is based on the start and end values predicted by the model. If you want to dig deeper on other metrics that can be used for a question and answering task, you can also check [this colab notebook resource](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb) from the Hugging Face team."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "xcW2wPnirsJk"
+   },
+   "outputs": [],
+   "source": [
+    "def compute_f1_metrics(pred):\n",
+    "    start_labels = pred.label_ids[0]\n",
+    "    start_preds = pred.predictions[0].argmax(-1)\n",
+    "    end_labels = pred.label_ids[1]\n",
+    "    end_preds = pred.predictions[1].argmax(-1)\n",
+    "\n",
+    "    f1_start = f1_score(start_labels, start_preds, average='macro')\n",
+    "    f1_end = f1_score(end_labels, end_preds, average='macro')\n",
+    "\n",
+    "    return {\n",
+    "        'f1_start': f1_start,\n",
+    "        'f1_end': f1_end,\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "KuhASU4evpUu"
+   },
+   "source": [
+    "Now, you will use the Hugging Face [Trainer](https://huggingface.co/transformers/main_classes/trainer.html) to fine-tune your model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "background_save": true
+    },
+    "id": "nxyOwf5utXAt"
+   },
+   "outputs": [],
+   "source": [
+    "# Training hyperparameters\n",
+    "training_args = TrainingArguments(\n",
+    "    output_dir='model_results',     # output directory\n",
+    "    overwrite_output_dir=True,\n",
+    "    num_train_epochs=3,              # total number of training epochs\n",
+    "    per_device_train_batch_size=8,   # batch size per device during training\n",
+    "    per_device_eval_batch_size=8,    # batch size for evaluation\n",
+    "    warmup_steps=20,                 # number of warmup steps for learning rate scheduler\n",
+    "    weight_decay=0.01,               # strength of weight decay\n",
+    "    logging_steps=50\n",
+    ")\n",
+    "\n",
+    "# Trainer object\n",
+    "trainer = Trainer(\n",
+    "    model=model,                        # the instantiated 🤗 Transformers model to be trained\n",
+    "    args=training_args,                 # training arguments, defined above\n",
+    "    train_dataset=processed_train_data, # training dataset\n",
+    "    eval_dataset=processed_test_data,   # evaluation dataset\n",
+    "    compute_metrics=compute_f1_metrics\n",
+    ")\n",
+    "\n",
+    "# Training loop\n",
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Ic_wNlBHCRMn"
+   },
+   "source": [
+    "And, in the next cell, you can evaluate the fine-tuned model's performance on the test set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "92N11A076wRA"
+   },
+   "outputs": [],
+   "source": [
+    "trainer.evaluate(processed_test_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_HubPkRbnzh_"
+   },
+   "source": [
+    "### Using your Fine-Tuned Model\n",
+    "\n",
+    "After training and evaluating your fine-tuned model, you can check its results for the same questions from the previous lab.\n",
+    "\n",
+    "For that, you will tell Pytorch to use your GPU or your CPU to run the model. Additionally, you will need to tokenize your input context and questions. Finally, you need to post-process the output results to transform them from tokens to human-readable strings using the `tokenizer`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = r\"\"\"\n",
+    "The Golden Age of Comic Books describes an era of American comic books from the\n",
+    "late 1930s to circa 1950. During this time, modern comic books were first published\n",
+    "and rapidly increased in popularity. The superhero archetype was created and many\n",
+    "well-known characters were introduced, including Superman, Batman, Captain Marvel\n",
+    "(later known as SHAZAM!), Captain America, and Wonder Woman.\n",
+    "Between 1939 and 1941 Detective Comics and its sister company, All-American Publications,\n",
+    "introduced popular superheroes such as Batman and Robin, Wonder Woman, the Flash,\n",
+    "Green Lantern, Doctor Fate, the Atom, Hawkman, Green Arrow and Aquaman.[7] Timely Comics,\n",
+    "the 1940s predecessor of Marvel Comics, had million-selling titles featuring the Human Torch,\n",
+    "the Sub-Mariner, and Captain America.[8]\n",
+    "As comic books grew in popularity, publishers began launching titles that expanded\n",
+    "into a variety of genres. Dell Comics' non-superhero characters (particularly the\n",
+    "licensed Walt Disney animated-character comics) outsold the superhero comics of the day.[12]\n",
+    "The publisher featured licensed movie and literary characters such as Mickey Mouse, Donald Duck,\n",
+    "Roy Rogers and Tarzan.[13] It was during this era that noted Donald Duck writer-artist\n",
+    "Carl Barks rose to prominence.[14] Additionally, MLJ's introduction of Archie Andrews\n",
+    "in Pep Comics #22 (December 1941) gave rise to teen humor comics,[15] with the Archie\n",
+    "Andrews character remaining in print well into the 21st century.[16]\n",
+    "At the same time in Canada, American comic books were prohibited importation under\n",
+    "the War Exchange Conservation Act[17] which restricted the importation of non-essential\n",
+    "goods. As a result, a domestic publishing industry flourished during the duration\n",
+    "of the war which were collectively informally called the Canadian Whites.\n",
+    "The educational comic book Dagwood Splits the Atom used characters from the comic\n",
+    "strip Blondie.[18] According to historian Michael A. Amundson, appealing comic-book\n",
+    "characters helped ease young readers' fear of nuclear war and neutralize anxiety\n",
+    "about the questions posed by atomic power.[19] It was during this period that long-running\n",
+    "humor comics debuted, including EC's Mad and Carl Barks' Uncle Scrooge in Dell's Four\n",
+    "Color Comics (both in 1952).[20][21]\n",
+    "\"\"\"\n",
+    "\n",
+    "questions = [\"What superheroes were introduced between 1939 and 1941 by Detective Comics and its sister company?\",\n",
+    "             \"What comic book characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\"]\n",
+    "\n",
+    "for question in questions:\n",
+    "    inputs = tokenizer.encode_plus(question, text, return_tensors=\"pt\")\n",
+    "\n",
+    "    input_ids = inputs[\"input_ids\"].tolist()[0]\n",
+    "    inputs.to(\"cuda\")\n",
+    "\n",
+    "    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
+    "    answer_model = model(**inputs)\n",
+    "    \n",
+    "    start_logits = answer_model['start_logits'].cpu().detach().numpy()\n",
+    "\n",
+    "    answer_start = np.argmax(start_logits)  \n",
+    "    \n",
+    "    end_logits = answer_model['end_logits'].cpu().detach().numpy()\n",
+    "    \n",
+    "    # Get the most likely beginning of answer with the argmax of the score\n",
+    "    answer_end = np.argmax(end_logits) + 1  # Get the most likely end of answer with the argmax of the score\n",
+    "\n",
+    "    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))\n",
+    "\n",
+    "    print(f\"Question: {question}\")\n",
+    "    print(f\"Answer: {answer}\\n\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_yTDQ6kn6pWS"
+   },
+   "source": [
+    "By fine-tuning the model for only 3 epochs you can already see an improvement!\n",
+    "\n",
+    "You can compare those results with those obtained using the base model (without fine-tuning), as you did in the previous lab. As a reminder, here are those results:\n",
+    "\n",
+    "```\n",
+    "What popular superheroes were introduced between 1939 and 1941?\n",
+    ">> teen humor comics\n",
+    "What superheroes were introduced between 1939 and 1941 by Detective Comics and its sister company?\n",
+    ">> Archie Andrews\n",
+    "What comic book characters were created between 1939 and 1941?\n",
+    ">> Archie\n",
+    "Andrews\n",
+    "What well-known characters were created between 1939 and 1941?\n",
+    ">> Archie\n",
+    "Andrews\n",
+    "What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\n",
+    ">> Archie Andrews\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "uf-v8mUSLqXN"
+   },
+   "source": [
+    "**Congratulations!**\n",
+    "\n",
+    "You have finished this series of ungraded labs. You were able to:\n",
+    "\n",
+    "* Explore the Hugging Face Pipelines, which can be used right out of the bat.\n",
+    "\n",
+    "* Fine-tune a model for the Extractive Question & Answering task.\n",
+    "\n",
+    "We also recommend you go through the free [Hugging Face course](https://huggingface.co/course/chapter1) to explore their ecosystem in more detail and find different ways to use the `transformers` library."
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

NLP with Attention Models/QA/QA_DistilBERT_pipline_FT/Files/tf/C4W3_HF_Lab1_QA_BERT.ipynb

@@ -0,0 +1,2110 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "u2UXutvEvpUj"
+   },
+   "source": [
+    "# Question Answering with BERT and HuggingFace\n",
+    "\n",
+    "You've seen how to use BERT and other transformer models for a wide range of natural language tasks, including machine translation, summarization, and question answering. Transformers have become the standard model for NLP, similar to convolutional models in computer vision. And all started with Attention!\n",
+    "\n",
+    "In practice, you'll rarely train a transformer model from scratch.  Transformers tend to be very large, so they take time, money, and lots of data to train fully. Instead, you'll want to start with a pre-trained model and fine-tune it with your dataset if you need to.\n",
+    "\n",
+    "[Hugging Face](https://huggingface.co/) (🤗) is the best resource for pre-trained transformers. Their open-source libraries simplify downloading and using transformer models like BERT, T5, and GPT-2. And the best part, you can use them alongside either TensorFlow, PyTorch or Flax.\n",
+    "\n",
+    "In this notebook, you'll use 🤗  transformers to use the DistilBERT model for question answering."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "tm675LmQvpUm"
+   },
+   "source": [
+    "## Pipelines\n",
+    "\n",
+    "Before fine-tuning a model, you will look at the pipelines from Hugging Face to use pre-trained transformer models for specific tasks. The `transformers` library provides pipelines for popular tasks like sentiment analysis, summarization, and text generation. A pipeline consists of a tokenizer, a model, and the model configuration. All these are packaged together into an easy-to-use object. Hugging Face makes life easier.\n",
+    "\n",
+    "Pipelines are intended to be used without fine-tuning and will often be immediately helpful in your projects. For example, `transformers` provides a pipeline for [question answering](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.QuestionAnsweringPipeline) that you can directly use to answer your questions if you give some context. Let's see how to do just that.\n",
+    "\n",
+    "You will import `pipeline` from `transformers` for creating pipelines."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "id": "uNJGGbRWvpUm"
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
+    "\n",
+    "from transformers import pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_CeFTIr7P3QR"
+   },
+   "source": [
+    "Now, you will create the pipeline for question-answering, which uses the [DistilBert](https://hf.co/distilbert-base-cased-distilled-squad) model for extractive question answering (i.e., answering questions with the exact wording provided in the context)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 177,
+     "referenced_widgets": [
+      "d7e158e614f44983b229d6dd0d8960f9",
+      "69aad11dbc914410b95f6c3cb17a2457",
+      "0302e718c6084fb0a96d92fd976738dc",
+      "72b47b116b0b4125a35d47e060f46807",
+      "6f5fbb8f0f5a4374a7bde870c64f1fa4",
+      "331c23df507e4d679e3aaf81af39cd22",
+      "e33617a01b03437986c143c9a69ba14f",
+      "b1271eb1b7e74250bd9273f229b49cd8",
+      "6b20c9e39d36404fb761b4d83954a278",
+      "322cc6ef697945ccbbc2b3029dfdf0e3",
+      "f9456ff5134242bc9541d9d60c753384",
+      "5b917388b6624637ad8d8f60516d4001",
+      "13e05f2f64a54245a2478393b1f6b409",
+      "3c63301478f54f95ba0f0f8c853a7266",
+      "bb455638d3ac451096fd7cce4cb0d82c",
+      "c4a6b418089147f6b50eab097ace0342",
+      "9f4a770ae6b84593ac7de85e15c305a9",
+      "65d019ca643045a2b0933411d059c920",
+      "794c088a92ba4f6798fa94cded51d0bd",
+      "deb9a0d8d3e1430bacab12c8b4ce7573",
+      "c53ce49d9b7c4a3c87ad7f7c75dce1f5",
+      "b099d3bb966a4e9b8d02710329030ff3",
+      "6d42d468b1d04c4a94fb2eed75e3c238",
+      "a371ed9c75184b78a80facda31086426",
+      "c363fb4238e0464cb3a4ab16250e554a",
+      "86dce8a2c404469dab0dc1a466788c0a",
+      "f6a334e9b5da4c82a1c4f1fbc1fe3c7e",
+      "e9f1a9476de147c3bc98c1c36960dad6",
+      "2f8a7ddfcee64b978ff23f8f43911e01",
+      "099a1ac5af9b4e38abb6afb9c333d37a",
+      "340a7a5171ba4b8aa9b70e945df1618c",
+      "b0f31194b8f24b5ab601f8edd5332e04",
+      "cbeb9e9dfdf6420d91ec1126fedd8e48",
+      "da1553ec3e044a4fb7bb9b0c2a84bfe0",
+      "b440f3b3937549d69a4f188bb8415531",
+      "1b937b91e8ee46c4a764a8091f365291",
+      "1369029047864cc68100a44ffaad35ca",
+      "80cb9869ae694ecda95aab298598c7a2",
+      "2aaa61bf4df248be970940e103af0276",
+      "3a7f8b6302034a0e889976ba0fbb4531",
+      "8cc02da6124448598923899b144afd6d",
+      "7361467d667d49ee85c432eec884d882",
+      "39c0c457e3bd46c1971ae2913fd66429",
+      "2985df1d86904019a8fdf69a356ade6d",
+      "6c291edce30b4cefb329113d5ecbe640",
+      "1249849d670f4824a0a21aa61d187b56",
+      "0e7b7a12422d49d4a33538e638bfc1c9",
+      "ec90909f38e0448bab37c735ea9b9ebe",
+      "f4274a98f06945a6a1e4a56b680c1790",
+      "c04562a21d36405890c11e74915839c7",
+      "e6b91c2208e44524a9883309ae431277",
+      "9adf994d114141f98aeea509a73e9c59",
+      "520e9bf4c0164e60a2c3288bd97ef93e",
+      "dd420d29751341faa84c025afa743bb5",
+      "4d56ed704097453a930baa9ecdfb1156"
+     ]
+    },
+    "id": "nKy4AAhLvpUo",
+    "outputId": "0419ab21-4237-4ad2-b076-9ed83377ed34"
+   },
+   "outputs": [],
+   "source": [
+    "# The task \"question-answering\" will return a QuestionAnsweringPipeline object\n",
+    "question_answerer = pipeline(task=\"question-answering\", model=\"distilbert-base-cased-distilled-squad\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4ltQLVWgvpUo"
+   },
+   "source": [
+    "Notice that this environment already has the model stored in the directory `distilbert-base-cased-distilled-squad`. However if you were to run that exact code on your local computer, Huggingface will download the model for you, which is a great feature!\n",
+    "\n",
+    "\n",
+    "After running the last cell, you have a pipeline for performing question answering given a context string. The pipeline `question_answerer` you just created needs you to pass the question and context as strings. It returns an answer to the question from the context you provided. For example, here are the first few paragraphs from the [Wikipedia entry for tea](https://en.wikipedia.org/wiki/Tea) that you will use as the context.\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "id": "D_-MzZNJvpUp"
+   },
+   "outputs": [],
+   "source": [
+    "context = \"\"\"\n",
+    "Tea is an aromatic beverage prepared by pouring hot or boiling water over cured or fresh leaves of Camellia sinensis,\n",
+    "an evergreen shrub native to China and East Asia. After water, it is the most widely consumed drink in the world.\n",
+    "There are many different types of tea; some, like Chinese greens and Darjeeling, have a cooling, slightly bitter,\n",
+    "and astringent flavour, while others have vastly different profiles that include sweet, nutty, floral, or grassy\n",
+    "notes. Tea has a stimulating effect in humans primarily due to its caffeine content.\n",
+    "\n",
+    "The tea plant originated in the region encompassing today's Southwest China, Tibet, north Myanmar and Northeast India,\n",
+    "where it was used as a medicinal drink by various ethnic groups. An early credible record of tea drinking dates to\n",
+    "the 3rd century AD, in a medical text written by Hua Tuo. It was popularised as a recreational drink during the\n",
+    "Chinese Tang dynasty, and tea drinking spread to other East Asian countries. Portuguese priests and merchants\n",
+    "introduced it to Europe during the 16th century. During the 17th century, drinking tea became fashionable among the\n",
+    "English, who started to plant tea on a large scale in India.\n",
+    "\n",
+    "The term herbal tea refers to drinks not made from Camellia sinensis: infusions of fruit, leaves, or other plant\n",
+    "parts, such as steeps of rosehip, chamomile, or rooibos. These may be called tisanes or herbal infusions to prevent\n",
+    "confusion with 'tea' made from the tea plant.\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "HyR3o2mrvpUq"
+   },
+   "source": [
+    "Now, you can ask your model anything related to that passage. For instance, \"Where is tea native to?\"."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "eiRohAWWvpUq",
+    "outputId": "a1ddfca3-3723-4d43-cbda-0509337b60d6",
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "result = question_answerer(question=\"Where is tea native to?\", context=context)\n",
+    "\n",
+    "print(result['answer'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "cRXzFlZ5vpUr"
+   },
+   "source": [
+    "You can also pass multiple questions to your pipeline within a list so that you can ask:\n",
+    "\n",
+    "*   \"Where is tea native to?\"\n",
+    "*   \"When was tea discovered?\"\n",
+    "*   \"What is the species name for tea?\"\n",
+    "\n",
+    "at the same time, and your `question-answerer` will return all the answers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "IMLyXeMZvpUr",
+    "outputId": "ac9badb1-083d-4234-9474-f112c1f2f20f"
+   },
+   "outputs": [],
+   "source": [
+    "questions = [\"Where is tea native to?\",\n",
+    "             \"When was tea discovered?\",\n",
+    "             \"What is the species name for tea?\"]\n",
+    "\n",
+    "results = question_answerer(question=questions, context=context)\n",
+    "\n",
+    "for q, r in zip(questions, results):\n",
+    "    print(f\"{q} \\n>> {r['answer']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "XXf18tVu8p70"
+   },
+   "source": [
+    "Although the models used in the Hugging Face pipelines generally give outstanding results, sometimes you will have particular examples where they don't perform so well. Let's use the following example with a context string about the Golden Age of Comic Books:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "id": "0v9C0TAqwinw"
+   },
+   "outputs": [],
+   "source": [
+    "context = \"\"\"\n",
+    "The Golden Age of Comic Books describes an era of American comic books from the\n",
+    "late 1930s to circa 1950. During this time, modern comic books were first published\n",
+    "and rapidly increased in popularity. The superhero archetype was created and many\n",
+    "well-known characters were introduced, including Superman, Batman, Captain Marvel\n",
+    "(later known as SHAZAM!), Captain America, and Wonder Woman.\n",
+    "Between 1939 and 1941 Detective Comics and its sister company, All-American Publications,\n",
+    "introduced popular superheroes such as Batman and Robin, Wonder Woman, the Flash,\n",
+    "Green Lantern, Doctor Fate, the Atom, Hawkman, Green Arrow and Aquaman.[7] Timely Comics,\n",
+    "the 1940s predecessor of Marvel Comics, had million-selling titles featuring the Human Torch,\n",
+    "the Sub-Mariner, and Captain America.[8]\n",
+    "As comic books grew in popularity, publishers began launching titles that expanded\n",
+    "into a variety of genres. Dell Comics' non-superhero characters (particularly the\n",
+    "licensed Walt Disney animated-character comics) outsold the superhero comics of the day.[12]\n",
+    "The publisher featured licensed movie and literary characters such as Mickey Mouse, Donald Duck,\n",
+    "Roy Rogers and Tarzan.[13] It was during this era that noted Donald Duck writer-artist\n",
+    "Carl Barks rose to prominence.[14] Additionally, MLJ's introduction of Archie Andrews\n",
+    "in Pep Comics #22 (December 1941) gave rise to teen humor comics,[15] with the Archie\n",
+    "Andrews character remaining in print well into the 21st century.[16]\n",
+    "At the same time in Canada, American comic books were prohibited importation under\n",
+    "the War Exchange Conservation Act[17] which restricted the importation of non-essential\n",
+    "goods. As a result, a domestic publishing industry flourished during the duration\n",
+    "of the war which were collectively informally called the Canadian Whites.\n",
+    "The educational comic book Dagwood Splits the Atom used characters from the comic\n",
+    "strip Blondie.[18] According to historian Michael A. Amundson, appealing comic-book\n",
+    "characters helped ease young readers' fear of nuclear war and neutralize anxiety\n",
+    "about the questions posed by atomic power.[19] It was during this period that long-running\n",
+    "humor comics debuted, including EC's Mad and Carl Barks' Uncle Scrooge in Dell's Four\n",
+    "Color Comics (both in 1952).[20][21]\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "fYbERLKQbhyH"
+   },
+   "source": [
+    "Let's ask the following question: \"What popular superheroes were introduced between 1939 and 1941?\" The answer is in the fourth paragraph of the context string."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "SEmAbSSGbg0J",
+    "outputId": "35b5e3c4-2fd2-4f37-b674-014681ece042"
+   },
+   "outputs": [],
+   "source": [
+    "question = \"What popular superheroes were introduced between 1939 and 1941?\"\n",
+    "\n",
+    "result = question_answerer(question=question, context=context)\n",
+    "print(result['answer'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "LGx_BHkN-ejY"
+   },
+   "source": [
+    "Here, the answer should be:\n",
+    "\"Batman and Robin, Wonder Woman, the Flash,\n",
+    "Green Lantern, Doctor Fate, the Atom, Hawkman, Green Arrow, and Aquaman\". Instead, the pipeline returned a different answer.  You can even try different question wordings:\n",
+    "\n",
+    "*   \"What superheroes were introduced between 1939 and 1941?\"\n",
+    "*   \"What comic book characters were created between 1939 and 1941?\"\n",
+    "*   \"What well-known characters were created between 1939 and 1941?\"\n",
+    "*   \"What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\"\n",
+    "\n",
+    "and you will only get incorrect answers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "f91kLn9VcRzK",
+    "outputId": "bb3942b6-321a-4466-ac18-9f173b115600"
+   },
+   "outputs": [],
+   "source": [
+    "questions = [\"What popular superheroes were introduced between 1939 and 1941?\",\n",
+    "             \"What superheroes were introduced between 1939 and 1941 by Detective Comics and its sister company?\",\n",
+    "             \"What comic book characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\"]\n",
+    "\n",
+    "results = question_answerer(question=questions, context=context)\n",
+    "\n",
+    "for q, r in zip(questions, results):\n",
+    "    print(f\"{q} \\n>> {r['answer']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "QCkLhf27cEsH"
+   },
+   "source": [
+    "It seems like this model is a **huge fan** of Archie Andrews. It even considers him a superhero!\n",
+    "\n",
+    "The example that fooled your `question_answerer` belongs to the [TyDi QA dataset](https://ai.google.com/research/tydiqa), a dataset from Google for question/answering in diverse languages. To achieve better results when you know that the pipeline isn't working as it should, you need to consider fine-tuning your model.\n",
+    "\n",
+    "In the next ungraded lab, you will get the chance to fine-tune the DistilBert model using the TyDi QA dataset.\n",
+    "\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "0302e718c6084fb0a96d92fd976738dc": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_b1271eb1b7e74250bd9273f229b49cd8",
+      "max": 473,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_6b20c9e39d36404fb761b4d83954a278",
+      "value": 473
+     }
+    },
+    "099a1ac5af9b4e38abb6afb9c333d37a": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "0e7b7a12422d49d4a33538e638bfc1c9": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_9adf994d114141f98aeea509a73e9c59",
+      "max": 435797,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_520e9bf4c0164e60a2c3288bd97ef93e",
+      "value": 435797
+     }
+    },
+    "1249849d670f4824a0a21aa61d187b56": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_c04562a21d36405890c11e74915839c7",
+      "placeholder": "​",
+      "style": "IPY_MODEL_e6b91c2208e44524a9883309ae431277",
+      "value": "Downloading: 100%"
+     }
+    },
+    "1369029047864cc68100a44ffaad35ca": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_39c0c457e3bd46c1971ae2913fd66429",
+      "placeholder": "​",
+      "style": "IPY_MODEL_2985df1d86904019a8fdf69a356ade6d",
+      "value": " 213k/213k [00:00<00:00, 170kB/s]"
+     }
+    },
+    "13e05f2f64a54245a2478393b1f6b409": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_9f4a770ae6b84593ac7de85e15c305a9",
+      "placeholder": "​",
+      "style": "IPY_MODEL_65d019ca643045a2b0933411d059c920",
+      "value": "Downloading: 100%"
+     }
+    },
+    "1b937b91e8ee46c4a764a8091f365291": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_8cc02da6124448598923899b144afd6d",
+      "max": 213450,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_7361467d667d49ee85c432eec884d882",
+      "value": 213450
+     }
+    },
+    "2985df1d86904019a8fdf69a356ade6d": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "2aaa61bf4df248be970940e103af0276": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "2f8a7ddfcee64b978ff23f8f43911e01": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "322cc6ef697945ccbbc2b3029dfdf0e3": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "331c23df507e4d679e3aaf81af39cd22": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "340a7a5171ba4b8aa9b70e945df1618c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "39c0c457e3bd46c1971ae2913fd66429": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "3a7f8b6302034a0e889976ba0fbb4531": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "3c63301478f54f95ba0f0f8c853a7266": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_794c088a92ba4f6798fa94cded51d0bd",
+      "max": 260793700,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_deb9a0d8d3e1430bacab12c8b4ce7573",
+      "value": 260793700
+     }
+    },
+    "4d56ed704097453a930baa9ecdfb1156": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "520e9bf4c0164e60a2c3288bd97ef93e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "5b917388b6624637ad8d8f60516d4001": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_13e05f2f64a54245a2478393b1f6b409",
+       "IPY_MODEL_3c63301478f54f95ba0f0f8c853a7266",
+       "IPY_MODEL_bb455638d3ac451096fd7cce4cb0d82c"
+      ],
+      "layout": "IPY_MODEL_c4a6b418089147f6b50eab097ace0342"
+     }
+    },
+    "65d019ca643045a2b0933411d059c920": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "69aad11dbc914410b95f6c3cb17a2457": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_331c23df507e4d679e3aaf81af39cd22",
+      "placeholder": "​",
+      "style": "IPY_MODEL_e33617a01b03437986c143c9a69ba14f",
+      "value": "Downloading: 100%"
+     }
+    },
+    "6b20c9e39d36404fb761b4d83954a278": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "6c291edce30b4cefb329113d5ecbe640": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_1249849d670f4824a0a21aa61d187b56",
+       "IPY_MODEL_0e7b7a12422d49d4a33538e638bfc1c9",
+       "IPY_MODEL_ec90909f38e0448bab37c735ea9b9ebe"
+      ],
+      "layout": "IPY_MODEL_f4274a98f06945a6a1e4a56b680c1790"
+     }
+    },
+    "6d42d468b1d04c4a94fb2eed75e3c238": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_a371ed9c75184b78a80facda31086426",
+       "IPY_MODEL_c363fb4238e0464cb3a4ab16250e554a",
+       "IPY_MODEL_86dce8a2c404469dab0dc1a466788c0a"
+      ],
+      "layout": "IPY_MODEL_f6a334e9b5da4c82a1c4f1fbc1fe3c7e"
+     }
+    },
+    "6f5fbb8f0f5a4374a7bde870c64f1fa4": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "72b47b116b0b4125a35d47e060f46807": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_322cc6ef697945ccbbc2b3029dfdf0e3",
+      "placeholder": "​",
+      "style": "IPY_MODEL_f9456ff5134242bc9541d9d60c753384",
+      "value": " 473/473 [00:00<00:00, 13.5kB/s]"
+     }
+    },
+    "7361467d667d49ee85c432eec884d882": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "794c088a92ba4f6798fa94cded51d0bd": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "80cb9869ae694ecda95aab298598c7a2": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "86dce8a2c404469dab0dc1a466788c0a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_b0f31194b8f24b5ab601f8edd5332e04",
+      "placeholder": "​",
+      "style": "IPY_MODEL_cbeb9e9dfdf6420d91ec1126fedd8e48",
+      "value": " 29.0/29.0 [00:00<00:00, 321B/s]"
+     }
+    },
+    "8cc02da6124448598923899b144afd6d": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "9adf994d114141f98aeea509a73e9c59": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "9f4a770ae6b84593ac7de85e15c305a9": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "a371ed9c75184b78a80facda31086426": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_e9f1a9476de147c3bc98c1c36960dad6",
+      "placeholder": "​",
+      "style": "IPY_MODEL_2f8a7ddfcee64b978ff23f8f43911e01",
+      "value": "Downloading: 100%"
+     }
+    },
+    "b099d3bb966a4e9b8d02710329030ff3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "b0f31194b8f24b5ab601f8edd5332e04": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b1271eb1b7e74250bd9273f229b49cd8": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b440f3b3937549d69a4f188bb8415531": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_2aaa61bf4df248be970940e103af0276",
+      "placeholder": "​",
+      "style": "IPY_MODEL_3a7f8b6302034a0e889976ba0fbb4531",
+      "value": "Downloading: 100%"
+     }
+    },
+    "bb455638d3ac451096fd7cce4cb0d82c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_c53ce49d9b7c4a3c87ad7f7c75dce1f5",
+      "placeholder": "​",
+      "style": "IPY_MODEL_b099d3bb966a4e9b8d02710329030ff3",
+      "value": " 261M/261M [00:04<00:00, 53.4MB/s]"
+     }
+    },
+    "c04562a21d36405890c11e74915839c7": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "c363fb4238e0464cb3a4ab16250e554a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_099a1ac5af9b4e38abb6afb9c333d37a",
+      "max": 29,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_340a7a5171ba4b8aa9b70e945df1618c",
+      "value": 29
+     }
+    },
+    "c4a6b418089147f6b50eab097ace0342": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "c53ce49d9b7c4a3c87ad7f7c75dce1f5": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "cbeb9e9dfdf6420d91ec1126fedd8e48": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "d7e158e614f44983b229d6dd0d8960f9": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_69aad11dbc914410b95f6c3cb17a2457",
+       "IPY_MODEL_0302e718c6084fb0a96d92fd976738dc",
+       "IPY_MODEL_72b47b116b0b4125a35d47e060f46807"
+      ],
+      "layout": "IPY_MODEL_6f5fbb8f0f5a4374a7bde870c64f1fa4"
+     }
+    },
+    "da1553ec3e044a4fb7bb9b0c2a84bfe0": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_b440f3b3937549d69a4f188bb8415531",
+       "IPY_MODEL_1b937b91e8ee46c4a764a8091f365291",
+       "IPY_MODEL_1369029047864cc68100a44ffaad35ca"
+      ],
+      "layout": "IPY_MODEL_80cb9869ae694ecda95aab298598c7a2"
+     }
+    },
+    "dd420d29751341faa84c025afa743bb5": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "deb9a0d8d3e1430bacab12c8b4ce7573": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "e33617a01b03437986c143c9a69ba14f": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "e6b91c2208e44524a9883309ae431277": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "e9f1a9476de147c3bc98c1c36960dad6": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "ec90909f38e0448bab37c735ea9b9ebe": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_dd420d29751341faa84c025afa743bb5",
+      "placeholder": "​",
+      "style": "IPY_MODEL_4d56ed704097453a930baa9ecdfb1156",
+      "value": " 436k/436k [00:01<00:00, 470kB/s]"
+     }
+    },
+    "f4274a98f06945a6a1e4a56b680c1790": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "f6a334e9b5da4c82a1c4f1fbc1fe3c7e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "f9456ff5134242bc9541d9d60c753384": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

NLP with Attention Models/QA/QA_DistilBERT_pipline_FT/Files/tf/C4W3_HF_Lab2_QA_BERT.ipynb

@@ -0,0 +1,644 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "u2UXutvEvpUj"
+   },
+   "source": [
+    "# Question Answering with BERT and HuggingFace 🤗 (Fine-tuning)\n",
+    "\n",
+    "In the previous Hugging Face ungraded lab, you saw how to use the pipeline objects to use transformer models for NLP tasks. In that lab, the model didn't output the desired answers to a series of precise questions for a context related to the history of comic books.\n",
+    "\n",
+    "In this lab, you will fine-tune the model from that lab to give better answers for that type of context. To do that, you'll be using the [TyDi QA dataset](https://ai.google.com/research/tydiqa) but on a filtered version with only English examples. Additionally, you will use a lot of the tools that Hugging Face has to offer.\n",
+    "\n",
+    "You have to note that, in general, you will fine-tune general-purpose transformer models to work for specific tasks. However, fine-tuning a general-purpose model can take a lot of time. That's why you will be using the model from the question answering pipeline in this lab.\n",
+    "\n",
+    "Begin by importing some libraries and/or objects you will use throughout the lab:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
+    "\n",
+    "import numpy as np\n",
+    "\n",
+    "from datasets import load_from_disk\n",
+    "from transformers import AutoTokenizer, AutoModelForQuestionAnswering, Trainer, TrainingArguments\n",
+    "\n",
+    "from sklearn.metrics import f1_score"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "FrEglXPmvpUr"
+   },
+   "source": [
+    "## Fine-tuning a BERT model\n",
+    "\n",
+    "As you saw in the previous lab, you can use these pipelines as they are. But sometimes, you'll need something more specific to your problem, or maybe you need it to perform better on your production data. In these cases, you'll need to fine-tune a model.\n",
+    "\n",
+    "Here, you'll fine-tune a pre-trained DistilBERT model on the TyDi QA dataset.\n",
+    "\n",
+    "To fine-tune your model, you will leverage three components provided by Hugging Face:\n",
+    "\n",
+    "* Datasets: Library that contains some datasets and different metrics to evaluate the performance of your models.\n",
+    "* Tokenizer: Object in charge of preprocessing your text to be given as input for the transformer models.\n",
+    "* Transformers: Library with the pre-trained model checkpoints and the trainer object.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "g0Rg-e4jBFFs"
+   },
+   "source": [
+    "### Datasets\n",
+    "\n",
+    "To get the dataset to fine-tune your model, you will use [🤗 Datasets](https://huggingface.co/docs/datasets/), a lightweight and extensible library to share and access datasets and evaluation metrics for NLP easily. You can download Hugging Face datasets directly using the `load_dataset` function from the `datasets` library. \n",
+    "\n",
+    "Hugging Face `datasets` allows to load data in several formats, such as CSV, JSON, text files and even parquet. You can see more about the supported formats in the [documentation](https://huggingface.co/docs/datasets/loading)\n",
+    "\n",
+    "A common approach is to use `load_dataset` and get the full dataset but **for this lab you will use a filtered version containing only the English examples**, which is already saved in this environment. Since this filtered dataset is saved using the Apache Arrow format, you can read it by using the `load_from_disk` function.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "x68dqaoXg5Ra"
+   },
+   "outputs": [],
+   "source": [
+    "#The path where the dataset is stored\n",
+    "path = './tydiqa_data/'\n",
+    "\n",
+    "#Load Dataset\n",
+    "tydiqa_data = load_from_disk(path)\n",
+    "\n",
+    "tydiqa_data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "1hfzBZU3T47O"
+   },
+   "source": [
+    "<a id='datasets_type'></a>\n",
+    "You can check below that the type of the loaded dataset is a `datasets.arrow_dataset.Dataset`. This object type corresponds to an Apache Arrow Table that allows creating a hash table that contains the position in memory where data is stored instead of loading the complete dataset into memory. But you don't have to worry too much about that. It is just an efficient way to work with lots of data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "gkeppC3GQiW6"
+   },
+   "outputs": [],
+   "source": [
+    "# Checking the object type for one of the elements in the dataset\n",
+    "type(tydiqa_data['train'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "q_HLaNtQaFlR"
+   },
+   "source": [
+    "You can also check the structure of the dataset:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "2l9ANJTrbP-U"
+   },
+   "outputs": [],
+   "source": [
+    "tydiqa_data['train']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "2xRO1yIkvpUt"
+   },
+   "source": [
+    "You can see that each example is like a dictionary object. This dataset consists of questions, contexts, and indices that point to the start and end position of the answer inside the context. You can access the index using the `annotations` key, which is a kind of dictionary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "KNVpW6lADk92"
+   },
+   "outputs": [],
+   "source": [
+    "idx = 600\n",
+    "\n",
+    "# start index\n",
+    "start_index = tydiqa_data['train'][idx]['annotations']['minimal_answers_start_byte'][0]\n",
+    "\n",
+    "# end index\n",
+    "end_index = tydiqa_data['train'][idx]['annotations']['minimal_answers_end_byte'][0]\n",
+    "\n",
+    "print(f\"Question: {tydiqa_data['train'][idx]['question_text']}\")\n",
+    "print(f\"\\nContext (truncated): {tydiqa_data['train'][idx]['document_plaintext'][0:512]} ...\")\n",
+    "print(f\"\\nAnswer: {tydiqa_data['train'][idx]['document_plaintext'][start_index:end_index]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Z-lZgDTEYm74"
+   },
+   "source": [
+    "The question answering model predicts a start and endpoint in the context to extract as the answer. That's why this NLP task is known as extractive question answering.\n",
+    "\n",
+    "To train your model, you need to pass start and endpoints as labels. So, you need to implement a function that extracts the start and end positions from the dataset.\n",
+    "\n",
+    "The dataset contains unanswerable questions. For these, the start and end indices for the answer are equal to `-1`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "Ty_QDcdKYw9a"
+   },
+   "outputs": [],
+   "source": [
+    "tydiqa_data['train'][0]['annotations']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "lHWcNMudcAuO"
+   },
+   "source": [
+    "Now, you have to flatten the dataset to work with an object with a table structure instead of a dictionary structure. This step facilitates the pre-processing steps."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "xDCAQQtoCs_r"
+   },
+   "outputs": [],
+   "source": [
+    "# Flattening the datasets\n",
+    "flattened_train_data = tydiqa_data['train'].flatten()\n",
+    "flattened_test_data =  tydiqa_data['validation'].flatten()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "q5wUa5xED0fK"
+   },
+   "source": [
+    "Also, to make the training more straightforward and faster, we will extract a subset of the train and test datasets. For that purpose, we will use the Hugging Face Dataset object's method called `select()`. This method allows you to take some data points by their index. Here, you will select the first 3000 rows but you can play with the number of data points, however, consider that this will increase the training time."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "BkcIhpEnDHSJ"
+   },
+   "outputs": [],
+   "source": [
+    "# Selecting a subset of the train dataset\n",
+    "flattened_train_data = flattened_train_data.select(range(3000))\n",
+    "\n",
+    "# Selecting a subset of the test dataset\n",
+    "flattened_test_data = flattened_test_data.select(range(1000))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "fBXrmwXhc13M"
+   },
+   "source": [
+    "### Tokenizers\n",
+    "\n",
+    "Now, you will use the [tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html) object from Hugging Face. You can load a tokenizer using different methods. Here, you will retrieve it from the pipeline object you created in the previous Hugging Face lab. With this tokenizer, you can ensure that the tokens you get for the dataset will match the tokens used in the original DistilBERT implementation.\n",
+    "\n",
+    "When loading a tokenizer with any method, you must pass the model checkpoint that you want to fine-tune. Here, you are using the`'distilbert-base-cased-distilled-squad'` checkpoint.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "LInV3b_HyAIF"
+   },
+   "outputs": [],
+   "source": [
+    "# Import the AutoTokenizer from the transformers library\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-cased-distilled-squad\")\n",
+    "\n",
+    "# Define max length of sequences in the tokenizer\n",
+    "tokenizer.model_max_length = 512"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "qz6YtVcOh3qP"
+   },
+   "source": [
+    "Given the characteristics of the dataset and the question-answering task, you will need to add some steps to pre-process the data after the tokenization:\n",
+    "\n",
+    "1. When there is no answer to a question given a context, you will use the `CLS` token, a unique token used to represent the start of the sequence.\n",
+    "\n",
+    "2.  Tokenizers can split a given string into substrings, resulting in a subtoken for each substring, creating misalignment between the list of dataset tags and the labels generated by the tokenizer. Therefore, you will need to align the start and end indices with the tokens associated with the target answer word.\n",
+    "\n",
+    "3. Finally, a tokenizer can truncate a very long sequence. So, if the start/end position of an answer is `None`, you will assume that it was truncated and assign the maximum length of the tokenizer to those positions.\n",
+    "\n",
+    "Those three steps are done within the `process_samples` function defined below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "3l-r4wI06LU7"
+   },
+   "outputs": [],
+   "source": [
+    "# Processing samples using the 3 steps described above\n",
+    "def process_samples(sample):\n",
+    "    tokenized_data = tokenizer(sample['document_plaintext'], sample['question_text'], truncation=\"only_first\", padding=\"max_length\")\n",
+    "\n",
+    "    input_ids = tokenized_data[\"input_ids\"]\n",
+    "\n",
+    "    # We will label impossible answers with the index of the CLS token.\n",
+    "    cls_index = input_ids.index(tokenizer.cls_token_id)\n",
+    "\n",
+    "    # If no answers are given, set the cls_index as answer.\n",
+    "    if sample[\"annotations.minimal_answers_start_byte\"][0] == -1:\n",
+    "        start_position = cls_index\n",
+    "        end_position = cls_index\n",
+    "    else:\n",
+    "        # Start/end character index of the answer in the text.\n",
+    "        gold_text = sample[\"document_plaintext\"][sample['annotations.minimal_answers_start_byte'][0]:sample['annotations.minimal_answers_end_byte'][0]]\n",
+    "        start_char = sample[\"annotations.minimal_answers_start_byte\"][0]\n",
+    "        end_char = sample['annotations.minimal_answers_end_byte'][0] #start_char + len(gold_text)\n",
+    "\n",
+    "        # sometimes answers are off by a character or two – fix this\n",
+    "        if sample['document_plaintext'][start_char-1:end_char-1] == gold_text:\n",
+    "            start_char = start_char - 1\n",
+    "            end_char = end_char - 1     # When the gold label is off by one character\n",
+    "        elif sample['document_plaintext'][start_char-2:end_char-2] == gold_text:\n",
+    "            start_char = start_char - 2\n",
+    "            end_char = end_char - 2     # When the gold label is off by two characters\n",
+    "\n",
+    "        start_token = tokenized_data.char_to_token(start_char)\n",
+    "        end_token = tokenized_data.char_to_token(end_char - 1)\n",
+    "\n",
+    "        # if start position is None, the answer passage has been truncated\n",
+    "        if start_token is None:\n",
+    "            start_token = tokenizer.model_max_length\n",
+    "        if end_token is None:\n",
+    "            end_token = tokenizer.model_max_length\n",
+    "\n",
+    "        start_position = start_token\n",
+    "        end_position = end_token\n",
+    "\n",
+    "    return {'input_ids': tokenized_data['input_ids'],\n",
+    "          'attention_mask': tokenized_data['attention_mask'],\n",
+    "          'start_positions': start_position,\n",
+    "          'end_positions': end_position}\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Q3LAsWSyk_Rm"
+   },
+   "source": [
+    "To apply the `process_samples` function defined above to the whole  dataset, you can use the `map` method as follows:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "rGbYd7QnFetG"
+   },
+   "outputs": [],
+   "source": [
+    "# Tokenizing and processing the flattened dataset\n",
+    "processed_train_data = flattened_train_data.map(process_samples)\n",
+    "processed_test_data = flattened_test_data.map(process_samples)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "wCpPhYKJluMA"
+   },
+   "source": [
+    "# Transformers\n",
+    "\n",
+    "The last component of Hugging Face that is useful for fine-tuning a transformer corresponds to the pre-trained models you can access in multiple ways.\n",
+    "\n",
+    "For this lab, you will use the same model from the question-answering pipeline that you loaded in the previous lab."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "jR3VqjNc1Vb3"
+   },
+   "outputs": [],
+   "source": [
+    "# Import the AutoModelForQuestionAnswering for the pre-trained model. You will only fine tune the head of the model\n",
+    "model = AutoModelForQuestionAnswering.from_pretrained(\"distilbert-base-cased-distilled-squad\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "K29BYtnsm1yH"
+   },
+   "source": [
+    "Now, you can take the necessary columns from the datasets to train/test and return them as Pytorch Tensors."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "0X14G89noLfW"
+   },
+   "outputs": [],
+   "source": [
+    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+    "\n",
+    "processed_train_data.set_format(type='pt', columns=columns_to_return)\n",
+    "processed_test_data.set_format(type='pt', columns=columns_to_return)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "yjoUFWu_nLRq"
+   },
+   "source": [
+    "Here, we give you the F1 score as a metric to evaluate your model's performance. We will use this metric for simplicity, although it is based on the start and end values predicted by the model. If you want to dig deeper on other metrics that can be used for a question and answering task, you can also check [this colab notebook resource](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb) from the Hugging Face team."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "xcW2wPnirsJk"
+   },
+   "outputs": [],
+   "source": [
+    "def compute_f1_metrics(pred):\n",
+    "    start_labels = pred.label_ids[0]\n",
+    "    start_preds = pred.predictions[0].argmax(-1)\n",
+    "    end_labels = pred.label_ids[1]\n",
+    "    end_preds = pred.predictions[1].argmax(-1)\n",
+    "\n",
+    "    f1_start = f1_score(start_labels, start_preds, average='macro')\n",
+    "    f1_end = f1_score(end_labels, end_preds, average='macro')\n",
+    "\n",
+    "    return {\n",
+    "        'f1_start': f1_start,\n",
+    "        'f1_end': f1_end,\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "KuhASU4evpUu"
+   },
+   "source": [
+    "Now, you will use the Hugging Face [Trainer](https://huggingface.co/transformers/main_classes/trainer.html) to fine-tune your model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "background_save": true
+    },
+    "id": "nxyOwf5utXAt"
+   },
+   "outputs": [],
+   "source": [
+    "# Training hyperparameters\n",
+    "training_args = TrainingArguments(\n",
+    "    output_dir='model_results',     # output directory\n",
+    "    overwrite_output_dir=True,\n",
+    "    num_train_epochs=3,              # total number of training epochs\n",
+    "    per_device_train_batch_size=8,   # batch size per device during training\n",
+    "    per_device_eval_batch_size=8,    # batch size for evaluation\n",
+    "    warmup_steps=20,                 # number of warmup steps for learning rate scheduler\n",
+    "    weight_decay=0.01,               # strength of weight decay\n",
+    "    logging_steps=50\n",
+    ")\n",
+    "\n",
+    "# Trainer object\n",
+    "trainer = Trainer(\n",
+    "    model=model,                        # the instantiated 🤗 Transformers model to be trained\n",
+    "    args=training_args,                 # training arguments, defined above\n",
+    "    train_dataset=processed_train_data, # training dataset\n",
+    "    eval_dataset=processed_test_data,   # evaluation dataset\n",
+    "    compute_metrics=compute_f1_metrics\n",
+    ")\n",
+    "\n",
+    "# Training loop\n",
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Ic_wNlBHCRMn"
+   },
+   "source": [
+    "And, in the next cell, you can evaluate the fine-tuned model's performance on the test set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "92N11A076wRA"
+   },
+   "outputs": [],
+   "source": [
+    "trainer.evaluate(processed_test_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_HubPkRbnzh_"
+   },
+   "source": [
+    "### Using your Fine-Tuned Model\n",
+    "\n",
+    "After training and evaluating your fine-tuned model, you can check its results for the same questions from the previous lab.\n",
+    "\n",
+    "For that, you will tell Pytorch to use your GPU or your CPU to run the model. Additionally, you will need to tokenize your input context and questions. Finally, you need to post-process the output results to transform them from tokens to human-readable strings using the `tokenizer`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = r\"\"\"\n",
+    "The Golden Age of Comic Books describes an era of American comic books from the\n",
+    "late 1930s to circa 1950. During this time, modern comic books were first published\n",
+    "and rapidly increased in popularity. The superhero archetype was created and many\n",
+    "well-known characters were introduced, including Superman, Batman, Captain Marvel\n",
+    "(later known as SHAZAM!), Captain America, and Wonder Woman.\n",
+    "Between 1939 and 1941 Detective Comics and its sister company, All-American Publications,\n",
+    "introduced popular superheroes such as Batman and Robin, Wonder Woman, the Flash,\n",
+    "Green Lantern, Doctor Fate, the Atom, Hawkman, Green Arrow and Aquaman.[7] Timely Comics,\n",
+    "the 1940s predecessor of Marvel Comics, had million-selling titles featuring the Human Torch,\n",
+    "the Sub-Mariner, and Captain America.[8]\n",
+    "As comic books grew in popularity, publishers began launching titles that expanded\n",
+    "into a variety of genres. Dell Comics' non-superhero characters (particularly the\n",
+    "licensed Walt Disney animated-character comics) outsold the superhero comics of the day.[12]\n",
+    "The publisher featured licensed movie and literary characters such as Mickey Mouse, Donald Duck,\n",
+    "Roy Rogers and Tarzan.[13] It was during this era that noted Donald Duck writer-artist\n",
+    "Carl Barks rose to prominence.[14] Additionally, MLJ's introduction of Archie Andrews\n",
+    "in Pep Comics #22 (December 1941) gave rise to teen humor comics,[15] with the Archie\n",
+    "Andrews character remaining in print well into the 21st century.[16]\n",
+    "At the same time in Canada, American comic books were prohibited importation under\n",
+    "the War Exchange Conservation Act[17] which restricted the importation of non-essential\n",
+    "goods. As a result, a domestic publishing industry flourished during the duration\n",
+    "of the war which were collectively informally called the Canadian Whites.\n",
+    "The educational comic book Dagwood Splits the Atom used characters from the comic\n",
+    "strip Blondie.[18] According to historian Michael A. Amundson, appealing comic-book\n",
+    "characters helped ease young readers' fear of nuclear war and neutralize anxiety\n",
+    "about the questions posed by atomic power.[19] It was during this period that long-running\n",
+    "humor comics debuted, including EC's Mad and Carl Barks' Uncle Scrooge in Dell's Four\n",
+    "Color Comics (both in 1952).[20][21]\n",
+    "\"\"\"\n",
+    "\n",
+    "questions = [\"What superheroes were introduced between 1939 and 1941 by Detective Comics and its sister company?\",\n",
+    "             \"What comic book characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known characters were created between 1939 and 1941?\",\n",
+    "             \"What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\"]\n",
+    "\n",
+    "for question in questions:\n",
+    "    inputs = tokenizer.encode_plus(question, text, return_tensors=\"pt\")\n",
+    "\n",
+    "    input_ids = inputs[\"input_ids\"].tolist()[0]\n",
+    "    inputs.to(\"cuda\")\n",
+    "\n",
+    "    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)\n",
+    "    answer_model = model(**inputs)\n",
+    "    \n",
+    "    start_logits = answer_model['start_logits'].cpu().detach().numpy()\n",
+    "\n",
+    "    answer_start = np.argmax(start_logits)  \n",
+    "    \n",
+    "    end_logits = answer_model['end_logits'].cpu().detach().numpy()\n",
+    "    \n",
+    "    # Get the most likely beginning of answer with the argmax of the score\n",
+    "    answer_end = np.argmax(end_logits) + 1  # Get the most likely end of answer with the argmax of the score\n",
+    "\n",
+    "    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))\n",
+    "\n",
+    "    print(f\"Question: {question}\")\n",
+    "    print(f\"Answer: {answer}\\n\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_yTDQ6kn6pWS"
+   },
+   "source": [
+    "By fine-tuning the model for only 3 epochs you can already see an improvement!\n",
+    "\n",
+    "You can compare those results with those obtained using the base model (without fine-tuning), as you did in the previous lab. As a reminder, here are those results:\n",
+    "\n",
+    "```\n",
+    "What popular superheroes were introduced between 1939 and 1941?\n",
+    ">> teen humor comics\n",
+    "What superheroes were introduced between 1939 and 1941 by Detective Comics and its sister company?\n",
+    ">> Archie Andrews\n",
+    "What comic book characters were created between 1939 and 1941?\n",
+    ">> Archie\n",
+    "Andrews\n",
+    "What well-known characters were created between 1939 and 1941?\n",
+    ">> Archie\n",
+    "Andrews\n",
+    "What well-known superheroes were introduced between 1939 and 1941 by Detective Comics?\n",
+    ">> Archie Andrews\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "uf-v8mUSLqXN"
+   },
+   "source": [
+    "**Congratulations!**\n",
+    "\n",
+    "You have finished this series of ungraded labs. You were able to:\n",
+    "\n",
+    "* Explore the Hugging Face Pipelines, which can be used right out of the bat.\n",
+    "\n",
+    "* Fine-tune a model for the Extractive Question & Answering task.\n",
+    "\n",
+    "We also recommend you go through the free [Hugging Face course](https://huggingface.co/course/chapter1) to explore their ecosystem in more detail and find different ways to use the `transformers` library."
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}