{ "cells": [ { "cell_type": "markdown", "id": "05014ac7", "metadata": { "colab_type": "text", "id": "7yuytuIllsv1" }, "source": [ "\n", "# Assignment 2: Transformer Summarizer\n", "\n", "Welcome to the second assignment of course 4. In this assignment you will explore summarization using the transformer model. **Unlike the lecture, you will be implementing an encoder-decoder model. However, don't worry; you will be guided through all the steps, and you will find numerous hints to assist you!**\n", "\n", "There are many hints in this notebook so feel free to use them as needed. Actually by the end of this notebook you will have implemented the full transformer (both encoder and decoder) but you will only be graded on the implementation of the decoder as the encoder is provided for you.\n" ] }, { "cell_type": "markdown", "id": "d00e9709", "metadata": { "colab_type": "text", "id": "4-3lxSnXRWPx" }, "source": [ "## Table of Contents\n", "\n", "- [Introduction](#0)\n", "- [1 - Importing the Dataset](#1)\n", "- [2 - Preprocess the Data](#2)\n", "- [3 - Positional Encoding](#3)\n", "- [4 - Masking](#4)\n", "- [5 - Self-attention](#5)\n", " - [Exercise 1 - scaled_dot_product_attention](#ex-1)\n", "- [6 - Encoder](#6)\n", " - [6.1 - Encoder Layer](#6-1)\n", " - [6.2 - Full Encoder](#6-2)\n", "- [7 - Decoder](#7)\n", " - [7.1 - Decoder Layer](#7-1)\n", " - [Exercise 2 - DecoderLayer](#ex-2)\n", " - [7.2 - Full Decoder](#7-2)\n", " - [Exercise 3 - Decoder](#ex-3)\n", "- [8 - Transformer](#8)\n", " - [Exercise 4 - Transformer](#ex-4)\n", "- [9 - Initialize the Model](#9)\n", "- [10 - Prepare for Training the Model](#10)\n", "- [11 - Summarization](#11)\n", " - [Exercise 5 - next_word](#ex-5)\n", "- [12 - Train the Model](#12)\n", "- [13 - Summarize some sentences!](#13)\n" ] }, { "cell_type": "markdown", "id": "ee0da363", "metadata": { "colab_type": "text", "id": "H4NlfEQhRWPy" }, "source": [ "\n", "## Introduction\n", "\n", "Summarization is an important task in natural language processing and could be useful for a consumer enterprise. For example, bots can be used to scrape articles, summarize them, and then you can use sentiment analysis to identify the sentiment about certain stocks. Who wants to read an article or a long email today anyway, when you can build a transformer to summarize text for you? Let's get started. By completing this assignment you will learn to: \n", "\n", "- Use built-in functions to preprocess your data\n", "- Implement DotProductAttention\n", "- Implement Causal Attention\n", "- Understand how attention works\n", "- Build the transformer model\n", "- Evaluate your model\n", "- Summarize an article\n", "\n", "As you can tell, this model is slightly different than the ones you have already implemented. This is heavily based on attention and does not rely on sequences, which allows for parallel computing. " ] }, { "cell_type": "code", "execution_count": 1, "id": "7b49d856", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "colab_type": "code", "deletable": false, "editable": false, "id": "CChWzW-rEHVb", "outputId": "a0b3e98b-7fc6-492d-c8ad-3a263b54f670", "tags": [ "graded" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-06-12 13:07:48.525884: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" ] } ], "source": [ "import os\n", "#os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import tensorflow as tf\n", "import matplotlib.pyplot as plt\n", "import time\n", "import utils\n", "\n", "import textwrap\n", "wrapper = textwrap.TextWrapper(width=70)\n", "\n", "tf.keras.utils.set_random_seed(10)" ] }, { "cell_type": "code", "execution_count": 2, "id": "cfe093e6", "metadata": { "deletable": false, "editable": false }, "outputs": [], "source": [ "import w2_unittest" ] }, { "cell_type": "markdown", "id": "d56fc570", "metadata": { "colab_type": "text", "id": "kEL2rvaHRWP4" }, "source": [ "\n", "## 1 - Import the Dataset\n", "You have the dataset saved in a .json file, which you can easily open with pandas. The loading function has already been taken care of in `utils.py`." ] }, { "cell_type": "code", "execution_count": 3, "id": "074bcce3", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dialogue:\n", "Lucas: Hey! How was your day?\r\n", "Demi: Hey there! \r\n", "Demi: It was pretty fine, actually, thank you!\r\n", "Demi: I just got promoted! :D\r\n", "Lucas: Whoa! Great news!\r\n", "Lucas: Congratulations!\r\n", "Lucas: Such a success has to be celebrated.\r\n", "Demi: I agree! :D\r\n", "Demi: Tonight at Death & Co.?\r\n", "Lucas: Sure!\r\n", "Lucas: See you there at 10pm?\r\n", "Demi: Yeah! See you there! :D\n", "\n", "Summary:\n", "Demi got promoted. She will celebrate that with Lucas at Death & Co at 10 pm.\n" ] } ], "source": [ "data_dir = \"data/corpus\"\n", "\n", "train_data, test_data = utils.get_train_test_data(data_dir)\n", "\n", "# Take one example from the dataset and print it\n", "example_summary, example_dialogue = train_data.iloc[10]\n", "print(f\"Dialogue:\\n{example_dialogue}\")\n", "print(f\"\\nSummary:\\n{example_summary}\")" ] }, { "cell_type": "markdown", "id": "04210324", "metadata": {}, "source": [ "\n", "## 2 - Preprocess the data\n", "\n", "First you will do some preprocessing of the data and split it into inputs and outputs. Here you also remove some of the characters that are specific to this dataset and add the `[EOS]` (end of sentence) token to the end, like it was discussed in the lecture videos. You will also add a `[SOS]` (start of sentence) token to the beginning of the sentences." ] }, { "cell_type": "code", "execution_count": 4, "id": "9ba397a0", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "document, summary = utils.preprocess(train_data)\n", "document_test, summary_test = utils.preprocess(test_data)" ] }, { "cell_type": "markdown", "id": "0fe70280", "metadata": {}, "source": [ "Now perform the standard preprocessing with the tensorflow library. You will need to modify the filters, because you dont want the `[EOS]` tokens to be removed.\n", "\n", "Then create the vocabulary by combining the data in the documents and the summaries and using `.fit_on_texts()`:" ] }, { "cell_type": "code", "execution_count": 5, "id": "5dfab3c8", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Size of vocabulary: 34250\n" ] } ], "source": [ "# The [ and ] from default tokens cannot be removed, because they mark the SOS and EOS token.\n", "filters = '!\"#$%&()*+,-./:;<=>?@\\\\^_`{|}~\\t\\n'\n", "oov_token = '[UNK]'\n", "\n", "tokenizer = tf.keras.preprocessing.text.Tokenizer(filters=filters, oov_token=oov_token, lower=False)\n", "\n", "documents_and_summary = pd.concat([document, summary], ignore_index=True)\n", "\n", "tokenizer.fit_on_texts(documents_and_summary)\n", "\n", "inputs = tokenizer.texts_to_sequences(document)\n", "targets = tokenizer.texts_to_sequences(summary)\n", "\n", "vocab_size = len(tokenizer.word_index) + 1\n", "\n", "print(f'Size of vocabulary: {vocab_size}')" ] }, { "cell_type": "markdown", "id": "7341b3f5", "metadata": {}, "source": [ "Now you can pad the tokenized sequences for the training data.\n", "\n", "For the purpose of this notebook you need to limit the length of the sequences, as transformers are really big models and are not meant to be trained in such small environments." ] }, { "cell_type": "code", "execution_count": 6, "id": "c5846dd5", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-06-12 13:08:54.786615: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:54.844892: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:54.847837: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:54.856373: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:54.859450: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:54.862158: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:54.998431: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:55.000355: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:55.002105: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n", "2025-06-12 13:08:55.003865: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n", "2025-06-12 13:08:55.003973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 20825 MB memory: -> device: 0, name: NVIDIA A10G, pci bus id: 0000:00:1c.0, compute capability: 8.6\n" ] } ], "source": [ "# Limit the size of the input and output data for being able to run it in this environment.\n", "encoder_maxlen = 150\n", "decoder_maxlen = 50\n", "\n", "# Pad the sequences.\n", "inputs = tf.keras.preprocessing.sequence.pad_sequences(inputs, maxlen=encoder_maxlen, padding='post', truncating='post')\n", "targets = tf.keras.preprocessing.sequence.pad_sequences(targets, maxlen=decoder_maxlen, padding='post', truncating='post')\n", "\n", "inputs = tf.cast(inputs, dtype=tf.int32)\n", "targets = tf.cast(targets, dtype=tf.int32)\n", "\n", "# Create the final training dataset.\n", "BUFFER_SIZE = 10000\n", "BATCH_SIZE = 64\n", "\n", "dataset = tf.data.Dataset.from_tensor_slices((inputs, targets)).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)" ] }, { "cell_type": "markdown", "id": "58b25fb2", "metadata": {}, "source": [ "\n", "## 3 - Positional Encoding\n", "\n", "In sequence to sequence tasks, the relative order of your data is extremely important to its meaning. When you were training sequential neural networks such as RNNs, you fed your inputs into the network in order. Information about the order of your data was automatically fed into your model. However, when you train a Transformer network using multi-head attention, you feed your data into the model all at once. While this dramatically reduces training time, there is no information about the order of your data. This is where positional encoding is useful.\n", "\n", "You have learned how to implement the positional encoding in one of this week's labs. Here you will use the `positional_encoding` function to create positional encodings for your transformer. The function is already implemented for you." ] }, { "cell_type": "code", "execution_count": 7, "id": "0e65672c", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "def positional_encoding(positions, d_model):\n", " \"\"\"\n", " Precomputes a matrix with all the positional encodings \n", " \n", " Arguments:\n", " positions (int): Maximum number of positions to be encoded \n", " d_model (int): Encoding size \n", " \n", " Returns:\n", " pos_encoding (tf.Tensor): A matrix of shape (1, position, d_model) with the positional encodings\n", " \"\"\"\n", " \n", " position = np.arange(positions)[:, np.newaxis]\n", " k = np.arange(d_model)[np.newaxis, :]\n", " i = k // 2\n", " \n", " # initialize a matrix angle_rads of all the angles \n", " angle_rates = 1 / np.power(10000, (2 * i) / np.float32(d_model))\n", " angle_rads = position * angle_rates\n", " \n", " # apply sin to even indices in the array; 2i\n", " angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])\n", " \n", " # apply cos to odd indices in the array; 2i+1\n", " angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])\n", " \n", " pos_encoding = angle_rads[np.newaxis, ...]\n", " \n", " return tf.cast(pos_encoding, dtype=tf.float32)" ] }, { "cell_type": "markdown", "id": "9e1f1063", "metadata": {}, "source": [ "\n", "## 4 - Masking\n", "\n", "There are two types of masks that are useful when building your Transformer network: the *padding mask* and the *look-ahead mask*. Both help the softmax computation give the appropriate weights to the words in your input sentence. \n", "\n", "You have already learned how to implement and use them in one of this week's labs. Here they are implemented for you." ] }, { "cell_type": "code", "execution_count": 8, "id": "cfc7471c", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "def create_padding_mask(decoder_token_ids):\n", " \"\"\"\n", " Creates a matrix mask for the padding cells\n", " \n", " Arguments:\n", " decoder_token_ids (matrix like): matrix of size (n, m)\n", " \n", " Returns:\n", " mask (tf.Tensor): binary tensor of size (n, 1, m)\n", " \"\"\" \n", " seq = 1 - tf.cast(tf.math.equal(decoder_token_ids, 0), tf.float32)\n", " \n", " # add extra dimensions to add the padding to the attention logits. \n", " # this will allow for broadcasting later when comparing sequences\n", " return seq[:, tf.newaxis, :] \n", "\n", "\n", "def create_look_ahead_mask(sequence_length):\n", " \"\"\"\n", " Returns a lower triangular matrix filled with ones\n", " \n", " Arguments:\n", " sequence_length (int): matrix size\n", " \n", " Returns:\n", " mask (tf.Tensor): binary tensor of size (sequence_length, sequence_length)\n", " \"\"\"\n", " mask = tf.linalg.band_part(tf.ones((1, sequence_length, sequence_length)), -1, 0)\n", " return mask " ] }, { "cell_type": "markdown", "id": "89110af6", "metadata": {}, "source": [ "\n", "## 5 - Self-Attention\n", "\n", "As the authors of the Transformers paper state, \"Attention is All You Need\". \n", "\n", "\"Encoder\"\n", "
Figure 1: Self-Attention calculation visualization
\n", " \n", "The use of self-attention paired with traditional convolutional networks allows for parallelization which speeds up training. You will implement **scaled dot product attention** which takes in a query, key, value, and a mask as inputs to return rich, attention-based vector representations of the words in your sequence. This type of self-attention can be mathematically expressed as:\n", "$$\n", "\\text { Attention }(Q, K, V)=\\operatorname{softmax}\\left(\\frac{Q K^{T}}{\\sqrt{d_{k}}}+{M}\\right) V\\tag{4}\\\n", "$$\n", "\n", "* $Q$ is the matrix of queries \n", "* $K$ is the matrix of keys\n", "* $V$ is the matrix of values\n", "* $M$ is the optional mask you choose to apply \n", "* ${d_k}$ is the dimension of the keys, which is used to scale everything down so the softmax doesn't explode\n", "\n", "\n", "### Exercise 1 - scaled_dot_product_attention \n", "\n", "Implement the function `scaled_dot_product_attention()` to create attention-based representations.\n", "\n", "**Reminder**: The boolean mask parameter can be passed in as `none` or as either padding or look-ahead. \n", " \n", "* Multiply (1. - mask) by -1e9 before adding it to the scaled attention logits. \n", "\n", "**Additional Hints**\n", "* You may find [tf.matmul](https://www.tensorflow.org/api_docs/python/tf/linalg/matmul) useful for matrix multiplication (check how you can use the parameter transpose_b).\n", "* You can use [tf.keras.activations.softmax](https://www.tensorflow.org/api_docs/python/tf/keras/activations/softmax) for softmax." ] }, { "cell_type": "code", "execution_count": 9, "id": "3f434073", "metadata": { "deletable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "# GRADED FUNCTION: scaled_dot_product_attention\n", "def scaled_dot_product_attention(q, k, v, mask):\n", " \"\"\"\n", " Calculate the attention weights.\n", " q, k, v must have matching leading dimensions.\n", " k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v.\n", " The mask has different shapes depending on its type(padding or look ahead) \n", " but it must be broadcastable for addition.\n", "\n", " Arguments:\n", " q (tf.Tensor): query of shape (..., seq_len_q, depth)\n", " k (tf.Tensor): key of shape (..., seq_len_k, depth)\n", " v (tf.Tensor): value of shape (..., seq_len_v, depth_v)\n", " mask (tf.Tensor): mask with shape broadcastable \n", " to (..., seq_len_q, seq_len_k). Defaults to None.\n", "\n", " Returns:\n", " output -- attention_weights\n", " \"\"\"\n", " ### START CODE HERE ###\n", " \n", " # Multiply q and k transposed.\n", " matmul_qk = tf.matmul(q,k.T)\n", "\n", " # scale matmul_qk with the square root of dk\n", " dk = tf.cast(k.shape[-1], tf.float32)\n", " scaled_attention_logits = matmul_qk / np.sqrt(dk)\n", "\n", " # add the mask to the scaled tensor.\n", " if mask is not None: # Don't replace this None\n", " scaled_attention_logits += (1 - mask)* (-1e9)\n", "\n", " # softmax is normalized on the last axis (seq_len_k) so that the scores add up to 1.\n", " attention_weights = tf.keras.activations.softmax(scaled_attention_logits)\n", "\n", " # Multiply the attention weights by v\n", " output = tf.matmul(attention_weights,v)\n", " \n", " ### END CODE HERE ###\n", "\n", " return output, attention_weights" ] }, { "cell_type": "code", "execution_count": 10, "id": "5ed8f5af", "metadata": { "deletable": false, "editable": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Output:\n", " [[[1. 0.62]\n", " [0.62 0.62]\n", " [0.74 0.31]]]\n", "\n", "Attention weigths:\n", " [[[0. 0.38 0. 0.23 0.38]\n", " [0.38 0. 0. 0.23 0.38]\n", " [0.26 0.43 0. 0.16 0.16]]]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-12 13:09:08.437381: I tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:606] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.\n" ] } ], "source": [ "# Test your function!\n", "q = np.array([[1, 1, 0, 1], [0, 1, 1, 1], [1, 0, 1, 1]]).astype(np.float32)\n", "k = np.array([[1, 1, 0, 1], [1, 0, 1, 1 ], [1, 1, 1, 0], [0, 0, 0, 1], [0, 1, 0, 1]]).astype(np.float32)\n", "v = np.array([[0, 0], [1, 0], [1, 0], [1, 1], [1, 1]]).astype(np.float32)\n", "mask = np.array([[[0, 1, 0, 1, 1], [1, 0, 0, 1, 1], [1, 1, 0, 1, 1]]])\n", "\n", "ou, atw = scaled_dot_product_attention(q, k, v, mask)\n", "ou = np.around(ou, decimals=2)\n", "atw = np.around(atw, decimals=2)\n", "\n", "print(f\"Output:\\n {ou}\")\n", "print(f\"\\nAttention weigths:\\n {atw}\")" ] }, { "cell_type": "markdown", "id": "7b970a6e", "metadata": {}, "source": [ "##### __Expected Output__\n", "\n", "```\n", "Output:\n", " [[[1. 0.62]\n", " [0.62 0.62]\n", " [0.74 0.31]]]\n", "\n", "Attention weigths:\n", " [[[0. 0.38 0. 0.23 0.38]\n", " [0.38 0. 0. 0.23 0.38]\n", " [0.26 0.43 0. 0.16 0.16]]]\n", "```" ] }, { "cell_type": "code", "execution_count": 11, "id": "4755bb0b", "metadata": { "deletable": false, "editable": false, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[92m All tests passed!\n" ] } ], "source": [ "# UNIT TEST\n", "w2_unittest.test_scaled_dot_product_attention(scaled_dot_product_attention)" ] }, { "cell_type": "markdown", "id": "8dcbd521", "metadata": {}, "source": [ "Excellent work! You can now implement self-attention. With that, you can start building the encoder block! " ] }, { "cell_type": "markdown", "id": "00b9c92a", "metadata": {}, "source": [ "\n", "## 6 - Encoder\n", "\n", "The Transformer Encoder layer pairs self-attention and convolutional neural network style of processing to improve the speed of training and passes K and V matrices to the Decoder, which you'll build later in the assignment. In this section of the assignment, you will implement the Encoder by pairing multi-head attention and a feed forward neural network (Figure 2a). \n", "\"Encoder\"\n", "
Figure 2a: Transformer encoder layer
\n", "\n", "* `MultiHeadAttention` you can think of as computing the self-attention several times to detect different features. \n", "* Feed forward neural network contains two Dense layers which we'll implement as the function `FullyConnected`\n", "\n", "Your input sentence first passes through a *multi-head attention layer*, where the encoder looks at other words in the input sentence as it encodes a specific word. The outputs of the multi-head attention layer are then fed to a *feed forward neural network*. The exact same feed forward network is independently applied to each position.\n", " \n", "* For the `MultiHeadAttention` layer, you will use the [MultiHeadAttention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention) implemented in Keras. If you're curious about how to split the query matrix Q, key matrix K, and value matrix V into different heads, you can look through the implementation. \n", "* You will also use the [Sequential API](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) with two dense layers to built the feed forward neural network layers." ] }, { "cell_type": "code", "execution_count": 12, "id": "c3fd59d0", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "def FullyConnected(embedding_dim, fully_connected_dim):\n", " \"\"\"\n", " Returns a sequential model consisting of two dense layers. The first dense layer has\n", " fully_connected_dim neurons and is activated by relu. The second dense layer has\n", " embedding_dim and no activation.\n", "\n", " Arguments:\n", " embedding_dim (int): output dimension\n", " fully_connected_dim (int): dimension of the hidden layer\n", "\n", " Returns:\n", " _ (tf.keras.Model): sequential model\n", " \"\"\"\n", " return tf.keras.Sequential([\n", " tf.keras.layers.Dense(fully_connected_dim, activation='relu'), # (batch_size, seq_len, d_model)\n", " tf.keras.layers.Dense(embedding_dim) # (batch_size, seq_len, d_model)\n", " ])" ] }, { "cell_type": "markdown", "id": "99d7003a", "metadata": {}, "source": [ "\n", "### 6.1 Encoder Layer\n", "\n", "Now you can pair multi-head attention and feed forward neural network together in an encoder layer! You will also use residual connections and layer normalization to help speed up training (Figure 2a).\n", "\n", "The encoder block (Figure 2) is is already implemented for you. Take a very close look at its implementation, as you will later have to create the decoder yourself, and a lot of the code is very similar. The encoder block performs the following steps: \n", "1. It takes the Q, V, K matrices and a boolean mask to a multi-head attention layer. Remember that to compute *self*-attention Q, V and K are the same. You will also perform Dropout in this multi-head attention layer during training. \n", "2. There is a skip connection to add your original input `x` and the output of the multi-head attention layer. \n", "3. After adding the skip connection, the output passes through the first normalization layer.\n", "4. Finally, steps 1-3 are repeated but with the feed forward neural network with a dropout layer instead of the multi-head attention layer. \n", "\n", "
\n", " Additional Information (Click to expand)\n", " \n", "* The `__init__` method creates all the layers that will be accesed by the the `call` method. Wherever you want to use a layer defined inside the `__init__` method you will have to use the syntax `self.[insert layer name]`. \n", "* You will find the documentation of [MultiHeadAttention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention) helpful. *Note that if query, key and value are the same, then this function performs self-attention.*\n", "* The call arguments for `self.mha` are (Where B is for batch_size, T is for target sequence shapes, and S is output_shape):\n", " - `query`: Query Tensor of shape (B, T, dim).\n", " - `value`: Value Tensor of shape (B, S, dim).\n", " - `key`: Optional key Tensor of shape (B, S, dim). If not given, will use the same value for both key and value, which is the most common case.\n", " - `attention_mask`: a boolean mask of shape (B, T, S), that prevents attention to certain positions. The boolean mask specifies which query elements can attend to which key elements, 1 indicates attention and 0 indicates no attention. Broadcasting can happen for the missing batch dimensions and the head dimension.\n", " - `return_attention_scores`: A boolean to indicate whether the output should be attention output if True, or (attention_output, attention_scores) if False. Defaults to False.\n", " - `training`: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Defaults to either using the training mode of the parent layer/model, or False (inference) if there is no parent layer. Take a look at [tf.keras.layers.Dropout](https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/keras/layers/Dropout) for more details (Additional reading in [Keras FAQ](https://keras.io/getting_started/faq/#whats-the-difference-between-the-training-argument-in-call-and-the-trainable-attribute))" ] }, { "cell_type": "code", "execution_count": 13, "id": "51c1452b", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "class EncoderLayer(tf.keras.layers.Layer):\n", " \"\"\"\n", " The encoder layer is composed by a multi-head self-attention mechanism,\n", " followed by a simple, positionwise fully connected feed-forward network. \n", " This architecture includes a residual connection around each of the two \n", " sub-layers, followed by layer normalization.\n", " \"\"\"\n", " def __init__(self, embedding_dim, num_heads, fully_connected_dim,\n", " dropout_rate=0.1, layernorm_eps=1e-6):\n", " \n", " super(EncoderLayer, self).__init__()\n", "\n", " self.mha = tf.keras.layers.MultiHeadAttention(\n", " num_heads=num_heads,\n", " key_dim=embedding_dim,\n", " dropout=dropout_rate\n", " )\n", "\n", " self.ffn = FullyConnected(\n", " embedding_dim=embedding_dim,\n", " fully_connected_dim=fully_connected_dim\n", " )\n", "\n", " self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=layernorm_eps)\n", " self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=layernorm_eps)\n", "\n", " self.dropout_ffn = tf.keras.layers.Dropout(dropout_rate)\n", " \n", " def call(self, x, training, mask):\n", " \"\"\"\n", " Forward pass for the Encoder Layer\n", " \n", " Arguments:\n", " x (tf.Tensor): Tensor of shape (batch_size, input_seq_len, fully_connected_dim)\n", " training (bool): Boolean, set to true to activate\n", " the training mode for dropout layers\n", " mask (tf.Tensor): Boolean mask to ensure that the padding is not \n", " treated as part of the input\n", " Returns:\n", " encoder_layer_out (tf.Tensor): Tensor of shape (batch_size, input_seq_len, embedding_dim)\n", " \"\"\"\n", " # calculate self-attention using mha(~1 line).\n", " # Dropout is added by Keras automatically if the dropout parameter is non-zero during training\n", " self_mha_output = self.mha(x, x, x, mask) # Self attention (batch_size, input_seq_len, fully_connected_dim)\n", " \n", " # skip connection\n", " # apply layer normalization on sum of the input and the attention output to get the \n", " # output of the multi-head attention layer\n", " skip_x_attention = self.layernorm1(x + self_mha_output) # (batch_size, input_seq_len, fully_connected_dim)\n", "\n", " # pass the output of the multi-head attention layer through a ffn\n", " ffn_output = self.ffn(skip_x_attention) # (batch_size, input_seq_len, fully_connected_dim)\n", " \n", " # apply dropout layer to ffn output during training\n", " # use `training=training`\n", " ffn_output = self.dropout_ffn(ffn_output, training=training)\n", " \n", " # apply layer normalization on sum of the output from multi-head attention (skip connection) and ffn output\n", " # to get the output of the encoder layer\n", " encoder_layer_out = self.layernorm2(skip_x_attention + ffn_output) # (batch_size, input_seq_len, embedding_dim)\n", " \n", " return encoder_layer_out\n", " " ] }, { "cell_type": "markdown", "id": "2e36f13b", "metadata": {}, "source": [ "\n", "### 6.2 - Full Encoder\n", "\n", "Now you're ready to build the full Transformer Encoder (Figure 2b), where you will embed your input and add the positional encodings you calculated. You will then feed your encoded embeddings to a stack of Encoder layers. \n", "\n", "\"Encoder\"\n", "
Figure 2b: Transformer Encoder
\n", "\n", "The Encoder class is implemented for you. It performs the following steps: \n", "1. Pass the input through the Embedding layer.\n", "2. Scale the embedding by multiplying it by the square root of the embedding dimension. \n", "3. Add the position encoding: self.pos_encoding `[:, :seq_len, :]` to the embedding.\n", "4. Pass the encoded embedding through a dropout layer\n", "5. Pass the output of the dropout layer through the stack of encoding layers using a for loop." ] }, { "cell_type": "code", "execution_count": 14, "id": "d677d14e", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "class Encoder(tf.keras.layers.Layer):\n", " \"\"\"\n", " The entire Encoder starts by passing the input to an embedding layer \n", " and using positional encoding to then pass the output through a stack of\n", " encoder Layers\n", " \n", " \"\"\" \n", " def __init__(self, num_layers, embedding_dim, num_heads, fully_connected_dim, input_vocab_size,\n", " maximum_position_encoding, dropout_rate=0.1, layernorm_eps=1e-6):\n", " super(Encoder, self).__init__()\n", "\n", " self.embedding_dim = embedding_dim\n", " self.num_layers = num_layers\n", "\n", " self.embedding = tf.keras.layers.Embedding(input_vocab_size, self.embedding_dim)\n", " self.pos_encoding = positional_encoding(maximum_position_encoding, \n", " self.embedding_dim)\n", "\n", "\n", " self.enc_layers = [EncoderLayer(embedding_dim=self.embedding_dim,\n", " num_heads=num_heads,\n", " fully_connected_dim=fully_connected_dim,\n", " dropout_rate=dropout_rate,\n", " layernorm_eps=layernorm_eps) \n", " for _ in range(self.num_layers)]\n", "\n", " self.dropout = tf.keras.layers.Dropout(dropout_rate)\n", " \n", " def call(self, x, training, mask):\n", " \"\"\"\n", " Forward pass for the Encoder\n", " \n", " Arguments:\n", " x (tf.Tensor): Tensor of shape (batch_size, seq_len)\n", " training (bool): Boolean, set to true to activate\n", " the training mode for dropout layers\n", " mask (tf.Tensor): Boolean mask to ensure that the padding is not \n", " treated as part of the input\n", "\n", " Returns:\n", " x (tf.Tensor): Tensor of shape (batch_size, seq_len, embedding dim)\n", " \"\"\"\n", " seq_len = tf.shape(x)[1]\n", " \n", " # Pass input through the Embedding layer\n", " x = self.embedding(x) # (batch_size, input_seq_len, embedding_dim)\n", " # Scale embedding by multiplying it by the square root of the embedding dimension\n", " x *= tf.math.sqrt(tf.cast(self.embedding_dim, tf.float32))\n", " # Add the position encoding to embedding\n", " x += self.pos_encoding[:, :seq_len, :]\n", " # Pass the encoded embedding through a dropout layer\n", " # use `training=training`\n", " x = self.dropout(x, training=training)\n", " # Pass the output through the stack of encoding layers \n", " for i in range(self.num_layers):\n", " x = self.enc_layers[i](x, training, mask)\n", "\n", " return x # (batch_size, input_seq_len, embedding_dim)" ] }, { "cell_type": "markdown", "id": "9c7356fd", "metadata": {}, "source": [ "\n", "## 7 - Decoder\n", "\n", "Now it is time to implement the decoder. You have seen it in the videos and you can use some help by looking at the encoder implementation above. The Decoder layer takes the K and V matrices generated by the Encoder and computes the second multi-head attention layer with the Q matrix from the output (Figure 3a).\n", "\n", "\"Decoder\"\n", "
Figure 3a: Transformer Decoder layer
\n", "\n", " \n", "### 7.1 - Decoder Layer\n", "Again, you'll pair multi-head attention with a feed forward neural network, but this time you'll implement two multi-head attention layers. You will also use residual connections and layer normalization to help speed up training (Figure 3a).\n", "\n", " \n", "### Exercise 2 - DecoderLayer\n", " \n", "Implement `DecoderLayer()` using the `call()` method\n", " \n", "1. Block 1 is a multi-head attention layer with a residual connection, and look-ahead mask. Like in the `EncoderLayer`, Dropout is defined within the multi-head attention layer.\n", "2. Block 2 will take into account the output of the Encoder, so the multi-head attention layer will receive K and V from the encoder, and Q from the Block 1. You will then apply a normalization layer and a residual connection, just like you did before with the `EncoderLayer`.\n", "3. Finally, Block 3 is a feed forward neural network with dropout and normalization layers and a residual connection.\n", " \n", "**Additional Hints:**\n", "* The first two blocks are fairly similar to the EncoderLayer except you will return `attention_scores` when computing self-attention" ] }, { "cell_type": "code", "execution_count": 15, "id": "d8d3a38d", "metadata": { "deletable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "# GRADED FUNCTION: DecoderLayer\n", "class DecoderLayer(tf.keras.layers.Layer):\n", " \"\"\"\n", " The decoder layer is composed by two multi-head attention blocks, \n", " one that takes the new input and uses self-attention, and the other \n", " one that combines it with the output of the encoder, followed by a\n", " fully connected block. \n", " \"\"\"\n", " def __init__(self, embedding_dim, num_heads, fully_connected_dim, dropout_rate=0.1, layernorm_eps=1e-6):\n", " super(DecoderLayer, self).__init__()\n", "\n", " self.mha1 = tf.keras.layers.MultiHeadAttention(\n", " num_heads=num_heads,\n", " key_dim=embedding_dim,\n", " dropout=dropout_rate\n", " )\n", "\n", " self.mha2 = tf.keras.layers.MultiHeadAttention(\n", " num_heads=num_heads,\n", " key_dim=embedding_dim,\n", " dropout=dropout_rate\n", " )\n", "\n", " self.ffn = FullyConnected(\n", " embedding_dim=embedding_dim,\n", " fully_connected_dim=fully_connected_dim\n", " )\n", "\n", " self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=layernorm_eps)\n", " self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=layernorm_eps)\n", " self.layernorm3 = tf.keras.layers.LayerNormalization(epsilon=layernorm_eps)\n", "\n", " self.dropout_ffn = tf.keras.layers.Dropout(dropout_rate)\n", " \n", " def call(self, x, enc_output, training, look_ahead_mask, padding_mask):\n", " \"\"\"\n", " Forward pass for the Decoder Layer\n", " \n", " Arguments:\n", " x (tf.Tensor): Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n", " enc_output (tf.Tensor): Tensor of shape(batch_size, input_seq_len, fully_connected_dim)\n", " training (bool): Boolean, set to true to activate\n", " the training mode for dropout layers\n", " look_ahead_mask (tf.Tensor): Boolean mask for the target_input\n", " padding_mask (tf.Tensor): Boolean mask for the second multihead attention layer\n", " Returns:\n", " out3 (tf.Tensor): Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n", " attn_weights_block1 (tf.Tensor): Tensor of shape (batch_size, num_heads, target_seq_len, target_seq_len)\n", " attn_weights_block2 (tf.Tensor): Tensor of shape (batch_size, num_heads, target_seq_len, input_seq_len)\n", " \"\"\"\n", " \n", " ### START CODE HERE ###\n", " # enc_output.shape == (batch_size, input_seq_len, fully_connected_dim)\n", " \n", " # BLOCK 1\n", " # calculate self-attention and return attention scores as attn_weights_block1.\n", " # Dropout will be applied during training (~1 line).\n", " mult_attn_out1, attn_weights_block1 = self.mha1(x,x,x,look_ahead_mask,return_attention_scores=True)\n", " \n", " # apply layer normalization (layernorm1) to the sum of the attention output and the input (~1 line)\n", " Q1 = self.layernorm1(x + mult_attn_out1)\n", "\n", " # BLOCK 2\n", " # calculate self-attention using the Q from the first block and K and V from the encoder output. \n", " # Dropout will be applied during training\n", " # Return attention scores as attn_weights_block2 (~1 line) \n", " mult_attn_out2, attn_weights_block2 = self.mha2(Q1,enc_output,enc_output,padding_mask,return_attention_scores=True)\n", " \n", " # # apply layer normalization (layernorm2) to the sum of the attention output and the Q from the first block (~1 line)\n", " mult_attn_out2 = self.layernorm2(mult_attn_out2 + Q1)\n", " \n", " # pass the output of the second block through a ffn layer\n", " ffn_output = self.ffn(mult_attn_out2)\n", " \n", " # apply a dropout layer to the ffn output\n", " # use `training=training`\n", " ffn_output = self.dropout_ffn(ffn_output,training=training)\n", " \n", " # apply layer normalization (layernorm3) to the sum of the ffn output and the output of the second block\n", " out3 = self.layernorm3(ffn_output + mult_attn_out2)\n", " ### END CODE HERE ###\n", "\n", " return out3, attn_weights_block1, attn_weights_block2\n", " " ] }, { "cell_type": "code", "execution_count": 16, "id": "41686c8b", "metadata": { "deletable": false, "editable": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using embedding_dim=12 and num_heads=16:\n", "\n", "q has shape:(1, 15, 12)\n", "Output of encoder has shape:(1, 7, 8)\n", "\n", "Output of decoder layer has shape:(1, 15, 12)\n", "Att Weights Block 1 has shape:(1, 16, 15, 15)\n", "Att Weights Block 2 has shape:(1, 16, 15, 7)\n" ] } ], "source": [ "# Test your function!\n", "key_dim = 12\n", "n_heads = 16\n", "\n", "decoderLayer_test = DecoderLayer(embedding_dim=key_dim, num_heads=n_heads, fully_connected_dim=32)\n", "\n", "q = np.ones((1, 15, key_dim))\n", "encoder_test_output = tf.convert_to_tensor(np.random.rand(1, 7, 8))\n", "look_ahead_mask = create_look_ahead_mask(q.shape[1])\n", "\n", "out, attn_w_b1, attn_w_b2 = decoderLayer_test(q, encoder_test_output, False, look_ahead_mask, None)\n", "\n", "print(f\"Using embedding_dim={key_dim} and num_heads={n_heads}:\\n\")\n", "print(f\"q has shape:{q.shape}\")\n", "print(f\"Output of encoder has shape:{encoder_test_output.shape}\\n\")\n", "\n", "print(f\"Output of decoder layer has shape:{out.shape}\")\n", "print(f\"Att Weights Block 1 has shape:{attn_w_b1.shape}\")\n", "print(f\"Att Weights Block 2 has shape:{attn_w_b2.shape}\")" ] }, { "cell_type": "markdown", "id": "af9b85a3", "metadata": {}, "source": [ "##### __Expected Output__\n", "\n", "```\n", "Output:\n", "Using embedding_dim=12 and num_heads=16:\n", "\n", "q has shape:(1, 15, 12)\n", "Output of encoder has shape:(1, 7, 8)\n", "\n", "Output of decoder layer has shape:(1, 15, 12)\n", "Att Weights Block 1 has shape:(1, 16, 15, 15)\n", "Att Weights Block 2 has shape:(1, 16, 15, 7)\n", "```" ] }, { "cell_type": "code", "execution_count": 17, "id": "932f7320", "metadata": { "deletable": false, "editable": false, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[92m All tests passed!\n" ] } ], "source": [ "# UNIT TEST\n", "w2_unittest.test_decoderlayer(DecoderLayer, create_look_ahead_mask)" ] }, { "cell_type": "markdown", "id": "66b82ccf", "metadata": {}, "source": [ " \n", "### 7.2 - Full Decoder\n", "You're almost there! Time to use your Decoder layer to build a full Transformer Decoder (Figure 3b). You will embed your output and add positional encodings. You will then feed your encoded embeddings to a stack of Decoder layers. \n", "\n", "\n", "\"Decoder\"\n", "
Figure 3b: Transformer Decoder
\n", "\n", " \n", "### Exercise 3 - Decoder\n", "\n", "Implement `Decoder()` using the `call()` method to embed your output, add positional encoding, and implement multiple decoder layers.\n", " \n", "In this exercise, you will initialize your Decoder with an Embedding layer, positional encoding, and multiple DecoderLayers. Your `call()` method will perform the following steps: \n", "1. Pass your generated output through the Embedding layer.\n", "2. Scale your embedding by multiplying it by the square root of your embedding dimension. Remember to cast the embedding dimension to data type `tf.float32` before computing the square root.\n", "3. Add the position encoding: self.pos_encoding `[:, :seq_len, :]` to your embedding.\n", "4. Pass the encoded embedding through a dropout layer, remembering to use the `training` parameter to set the model training mode. \n", "5. Pass the output of the dropout layer through the stack of Decoding layers using a for loop." ] }, { "cell_type": "code", "execution_count": 18, "id": "57dde3be", "metadata": { "deletable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "# GRADED FUNCTION: Decoder\n", "class Decoder(tf.keras.layers.Layer):\n", " \"\"\"\n", " The entire Encoder starts by passing the target input to an embedding layer \n", " and using positional encoding to then pass the output through a stack of\n", " decoder Layers\n", " \n", " \"\"\" \n", " def __init__(self, num_layers, embedding_dim, num_heads, fully_connected_dim, target_vocab_size,\n", " maximum_position_encoding, dropout_rate=0.1, layernorm_eps=1e-6):\n", " super(Decoder, self).__init__()\n", "\n", " self.embedding_dim = embedding_dim\n", " self.num_layers = num_layers\n", "\n", " self.embedding = tf.keras.layers.Embedding(target_vocab_size, self.embedding_dim)\n", " self.pos_encoding = positional_encoding(maximum_position_encoding, self.embedding_dim)\n", "\n", " self.dec_layers = [DecoderLayer(embedding_dim=self.embedding_dim,\n", " num_heads=num_heads,\n", " fully_connected_dim=fully_connected_dim,\n", " dropout_rate=dropout_rate,\n", " layernorm_eps=layernorm_eps) \n", " for _ in range(self.num_layers)]\n", " self.dropout = tf.keras.layers.Dropout(dropout_rate)\n", " \n", " def call(self, x, enc_output, training, \n", " look_ahead_mask, padding_mask):\n", " \"\"\"\n", " Forward pass for the Decoder\n", " \n", " Arguments:\n", " x (tf.Tensor): Tensor of shape (batch_size, target_seq_len)\n", " enc_output (tf.Tensor): Tensor of shape(batch_size, input_seq_len, fully_connected_dim)\n", " training (bool): Boolean, set to true to activate\n", " the training mode for dropout layers\n", " look_ahead_mask (tf.Tensor): Boolean mask for the target_input\n", " padding_mask (tf.Tensor): Boolean mask for the second multihead attention layer\n", " Returns:\n", " x (tf.Tensor): Tensor of shape (batch_size, target_seq_len, fully_connected_dim)\n", " attention_weights (dict[str: tf.Tensor]): Dictionary of tensors containing all the attention weights\n", " each of shape Tensor of shape (batch_size, num_heads, target_seq_len, input_seq_len)\n", " \"\"\"\n", "\n", " seq_len = tf.shape(x)[1]\n", " attention_weights = {}\n", " \n", " ### START CODE HERE ###\n", " # create word embeddings \n", " x = self.embedding(x)\n", " \n", " # scale embeddings by multiplying by the square root of their dimension\n", " x *= tf.math.sqrt(tf.cast(self.embedding_dim, tf.float32))\n", " \n", " # add positional encodings to word embedding\n", " x += self.pos_encoding[:,:seq_len,:]\n", "\n", " # apply a dropout layer to x\n", " # use `training=training`\n", " x = self.dropout(x,training=training)\n", "\n", " # use a for loop to pass x through a stack of decoder layers and update attention_weights (~4 lines total)\n", " for i in range(self.num_layers):\n", " # pass x and the encoder output through a stack of decoder layers and save the attention weights\n", " # of block 1 and 2 (~1 line)\n", " x, block1, block2 = self.dec_layers[i](x, enc_output,training, look_ahead_mask, padding_mask)\n", "\n", " #update attention_weights dictionary with the attention weights of block 1 and block 2\n", " attention_weights['decoder_layer{}_block1_self_att'.format(i+1)] = block1\n", " attention_weights['decoder_layer{}_block2_decenc_att'.format(i+1)] = block2\n", " ### END CODE HERE ###\n", " \n", " # x.shape == (batch_size, target_seq_len, fully_connected_dim)\n", " return x, attention_weights" ] }, { "cell_type": "code", "execution_count": 19, "id": "04e877fb", "metadata": { "deletable": false, "editable": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using num_layers=5, embedding_dim=13 and num_heads=17:\n", "\n", "x has shape:(3, 4)\n", "Output of encoder has shape:(3, 7, 9)\n", "\n", "Output of decoder has shape:(3, 4, 13)\n", "\n", "Attention weights:\n", "decoder_layer1_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer1_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer2_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer2_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer3_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer3_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer4_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer4_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer5_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer5_block2_decenc_att has shape:(3, 17, 4, 7)\n" ] } ], "source": [ "# Test your function!\n", "n_layers = 5\n", "emb_d = 13\n", "n_heads = 17\n", "fully_connected_dim = 16\n", "target_vocab_size = 300\n", "maximum_position_encoding = 6\n", "\n", "x = np.array([[3, 2, 1, 1], [2, 1, 1, 0], [2, 1, 1, 0]])\n", "\n", "encoder_test_output = tf.convert_to_tensor(np.random.rand(3, 7, 9))\n", "\n", "look_ahead_mask = create_look_ahead_mask(x.shape[1])\n", "\n", "decoder_test = Decoder(n_layers, emb_d, n_heads, fully_connected_dim, target_vocab_size,maximum_position_encoding)\n", " \n", "outd, att_weights = decoder_test(x, encoder_test_output, False, look_ahead_mask, None)\n", "\n", "print(f\"Using num_layers={n_layers}, embedding_dim={emb_d} and num_heads={n_heads}:\\n\")\n", "print(f\"x has shape:{x.shape}\")\n", "print(f\"Output of encoder has shape:{encoder_test_output.shape}\\n\")\n", "\n", "print(f\"Output of decoder has shape:{outd.shape}\\n\")\n", "print(\"Attention weights:\")\n", "for name, tensor in att_weights.items():\n", " print(f\"{name} has shape:{tensor.shape}\")" ] }, { "cell_type": "markdown", "id": "9aa2ff15", "metadata": {}, "source": [ "##### __Expected Output__\n", "\n", "```\n", "Using num_layers=5, embedding_dim=13 and num_heads=17:\n", "\n", "x has shape:(3, 4)\n", "Output of encoder has shape:(3, 7, 9)\n", "\n", "Output of decoder has shape:(3, 4, 13)\n", "\n", "Attention weights:\n", "decoder_layer1_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer1_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer2_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer2_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer3_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer3_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer4_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer4_block2_decenc_att has shape:(3, 17, 4, 7)\n", "decoder_layer5_block1_self_att has shape:(3, 17, 4, 4)\n", "decoder_layer5_block2_decenc_att has shape:(3, 17, 4, 7)\n", "```" ] }, { "cell_type": "code", "execution_count": 20, "id": "e92745de", "metadata": { "deletable": false, "editable": false, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[92m All tests passed!\n" ] } ], "source": [ "# UNIT TEST\n", "w2_unittest.test_decoder(Decoder, create_look_ahead_mask, create_padding_mask)" ] }, { "cell_type": "markdown", "id": "848ba4b5", "metadata": {}, "source": [ " \n", "## 8 - Transformer\n", "\n", "Phew! This has been quite the assignment! Congratulations! You've done all the hard work, now it's time to put it all together. \n", "\n", "\"Transformer\"\n", "
Figure 4: Transformer
\n", " \n", "The flow of data through the Transformer Architecture is as follows:\n", "* First your input passes through an Encoder, which is just repeated Encoder layers that you implemented:\n", " - embedding and positional encoding of your input\n", " - multi-head attention on your input\n", " - feed forward neural network to help detect features\n", "* Then the predicted output passes through a Decoder, consisting of the decoder layers that you implemented:\n", " - embedding and positional encoding of the output\n", " - multi-head attention on your generated output\n", " - multi-head attention with the Q from the first multi-head attention layer and the K and V from the Encoder\n", " - a feed forward neural network to help detect features\n", "* Finally, after the Nth Decoder layer, one dense layer and a softmax are applied to generate prediction for the next output in your sequence.\n", "\n", " \n", "### Exercise 4 - Transformer\n", "\n", "Implement `Transformer()` using the `call()` method\n", "1. Pass the input through the Encoder with the appropiate mask.\n", "2. Pass the encoder output and the target through the Decoder with the appropiate mask.\n", "3. Apply a linear transformation and a softmax to get a prediction." ] }, { "cell_type": "code", "execution_count": 21, "id": "c9e6cb07", "metadata": { "deletable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "# GRADED FUNCTION: Transformer\n", "class Transformer(tf.keras.Model):\n", " \"\"\"\n", " Complete transformer with an Encoder and a Decoder\n", " \"\"\"\n", " def __init__(self, num_layers, embedding_dim, num_heads, fully_connected_dim, input_vocab_size, \n", " target_vocab_size, max_positional_encoding_input,\n", " max_positional_encoding_target, dropout_rate=0.1, layernorm_eps=1e-6):\n", " super(Transformer, self).__init__()\n", "\n", " self.encoder = Encoder(num_layers=num_layers,\n", " embedding_dim=embedding_dim,\n", " num_heads=num_heads,\n", " fully_connected_dim=fully_connected_dim,\n", " input_vocab_size=input_vocab_size,\n", " maximum_position_encoding=max_positional_encoding_input,\n", " dropout_rate=dropout_rate,\n", " layernorm_eps=layernorm_eps)\n", "\n", " self.decoder = Decoder(num_layers=num_layers, \n", " embedding_dim=embedding_dim,\n", " num_heads=num_heads,\n", " fully_connected_dim=fully_connected_dim,\n", " target_vocab_size=target_vocab_size, \n", " maximum_position_encoding=max_positional_encoding_target,\n", " dropout_rate=dropout_rate,\n", " layernorm_eps=layernorm_eps)\n", "\n", " self.final_layer = tf.keras.layers.Dense(target_vocab_size, activation='softmax')\n", " \n", " def call(self, input_sentence, output_sentence, training, enc_padding_mask, look_ahead_mask, dec_padding_mask):\n", " \"\"\"\n", " Forward pass for the entire Transformer\n", " Arguments:\n", " input_sentence (tf.Tensor): Tensor of shape (batch_size, input_seq_len)\n", " An array of the indexes of the words in the input sentence\n", " output_sentence (tf.Tensor): Tensor of shape (batch_size, target_seq_len)\n", " An array of the indexes of the words in the output sentence\n", " training (bool): Boolean, set to true to activate\n", " the training mode for dropout layers\n", " enc_padding_mask (tf.Tensor): Boolean mask to ensure that the padding is not \n", " treated as part of the input\n", " look_ahead_mask (tf.Tensor): Boolean mask for the target_input\n", " dec_padding_mask (tf.Tensor): Boolean mask for the second multihead attention layer\n", " Returns:\n", " final_output (tf.Tensor): The final output of the model\n", " attention_weights (dict[str: tf.Tensor]): Dictionary of tensors containing all the attention weights for the decoder\n", " each of shape Tensor of shape (batch_size, num_heads, target_seq_len, input_seq_len)\n", " \n", " \"\"\"\n", " ### START CODE HERE ###\n", " # call self.encoder with the appropriate arguments to get the encoder output\n", " enc_output = self.encoder(input_sentence, training, enc_padding_mask)\n", " \n", " # call self.decoder with the appropriate arguments to get the decoder output\n", " # dec_output.shape == (batch_size, tar_seq_len, fully_connected_dim)\n", " dec_output, attention_weights = self.decoder(output_sentence, enc_output, training, look_ahead_mask, dec_padding_mask)\n", " \n", " # pass decoder output through a linear layer and softmax (~1 line)\n", " final_output = self.final_layer(dec_output)\n", " ### END CODE HERE ###\n", "\n", " return final_output, attention_weights" ] }, { "cell_type": "code", "execution_count": 22, "id": "3cd93c99", "metadata": { "deletable": false, "editable": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Using num_layers=3, target_vocab_size=350 and num_heads=17:\n", "\n", "sentence_a has shape:(1, 7)\n", "sentence_b has shape:(1, 7)\n", "\n", "Output of transformer (summary) has shape:(1, 7, 350)\n", "\n", "Attention weights:\n", "decoder_layer1_block1_self_att has shape:(1, 17, 7, 7)\n", "decoder_layer1_block2_decenc_att has shape:(1, 17, 7, 7)\n", "decoder_layer2_block1_self_att has shape:(1, 17, 7, 7)\n", "decoder_layer2_block2_decenc_att has shape:(1, 17, 7, 7)\n", "decoder_layer3_block1_self_att has shape:(1, 17, 7, 7)\n", "decoder_layer3_block2_decenc_att has shape:(1, 17, 7, 7)\n" ] } ], "source": [ "# Test your function!\n", "n_layers = 3\n", "emb_d = 13\n", "n_heads = 17\n", "fully_connected_dim = 8\n", "input_vocab_size = 300\n", "target_vocab_size = 350\n", "max_positional_encoding_input = 12\n", "max_positional_encoding_target = 12\n", "\n", "transformer = Transformer(n_layers, \n", " emb_d, \n", " n_heads, \n", " fully_connected_dim, \n", " input_vocab_size, \n", " target_vocab_size, \n", " max_positional_encoding_input,\n", " max_positional_encoding_target)\n", "\n", "# 0 is the padding value\n", "sentence_a = np.array([[2, 3, 1, 3, 0, 0, 0]])\n", "sentence_b = np.array([[1, 3, 4, 0, 0, 0, 0]])\n", "\n", "enc_padding_mask = create_padding_mask(sentence_a)\n", "dec_padding_mask = create_padding_mask(sentence_a)\n", "\n", "look_ahead_mask = create_look_ahead_mask(sentence_a.shape[1])\n", "\n", "test_summary, att_weights = transformer(\n", " sentence_a,\n", " sentence_b,\n", " False,\n", " enc_padding_mask,\n", " look_ahead_mask,\n", " dec_padding_mask\n", ")\n", "\n", "print(f\"Using num_layers={n_layers}, target_vocab_size={target_vocab_size} and num_heads={n_heads}:\\n\")\n", "print(f\"sentence_a has shape:{sentence_a.shape}\")\n", "print(f\"sentence_b has shape:{sentence_b.shape}\")\n", "\n", "print(f\"\\nOutput of transformer (summary) has shape:{test_summary.shape}\\n\")\n", "print(\"Attention weights:\")\n", "for name, tensor in att_weights.items():\n", " print(f\"{name} has shape:{tensor.shape}\")" ] }, { "cell_type": "markdown", "id": "95c9f812", "metadata": {}, "source": [ "##### __Expected Output__\n", "\n", "```\n", "Using num_layers=3, target_vocab_size=350 and num_heads=17:\n", "\n", "sentence_a has shape:(1, 7)\n", "sentence_b has shape:(1, 7)\n", "\n", "Output of transformer (summary) has shape:(1, 7, 350)\n", "\n", "Attention weights:\n", "decoder_layer1_block1_self_att has shape:(1, 17, 7, 7)\n", "decoder_layer1_block2_decenc_att has shape:(1, 17, 7, 7)\n", "decoder_layer2_block1_self_att has shape:(1, 17, 7, 7)\n", "decoder_layer2_block2_decenc_att has shape:(1, 17, 7, 7)\n", "decoder_layer3_block1_self_att has shape:(1, 17, 7, 7)\n", "decoder_layer3_block2_decenc_att has shape:(1, 17, 7, 7)\n", "```" ] }, { "cell_type": "code", "execution_count": 23, "id": "a2d035a5", "metadata": { "deletable": false, "editable": false, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[92m All tests passed!\n" ] } ], "source": [ "# UNIT TEST\n", "w2_unittest.test_transformer(Transformer, create_look_ahead_mask, create_padding_mask)" ] }, { "cell_type": "markdown", "id": "33e8a0c2", "metadata": {}, "source": [ "\n", "## 9 - Initialize the Model\n", "Now that you have defined the model, you can initialize and train it. First you can initialize the model with the parameters below. Note that generally these models are much larger and you are using a smaller version to fit this environment and to be able to train it in just a few minutes.\n", "\n", "The base model described in the original Transformer paper used `num_layers=6`, `embedding_dim=512`, and `fully_connected_dim=2048`." ] }, { "cell_type": "code", "execution_count": 24, "id": "a5f79f64", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "# Define the model parameters\n", "num_layers = 2\n", "embedding_dim = 128\n", "fully_connected_dim = 128\n", "num_heads = 2\n", "positional_encoding_length = 256\n", "\n", "# Initialize the model\n", "transformer = Transformer(\n", " num_layers, \n", " embedding_dim, \n", " num_heads, \n", " fully_connected_dim,\n", " vocab_size, \n", " vocab_size, \n", " positional_encoding_length, \n", " positional_encoding_length,\n", ")" ] }, { "cell_type": "markdown", "id": "71473c27", "metadata": {}, "source": [ "\n", "## 10 - Prepare for Training the Model\n", "\n", "The original transformer paper uses Adam optimizer with custom learning rate scheduling, which we define in the cell below. This was empirically shown to produce faster convergence." ] }, { "cell_type": "code", "execution_count": 25, "id": "eb402089", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n", " def __init__(self, d_model, warmup_steps=4000):\n", " super(CustomSchedule, self).__init__()\n", " self.d_model = tf.cast(d_model, dtype=tf.float32)\n", " self.warmup_steps = warmup_steps\n", " \n", " def __call__(self, step):\n", " step = tf.cast(step, dtype=tf.float32)\n", " arg1 = tf.math.rsqrt(step)\n", " arg2 = step * (self.warmup_steps ** -1.5)\n", "\n", " return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)\n", "\n", "learning_rate = CustomSchedule(embedding_dim)\n", "\n", "optimizer = tf.keras.optimizers.Adam(0.0002, beta_1=0.9, beta_2=0.98, epsilon=1e-9)" ] }, { "cell_type": "markdown", "id": "ad854ab6", "metadata": {}, "source": [ "Below you can plot, how the custom learning rate looks like." ] }, { "cell_type": "code", "execution_count": 26, "id": "35a17a59", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "data": { "text/plain": [ "Text(0.5, 0, 'Train Step')" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAGwCAYAAACJjDBkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABrBklEQVR4nO3de1xUdf4/8NcMMDNcB5DLgCLg/YaXvCCmmSuFZSbVlpq/dF2/2bZauVqZruJWtprZVpZlbRdrt/JSrZmpRXjLRBQEFUW8IeBluMNwv8x8fn8gRydRAWc4zPB6Ph7zQM58zpn3h0Hn5fl8zucohBACRERERNQsSrkLICIiIrJFDFFERERELcAQRURERNQCDFFERERELcAQRURERNQCDFFERERELcAQRURERNQCjnIXYM9MJhMuXboEd3d3KBQKucshIiKiJhBCoLS0FIGBgVAqb3y+iSHKii5duoSgoCC5yyAiIqIWyM7ORqdOnW74PEOUFbm7uwOofxM8PDxkroaIiIiawmAwICgoSPocvxGGKCtqGMLz8PBgiCIiIrIxt5qKw4nlRERERC3AEEVERETUAgxRRERERC3AEEVERETUAgxRRERERC3AEEVERETUAgxRRERERC3AEEVERETUAgxRRERERC3AEEVERETUArKHqDVr1iAkJAQajQbh4eE4ePDgTdtv2rQJvXr1gkajQVhYGLZt22b2vBACMTExCAgIgLOzMyIjI3H69GmzNq+99hpGjBgBFxcXeHp63vT1CgoK0KlTJygUChQXF7eki0RERGSHZA1RGzZswLx587B06VIcPnwYAwYMQFRUFHJzcxttv3//fkyZMgUzZ85EcnIyoqOjER0djdTUVKnNypUrsXr1aqxduxYJCQlwdXVFVFQUqqqqpDY1NTV49NFH8fTTT9+yxpkzZ6J///6331kiIiKyKwohhJDrxcPDwzF06FC89957AACTyYSgoCA888wzeOmll65rP2nSJJSXl2Pr1q3StuHDh2PgwIFYu3YthBAIDAzE/Pnz8fzzzwMASkpK4O/vj3Xr1mHy5Mlmx1u3bh3mzp17wzNMH3zwATZs2ICYmBiMHTsWRUVFNz1zVV1djerqaun7hrtAl5SUtPsbEAshYDQJODrIfvKTiIjopgwGA7Ra7S0/v2X7RKupqUFSUhIiIyOvFqNUIjIyEvHx8Y3uEx8fb9YeAKKioqT2GRkZ0Ov1Zm20Wi3Cw8NveMwbOXHiBF555RV88cUXUCqb9mNavnw5tFqt9AgKCmrWa9qzOV8lY/jyOOSWVt26MRERkQ2QLUTl5+fDaDTC39/fbLu/vz/0en2j++j1+pu2b/janGM2prq6GlOmTMEbb7yBzp07N3m/hQsXoqSkRHpkZ2c3eV97JoTAj8cuI7+sBp/sy5C7HCIiIotwlLuAtmjhwoXo3bs3/t//+3/N2k+tVkOtVlupKtuVW3p1iPOUvlTGSoiIiCxHtjNRPj4+cHBwQE5Ojtn2nJwc6HS6RvfR6XQ3bd/wtTnHbMzOnTuxadMmODo6wtHREWPHjpVqXrp0aZOPQ/WyCiukPx86X4SaOpOM1RAREVmGbCFKpVJh8ODBiIuLk7aZTCbExcUhIiKi0X0iIiLM2gNAbGys1D40NBQ6nc6sjcFgQEJCwg2P2Zhvv/0WR44cQUpKClJSUvDxxx8DAH799VfMnj27ycehelkFV0NUWXUdDmcVyVgNERGRZcg6nDdv3jxMnz4dQ4YMwbBhw/D222+jvLwcM2bMAABMmzYNHTt2xPLlywEAzz33HEaPHo0333wT48ePx/r165GYmIiPPvoIAKBQKDB37lwsW7YM3bt3R2hoKJYsWYLAwEBER0dLr5uVlYXCwkJkZWXBaDQiJSUFANCtWze4ubmha9euZnXm5+cDAHr37n3LdaXoepnXnIkCgD2n8jC8SweZqiEiIrIMWUPUpEmTkJeXh5iYGOj1egwcOBA7duyQJoZnZWWZXRk3YsQIfPXVV1i8eDEWLVqE7t27Y/PmzejXr5/U5sUXX0R5eTlmzZqF4uJijBw5Ejt27IBGo5HaxMTE4PPPP5e+HzRoEABg165duPvuu63c6/Yn+0qI6unvjvScUuxJz8OCcb1kroqIiOj2yLpOlL1r6joT9u6RD/YjKbMIr0zsi6VbjkMI4OCisfDz0Nx6ZyIiolbW5teJovYj88qcqEFBXgjrqAUA7D2dL2dJREREt40hiqyqoqYO+WX1Sxx09nbB6B6+AOrnRREREdkyhiiyquzCSgCA1tkJWhcnKUT9ejoPdUYudUBERLaLIYqsKrOgHED9WSgAGBjkCU8XJxRX1CIpk0sdEBGR7WKIIqtqWGizIUQ5Oijxh55+AIBf0nJuuB8REVFbxxBFViWFqA4u0rZ7+tQvYRF7Ige8OJSIiGwVQxRZ1e/PRAHAqB6+UDkocb6gAmfzyuQqjYiI6LYwRJFVNRai3NSOiOhav2J57IlcWeoiIiK6XQxRZDVGk8CFK1fnXRuigGuH9PStXhcREZElMESR1eQYqlBjNMFRqUCA1nx18rG96yeXJ2cXI6+0Wo7yiIiIbgtDFFlNw1BeRy9nODqY/6oFaJ0R1lELIYBdJzmkR0REtochiqwmq+D6+VDXahjS+5lDekREZIMYoshqGptUfq2ovjoAwN5T+TBU1bZaXURERJbAEEVWc6sQ1cPfDV19XVFjNCGOC28SEZGNYYgiq8m8EqKCOzQeohQKBcaHBQAAfjzKIT0iIrItDFFkNdlXQlTQDc5EAcD9/etD1N7TeSjlkB4REdkQhiiyitKqWhSW1wC48XAeAPT0d0cXX1fU1JkQl8ar9IiIyHYwRJFVNMyH8nZVwV3jdMN2ZkN6xy63Sm1ERESWwBBFVtGUobwG918JUXtOcUiPiIhsB0MUWUXDmajgJoSoXjp3dPGpH9LbyYU3iYjIRjBEkVVk3mKhzWspFAqMvzLBfEvKJavWRUREZCkMUWQVt1oj6vcmDgwEUD+kV1DGe+kREVHbxxBFViGFqBusEfV73fzcEdZRizqT4ARzIiKyCQxRZHF1RhMuFlUCaPqZKACIHtQRAPDd4YtWqYuIiMiSGKLI4i6XVKHOJKByUMLfQ9Pk/R4cEAgHpQIp2cXIyC+3YoVERES3jyGKLK5hKK+TtzMclIom7+frrsbIbj4AgP8l82wUERG1bQxRZHHNnVR+rYfvqB/S25x8EUIIi9ZFRERkSQxRZHG3E6Lu6eMPF5UDsgorcDiryNKlERERWQxDFFlcVjPWiPo9F5UjxvXTAQC+SeKQHhERtV0MUWRxt3MmCgD+OLgTAOCHI5dQUVNnsbqIiIgsiSGKLE665UsH1xbtPzy0A4I7uKCsug4/HuWaUURE1DYxRJFFlVTUoqSy/ibCQd7OLTqGUqnApKFBAIANh7ItVhsREZElMUSRRTWchfJxU8NF5dji4/zxjk5wUCqQmFmEM7mlliqPiIjIYhiiyKKuDuW1bD5UAz8PDf7Qyw8Az0YREVHbxBBFFpVZWL/SeEsnlV9r8pUhvW8PX0RNnem2j0dERGRJDFFkUdlXzkQFWSBEje7hC38PNQrLa/BLWs5tH4+IiMiSGKLIojKvrBEVbIEQ5eigxKOD689G/fdA5m0fj4iIyJJkD1Fr1qxBSEgINBoNwsPDcfDgwZu237RpE3r16gWNRoOwsDBs27bN7HkhBGJiYhAQEABnZ2dERkbi9OnTZm1ee+01jBgxAi4uLvD09LzuNY4cOYIpU6YgKCgIzs7O6N27N955553b7mt7IK0RdZtzohpMHhYEpQLYf7YAp3M4wZyIiNoOWUPUhg0bMG/ePCxduhSHDx/GgAEDEBUVhdzc3Ebb79+/H1OmTMHMmTORnJyM6OhoREdHIzU1VWqzcuVKrF69GmvXrkVCQgJcXV0RFRWFqqoqqU1NTQ0effRRPP30042+TlJSEvz8/PDf//4Xx48fx9///ncsXLgQ7733nmV/AHam1mjCpeJKAJaZEwUAnbxccE8ffwDAF/E8G0VERG2HQsh4l9fw8HAMHTpUCicmkwlBQUF45pln8NJLL13XftKkSSgvL8fWrVulbcOHD8fAgQOxdu1aCCEQGBiI+fPn4/nnnwcAlJSUwN/fH+vWrcPkyZPNjrdu3TrMnTsXxcXFt6x19uzZSEtLw86dO2/Yprq6GtXV1dL3BoMBQUFBKCkpgYeHxy1fw9adzy/H3at2Q+2oxMlXx0GhUFjkuPvP5OPxjxPgonLAgUVj4aFxsshxiYiIGmMwGKDVam/5+S3bmaiamhokJSUhMjLyajFKJSIjIxEfH9/oPvHx8WbtASAqKkpqn5GRAb1eb9ZGq9UiPDz8hsdsqpKSEnh7e9+0zfLly6HVaqVHUFDQbb2mrbn2di+WClAAENG1A7r7uaGixohvky5Y7LhERES3Q7YQlZ+fD6PRCH9/f7Pt/v7+0Ov1je6j1+tv2r7ha3OO2RT79+/Hhg0bMGvWrJu2W7hwIUpKSqRHdnb7Wt/odu+ZdyMKhQLTRoQAAP4TnwmTSbaTp0RERBLZJ5a3dampqZg4cSKWLl2Ke++996Zt1Wo1PDw8zB7tiaUnlV/r4UEd4a52xLn8cvx6Jt/ixyciImou2UKUj48PHBwckJNjvv5PTk4OdDpdo/vodLqbtm/42pxj3syJEycwduxYzJo1C4sXL272/u1NVoF1zkQBgKvaEY8M7gQAWPdbhsWPT0RE1FyyhSiVSoXBgwcjLi5O2mYymRAXF4eIiIhG94mIiDBrDwCxsbFS+9DQUOh0OrM2BoMBCQkJNzzmjRw/fhxjxozB9OnT8dprrzVr3/bKUrd8uZHpI0KgUAC70vO43AEREclO1uG8efPm4d///jc+//xzpKWl4emnn0Z5eTlmzJgBAJg2bRoWLlwotX/uueewY8cOvPnmmzh58iT+8Y9/IDExEXPmzAFQP3dm7ty5WLZsGbZs2YJjx45h2rRpCAwMRHR0tHScrKwspKSkICsrC0ajESkpKUhJSUFZWRmA+iG8MWPG4N5778W8efOg1+uh1+uRl5fXej8cGyOEsNqcqAahPq6498pyB//+9ZxVXoOIiKipHOV88UmTJiEvLw8xMTHQ6/UYOHAgduzYIU0Mz8rKglJ5NeeNGDECX331FRYvXoxFixahe/fu2Lx5M/r16ye1efHFF1FeXo5Zs2ahuLgYI0eOxI4dO6DRaKQ2MTEx+Pzzz6XvBw0aBADYtWsX7r77bnzzzTfIy8vDf//7X/z3v/+V2gUHB+P8+fPW+nHYtKKKWpRV1wGoX9vJWmbd1RU/Hc/B5uRLeP7envDz0Nx6JyIiIiuQdZ0oe9fUdSbsQUp2MaLX/AadhwYHFo216mv98YP9SMwswl/v7ooXx/Wy6msREVH70+bXiSL7kllQDsB6Q3nXmnVXFwD199NrOPtFRETU2hiiyCKyr8yHCmqFEBXZ2x9dfF1hqKrDhkPtay0uIiJqOxiiyCIyC6x7Zd61lEoFnhxVfzbq030ZqDWarP6aREREv8cQRRZh7Svzfu+hQR3h667GxeJK/O/wxVZ5TSIiomsxRJFFtOZwHgBonBzw1JW5Ue/tOsOzUURE1OoYoui2VdcZcdlQBaB1hvMaPB7eGR1cVcgqrMD3KZda7XWJiIgAhiiygAtFlRACcFE5oIOrqtVe10XliCevnI1as+sM6ng2ioiIWhFDFN22a+dDKRSKVn3tJ4YHw8vFCRn55dh69HKrvjYREbVvDFF026x54+FbcVU74v+uXKn37s7TMJq4diwREbUOhii6ba19Zd7vTYsIhtbZCWfzyrH1KOdGERFR62CIotsmhahWnFR+LXeNE54cFQoA+FfsKV6pR0RErYIhim6bnMN5DWbcGQofNxUyCyq4ijkREbUKhii6LUII2YfzgPq5Uc/8oTsAYHXcaVTWGGWrhYiI2geGKLot+WU1qKw1QqEAOnnJF6IAYMqwzujk5Yzc0mqs239e1lqIiMj+MUTRbckqLAcABGqdoXKU99dJ5ajEvHt6AAA+2H0GJRW1stZDRET2jSGKbkuWdLsXZ5krqTdxYEf09HeHoaoOa/eelbscIiKyYwxRdFsyr0wqD/Z2lbmSeg5KBV6I6gkA+HRfBi4UVchcERER2SuGKLotci9v0Jixvf0wvIs3qutMeH1HutzlEBGRnWKIotuSLQ3ntZ0QpVAosOSBPlAogB+OXEJSZqHcJRERkR1iiKLbcnU4r+2EKADoG6jFpCFBAIBXfjgBE28HQ0REFsYQRS1WWWNEbmk1AHnXiLqR+ff2hJvaEUculGBzykW5yyEiIjvDEEUt1jBp213tCE8XJ5mruZ6vuxqzx3QDALy+4yQqaupkroiIiOwJQxS1WMNQXucOLlAoFDJX07gZd4YgyNsZOYZqvLfzjNzlEBGRHWGIohZrC7d7uRWNkwMWj+8DAPj3r+dwJrdU5oqIiMheMERRi9lCiAKAe/v44w+9/FBrFFi8ORVCcJI5ERHdPoYoarG2uEZUYxQKBV5+sC80TkocOFfISeZERGQRDFHUYrZyJgqoX8fqmT90BwC89mMa76tHRES3jSGKWsRkEtJCm23lli+38uSoLujq64r8shq88fNJucshIiIbxxBFLZJbWo3qOhMclAoEeGrkLqdJVI5KvDqxHwDgy4QsJGUWyVwRERHZMoYoapGGobxATw2cHGzn12hENx88fEdHCAG8+M0RVNUa5S6JiIhslO18+lGbkmVjQ3nXinmgD3zd1TibV47VcaflLoeIiGwUQxS1SFZBOYC2dePhpvJ0UWFZdP2w3od7z+HYhRKZKyIiIlvEEEUtYktX5jUmqq8OD/QPgNEk8MI3R1BTZ5K7JCIisjEMUdQimQ3DeW18jaibefnBvvB2VeGkvhTv7+YtYYiIqHkYoqhFsm38TBQAdHBT4+UH+wIA3tt5BkcvFMtbEBER2RSGKGq28uo65JfVALDNOVHXeqB/AO4P06HOJDB3fQoqaurkLomIiGwEQxQ1W8N8KE8XJ2idnWSu5vYoFAr886Ew+HuocS6/HK/9mCZ3SUREZCNkD1Fr1qxBSEgINBoNwsPDcfDgwZu237RpE3r16gWNRoOwsDBs27bN7HkhBGJiYhAQEABnZ2dERkbi9Gnzy9hfe+01jBgxAi4uLvD09Gz0dbKysjB+/Hi4uLjAz88PL7zwAurqeJYCsP1J5b/n6aLCm48OBFC/CGdcWo68BRERkU2QNURt2LAB8+bNw9KlS3H48GEMGDAAUVFRyM3NbbT9/v37MWXKFMycORPJycmIjo5GdHQ0UlNTpTYrV67E6tWrsXbtWiQkJMDV1RVRUVGoqqqS2tTU1ODRRx/F008/3ejrGI1GjB8/HjU1Ndi/fz8+//xzrFu3DjExMZb9AdiohvlQtj6Ud62R3X3wfyNDAQAvfnMUeaXVMldERERtnpDRsGHDxOzZs6XvjUajCAwMFMuXL2+0/WOPPSbGjx9vti08PFw89dRTQgghTCaT0Ol04o033pCeLy4uFmq1Wnz99dfXHe+zzz4TWq32uu3btm0TSqVS6PV6adsHH3wgPDw8RHV1dZP7V1JSIgCIkpKSJu9jCxb/75gIXrBVvL49Te5SLKqypk5EvbVHBC/YKv70aYIwGk1yl0RERDJo6ue3bGeiampqkJSUhMjISGmbUqlEZGQk4uPjG90nPj7erD0AREVFSe0zMjKg1+vN2mi1WoSHh9/wmDd6nbCwMPj7+5u9jsFgwPHjx2+4X3V1NQwGg9nDHtnbcF4DjZMD3pk8CCpHJXal5+Hfv56TuyQiImrDZAtR+fn5MBqNZkEFAPz9/aHX6xvdR6/X37R9w9fmHLM5r3PtazRm+fLl0Gq10iMoKKjJr2lLpOUNbHiNqBvpqXPH0gl9AAArf0pH4vlCmSsiIqK2SvaJ5fZk4cKFKCkpkR7Z2dlyl2RxRpNAdpF9nolq8Piwzpg4MBBGk8Ccr5JRUMb5UUREdD3ZQpSPjw8cHByQk2N+JVROTg50Ol2j++h0upu2b/janGM253WufY3GqNVqeHh4mD3sjd5QhVqjgJODAgFaZ7nLsYqGZQ+6+LpCb6jC3zYegckk5C6LiIjaGNlClEqlwuDBgxEXFydtM5lMiIuLQ0RERKP7REREmLUHgNjYWKl9aGgodDqdWRuDwYCEhIQbHvNGr3Ps2DGzqwRjY2Ph4eGBPn36NPk49iiroP4sVCcvFzgoFTJXYz2uake8P/UOaJyU2HsqDx/sOSt3SURE1MbIOpw3b948/Pvf/8bnn3+OtLQ0PP300ygvL8eMGTMAANOmTcPChQul9s899xx27NiBN998EydPnsQ//vEPJCYmYs6cOQDqzyDMnTsXy5Ytw5YtW3Ds2DFMmzYNgYGBiI6Olo6TlZWFlJQUZGVlwWg0IiUlBSkpKSgrKwMA3HvvvejTpw+eeOIJHDlyBD/99BMWL16M2bNnQ61Wt94PqA3KKiwHYF/LG9xIL50HXpnYDwDw5s/p+PV0nswVERFRW+Io54tPmjQJeXl5iImJgV6vx8CBA7Fjxw5pEndWVhaUyqs5b8SIEfjqq6+wePFiLFq0CN27d8fmzZvRr18/qc2LL76I8vJyzJo1C8XFxRg5ciR27NgBjUYjtYmJicHnn38ufT9o0CAAwK5du3D33XfDwcEBW7duxdNPP42IiAi4urpi+vTpeOWVV6z9I2nzrl6ZZ59Deb/32JAgJJ4vxMbEC5jzVTK2zLkTwR1c5S6LiIjaAIUQgpM9rMRgMECr1aKkpMRu5kfN+eowth69jL/f3xtP3tVF7nJaRXWdEZM/OoDkrGL08HfDd3+9E25qWf//QUREVtTUz29enUfNYo+rld+K2tEBa//fYPi5q3EqpwzzNqRwojkRETFEUfPY60Kbt+LvocGHTwyGykGJn0/kYPXO07feiYiI7BpDFDWZoaoWRRW1AOxzoc1bGdTZC689VD//7u1fTmP7scsyV0RERHJiiKIma1jeoIOrqt3OCXp0SBBm3BkCAJi7IQWHs4rkLYiIiGTDEEVN1h7nQzXm7/f3xthefqiuM+HJzxORWVAud0lERCQDhihqsswrISq4HQ7lXcvRQYnVUwahX0cPFJTXYMZnh1BcUSN3WURE1MoYoqjJ2uuk8sa4qh3x6fShCNRqcC6/HLO+SEJ1nVHusoiIqBUxRFGTcTjPnJ+HBp/NGAZ3tSMOni/EfN5jj4ioXWGIoibLvDKxPJghStJT544P/t9gOCoV2Hr0MpZuOQ6uX0tE1D4wRFGT1BlNuFhcCaB9Lm9wMyO7++DNxwZAoQD+cyATb8WekrskIiJqBQxR1CSXS6pgNAmoHJXwd9fceod2ZuLAjnjlwb4AgNU7z+DTfRkyV0RERNbGEEVN0jCUF+TlDKVSIXM1bdMTESGYd08PAMArW0/g26QLMldERETWxBBFTcIr85rmmT90w5/vDAUAvPjtUexI5armRET2iiGKmiSzsH5ByeAOrjJX0rYpFAosHt8bj9zRCUaTwJyvkvHTcb3cZRERkRUwRFGTcHmDplMqFVj5x/6YODAQdSaBOV8dxi8ncuQui4iILIwhipqEw3nN46BU4M1HB2DCgEDUGgWe/jIJcWkMUkRE9oQhim5JCHF1jSgub9Bkjg5KvPXYAIwPC6gPUv89jF0nc+Uui4iILIQhim6ppLIWpVV1AIAgL4ao5nB0UOLtyQNxf5gONUYTnvpPEmI5tEdEZBduK0RVVVVZqg5qwxqG8nzd1XBWOchcje1xclDincmDcF+/+iD1l/8m4fuUi3KXRUREt6nZIcpkMuHVV19Fx44d4ebmhnPnzgEAlixZgk8++cTiBZL8eLuX2+fkoMS7Uwbh4Ts6wmgSmLshBV8lZMldFhER3YZmh6hly5Zh3bp1WLlyJVQqlbS9X79++Pjjjy1aHLUNnFRuGY4OSqz64wA8MTwYQgCL/ncMH+09K3dZRETUQs0OUV988QU++ugjTJ06FQ4OV4d2BgwYgJMnT1q0OGobuLyB5SiVCrwysS+evrsrAOCf207izZ/TedNiIiIb1OwQdfHiRXTr1u267SaTCbW1tRYpitoWXplnWQqFAgvG9cILUT0BAO/uPINF/zuGOqNJ5sqIiKg5mh2i+vTpg19//fW67d988w0GDRpkkaKobeFwnnXMHtMNr0b3g1IBfH0wG09+kYjy6jq5yyIioiZybO4OMTExmD59Oi5evAiTyYTvvvsO6enp+OKLL7B161Zr1Egyqqkz4XJJJQCgM89EWdwTw4Ph767GM18nY1d6Hqb8+wA+mT4Uvu5quUsjIqJbaPaZqIkTJ+KHH37AL7/8AldXV8TExCAtLQ0//PAD7rnnHmvUSDK6WFwJkwA0Tkr4uvGD3Rru7avDV08Oh5eLE45eKMEjH+zHubwyucsiIqJbaPaZKAAYNWoUYmNjLV0LtUHXDuUpFAqZq7Ffg4O98O3TIzD9s4PIKqzAIx/sx0fThmBoiLfcpRER0Q00+0xUly5dUFBQcN324uJidOnSxSJFUdtxNUS5ylyJ/evi64bvnr4T/TtpUVRRi8f/fQAbD2XLXRYREd1As0PU+fPnYTQar9teXV2Nixe5CrO9ySooB8BJ5a3F112N9bOG475+OtQaBV789iiWbT0Bo4lLIBARtTVNHs7bsmWL9OeffvoJWq1W+t5oNCIuLg4hISEWLY7kd/VMlLPMlbQfLipHrHn8DrwTdxrvxJ3Gx/sycDq3DO8+PggeGie5yyMioiuaHKKio6MB1K9xM336dLPnnJycEBISgjfffNOixZH8rq4RxeG81qRUKvC3e3qgh7875m9KwZ5TeXhozW/4ePpQhPrwvSAiaguaPJxnMplgMpnQuXNn5ObmSt+bTCZUV1cjPT0dDzzwgDVrpVYmhOBq5TIb3z8A3/xlBAK0GpzNK8eD7+7Dz8f1cpdFRERowZyojIwM+Pj4WKMWamMKy2tQXmOEQgF08uJwnlz6ddTi+zl3YkiwF0qr6zDrP0l4fcdJrnBORCSzFi1xUF5ejj179iArKws1NTVmzz377LMWKYzkl3nlLJTOQwONk8MtWpM1+blr8PWs4Vi+7SQ+/S0DH+w+iyPZxVg9ZRB8uH4XEZEsmh2ikpOTcf/996OiogLl5eXw9vZGfn4+XFxc4OfnxxBlRziU17Y4OSgRM6EPBnX2xIJvj2L/2QI8sHof1kwdhMHBXE+KiKi1NXs4729/+xsmTJiAoqIiODs748CBA8jMzMTgwYOxatUqa9RIMskq4D3z2qIJAwKxZc6d6OrrCr2hCpM+PICP9p6FicsgEBG1qmaHqJSUFMyfPx9KpRIODg6orq5GUFAQVq5ciUWLFlmjRpJJw3BeMENUm9PNzx3fzxmJ8f0DUGcS+Oe2k5j+2UHkllbJXRoRUbvR7BDl5OQEpbJ+Nz8/P2RlZQEAtFotsrObv7rymjVrEBISAo1Gg/DwcBw8ePCm7Tdt2oRevXpBo9EgLCwM27ZtM3teCIGYmBgEBATA2dkZkZGROH36tFmbwsJCTJ06FR4eHvD09MTMmTNRVmZ+r7KffvoJw4cPh7u7O3x9ffHII4/g/Pnzze6fLZPWiOKNh9skN7Uj3psyCMsfDoPGSYlfT+fj/nd+xZ5TeXKXRkTULjQ7RA0aNAiHDh0CAIwePRoxMTH48ssvMXfuXPTr169Zx9qwYQPmzZuHpUuX4vDhwxgwYACioqKQm5vbaPv9+/djypQpmDlzJpKTkxEdHY3o6GikpqZKbVauXInVq1dj7dq1SEhIgKurK6KiolBVdfV/6FOnTsXx48cRGxuLrVu3Yu/evZg1a5b0fEZGBiZOnIg//OEPSElJwU8//YT8/Hw8/PDDzeqfrcsu5HBeW6dQKDBlWGf8MGckeunckV9Wg+mfHsQ/t6Whpo5X7xERWZVopkOHDomdO3cKIYTIyckRUVFRwt3dXdxxxx0iOTm5WccaNmyYmD17tvS90WgUgYGBYvny5Y22f+yxx8T48ePNtoWHh4unnnpKCCGEyWQSOp1OvPHGG9LzxcXFQq1Wi6+//loIIcSJEycEAHHo0CGpzfbt24VCoRAXL14UQgixadMm4ejoKIxGo9Rmy5YtQqFQiJqamib3r6SkRAAQJSUlTd6nraisqRMhL20VwQu2ivzSKrnLoSaorKkTi/93TAQvqH/fxq/eK9L1BrnLIiKyOU39/G72maghQ4ZgzJgxAOqH83bs2AGDwYCkpCQMHDiwycepqalBUlISIiMjpW1KpRKRkZGIj49vdJ/4+Hiz9gAQFRUltc/IyIBerzdro9VqER4eLrWJj4+Hp6cnhgwZIrWJjIyEUqlEQkICAGDw4MFQKpX47LPPYDQaUVJSgv/85z+IjIyEk9ONb7tRXV0Ng8Fg9rBVF4oqIQTgqnKAt6tK7nKoCTRODng1uh8+fGIwPF2ckHrRgAfe3YeP9p7lvfeIiKyg2SHqRg4fPtysFcvz8/NhNBrh7+9vtt3f3x96feMrMuv1+pu2b/h6qzZ+fn5mzzs6OsLb21tqExoaip9//hmLFi2CWq2Gp6cnLly4gI0bN960T8uXL4dWq5UeQUFBN23flklDeR1coVAoZK6GmiOqrw4/zb0LY3r6oqbOhH9uO4nJH8Uj88rNpImIyDKaFaJ++uknPP/881i0aBHOnTsHADh58iSio6MxdOhQmEz2MQdDr9fjySefxPTp03Ho0CHs2bMHKpUKf/zjHyHEjf9Hv3DhQpSUlEiPlky0bysaPnB542Hb5O+hwad/GooVD4fBVeWAQ+eLMO7tX/GfA5k3/R0mIqKma/Jim5988gmefPJJeHt7o6ioCB9//DH+9a9/4ZlnnsGkSZOQmpqK3r17N/mFfXx84ODggJycHLPtOTk50Ol0je6j0+lu2r7ha05ODgICAszaNAw16nS66yau19XVobCwUNp/zZo10Gq1WLlypdTmv//9L4KCgpCQkIDhw4c3Wp9arYZabR+rR2cVVgLgpHJbplAoMHlYZ9zZzQcvfHMEB84VYsnmVPx8XI9/PhTGRVSJiG5Tk89EvfPOO3j99deRn5+PjRs3Ij8/H++//z6OHTuGtWvXNitAAYBKpcLgwYMRFxcnbTOZTIiLi0NERESj+0RERJi1B4DY2FipfWhoKHQ6nVkbg8GAhIQEqU1ERASKi4uRlJQktdm5cydMJhPCw8MBABUVFdIyDg0cHBykGtuDrMIrZ6I6uMpcCd2uIG8XfPV/wxHzQB+oHeuXQrj3rb34+NdzvP8eEdHtaOpMdRcXF5GRkSGEqL8KzsnJSezbt+825r4LsX79eqFWq8W6devEiRMnxKxZs4Snp6fQ6/VCCCGeeOIJ8dJLL0ntf/vtN+Ho6ChWrVol0tLSxNKlS4WTk5M4duyY1GbFihXC09NTfP/99+Lo0aNi4sSJIjQ0VFRWVkptxo0bJwYNGiQSEhLEvn37RPfu3cWUKVOk5+Pi4oRCoRAvv/yyOHXqlEhKShJRUVEiODhYVFRUNLl/tnx13j3/2i2CF2wVu9Nz5S6FLOhsbql4bO1+6Qq+Ce/+Ko5ftL3fTyIia2rq53eTQ5RCoRA5OTnS925ubuLs2bMtr/CKd999V3Tu3FmoVCoxbNgwceDAAem50aNHi+nTp5u137hxo+jRo4dQqVSib9++4scffzR73mQyiSVLlgh/f3+hVqvF2LFjRXp6ulmbgoICMWXKFOHm5iY8PDzEjBkzRGlpqVmbr7/+WgwaNEi4uroKX19f8eCDD4q0tLRm9c1WQ5TJZBI9F28TwQu2inN5ZXKXQxZmNJrEVwmZot/SHSJ4wVbRZeGPYsX2NFFZUyd3aUREbUJTP78VQjRtlqlSqcSyZcvg5uYGAFiwYAFeeOEF+Pj4mLXjDYivMhgM0Gq1KCkpgYeHh9zlNFmuoQrD/hkHpQI4+ep9UDla7CJOakNyDVVYuuU4tqfWX5Ua0sEFr0zsh7t6+MpcGRGRvJr6+d3kEBUSEnLLS90VCoV01R7ZbohKPF+IP66NR0dPZ/z20h/kLoes7Ofjeiz5PhU5hmoAwLi+OiyZ0AcdPXllJhG1T039/G7y1Xnt7b5x7VkWb/fSrtzbV4fhXTvgrdhT+CI+EzuO67H7VC7mjOmGJ+/qArWjg9wlEhG1SRynoetkFtSHqGDeeLjd8NA4YemEvvjx2ZEYFuKNqloTVv18ClFv7cWu9MbvZUlE1N4xRNF1GlYr5zpC7U8vnQc2PDUcb08aCF93Nc4XVGDGZ4fw5BeJyMjniudERNdiiKLrNAzn8UxU+6RQKBA9qCN2zh+N/xsZCgelArEncnDvW3vwyg8nUFxRI3eJRERtAkMUXSeTc6IIgLvGCYsf6IMdz43C3T19UWsU+PS3DIx+Yzc+/vUcauq4UCcRtW8MUWSmssaIvNL6q7QYoggAuvu7Y92MYfjiz8PQS+eOkspaLPsxDfe8tQfbj13mvfiIqN1q8tV5DQwGQ6PbFQoF1Go1VCrVbRdF8skuqj8L5aFxhKcL30u66q4evrizmw82JWbjzdhTyCyowNNfHsbQEC8sGNcLQ0K85S6RiKhVNftMlKenJ7y8vK57eHp6wtnZGcHBwVi6dGm7ucecvWm4Mq8z50NRIxyU9Tc13v383Xj2D92gcVLi0Pki/HFtPP687hCOXyqRu0QiolbT7DNR69atw9///nf86U9/wrBhwwAABw8exOeff47FixcjLy8Pq1atglqtxqJFiyxeMFkX14iipnBVO2LevT0xJbwzVsedxsbEC9h5Mhc7T+bigf4BmHdPD3TxdZO7TCIiq2p2iPr888/x5ptv4rHHHpO2TZgwAWFhYfjwww8RFxeHzp0747XXXmOIskFZBfWXsXf2dpW5ErIFAVpnLH+4P2bd1RVvxZ7CliOXsPXoZWxP1eOPd3TCs5HdufI5EdmtZg/n7d+/H4MGDbpu+6BBgxAfHw8AGDlyJLKysm6/Omp1PBNFLRHq44rVUwZh27OjENnbD0aTwIbEbIx5YzeWbE7FxeJKuUskIrK4ZoeooKAgfPLJJ9dt/+STTxAUFAQAKCgogJeX1+1XR62OIYpuR59AD3w8fSi+fXoEhnfxRo3RhP8cyMTdb+zCwu+OSQu5EhHZg2YP561atQqPPvootm/fjqFDhwIAEhMTcfLkSXzzzTcAgEOHDmHSpEmWrZSszmQSyC6qP2PAhTbpdgwO9sLXTw5H/LkCvBt3BvHnCvD1wSxsSszGw3d0xF/v7oYQHw4ZE5FtU4gWLPKSkZGBDz/8EKdOnQIA9OzZE0899RRCQkIsXZ9Na+pdoNuKyyWViFi+Ew5KBdJfHQdHBy4jRpZx6HwhVsedxq+n8wEASgUQPbAj/jqmG7r5cQI6EbUtTf38blGIoqaxtRCVcK4Akz46gM7eLtj74hi5yyE7dDirCO/Gncau9DwAgEIB3NPbH0+N7oLBwVxniojahqZ+fjd7OA8AiouLcfDgQeTm5l63HtS0adNackhqAzJ5zzyysjs6e+GzGcNw7EIJ3t15Gj+fyJEeQ4K98NTorhjbyw9KpULuUomIbqnZIeqHH37A1KlTUVZWBg8PDygUV/+xUygUDFE2rGHSbxAnlZOVhXXS4qNpQ3Amtwwf/3oO3x2+iMTMIiR+kYiuvq546q6umDgoEGpHB7lLJSK6oWZPepk/fz7+/Oc/o6ysDMXFxSgqKpIehYWF1qiRWgmvzKPW1s3PDSse6Y99C8bg6bu7wl3jiLN55Xjx26MY9fourNl1BkXlNXKXSUTUqGaHqIsXL+LZZ5+Fiws/aO1Nwy1fghmiqJX5eWiwYFwv7H/pD/j7/b2h89Agt7Qab/yUjuHL4/DSt0eRdrnx+3YSEcml2SEqKioKiYmJ1qiFZMbhPJKbu8YJT97VBXtfHIM3Hx2AvoEeqK4zYf2hbNz3zq+Y/FE8dqTqYTTxehgikl+z50SNHz8eL7zwAk6cOIGwsDA4OTmZPf/ggw9arDhqPWXVdSi4MmzCmw+T3FSOSjwyuBMevqMjkjKL8Nn+89iRqseBc4U4cK4QHT2dMS0iGJOGBsHTRSV3uUTUTjV7iQOl8sYnrxQKBYxG420XZS9saYmDE5cMuH/1r/BycUJyzL1yl0N0ncsllfjvgUx8lZCFoopaAIDGSYkHBwTi8fBgDOikNbvQhYiopay2xMHvlzQg+8BJ5dTWBWid8UJULzzzh+7YknIJn+0/j7TLBmxMvICNiRfQJ8ADj4d3RvSgjnBTt2j1FiKiZuG/NAQAyCosBwB07sBbcVDbpnFywGNDg/DokE5IzCzCVwlZ+PHYZZy4bMDizan457Y0TBwYiMeHBSOsk1buconIjjUpRK1evRqzZs2CRqPB6tWrb9r22WeftUhh1LqunolylrkSoqZRKBQYGuKNoSHeiHmgD749fAFfHczCubxyfH0wG18fzEb/TlpMGdYZD/QPgLvG6dYHJSJqhibNiQoNDUViYiI6dOiA0NDQGx9MocC5c+csWqAts6U5UdM+PYi9p/Lw+iNhmDS0s9zlELWIEAIJGYX4KiEL21Mvo9ZY/8+bxkmJcX11eHRIECK6dOCK6ER0UxadE5WRkdHon8l+ZBVcGc7z5nAe2S6FQoHhXTpgeJcOKCirPzu1MfECzuSWYXPKJWxOuYSOns545I6OeGRwJwRz+JqIbgNvQGxFtnImymgS6Ll4O+pMAr+99Ad09OSQHtkPIQSOXCjBpsRsbDlyCaVVddJzw0K98ejgTrg/LACunIxORFc09fO72SHKaDRi3bp1iIuLa/QGxDt37mxZxXbIVkLUhaIKjHx9F5wcFDj56n1w4FAH2amqWiN+PpGDTYnZ2HcmHw3/+rmoHHBvH39MHNgRI7v7wMmh2esQE5EdsdoSB8899xzWrVuH8ePHo1+/flyXxQ5kXbndS5CXCwMU2TWNkwMeHBCIBwcE4nJJJb47fBHfJF1ARn65NNzn7arC+LAATBwYiDs6e3H+FBHdULND1Pr167Fx40bcf//91qiHZJDF271QOxSgdcbsMd3w17u7Ijm7GFtSLmHr0UvIL6vBfw5k4j8HMtHR0xkTBwZi4sCO6Klzl7tkImpjmh2iVCoVunXrZo1aSCZcaJPaM4VCgTs6e+GOzl5YPL43fjtbgO9TLuKnVD0uFlfi/d1n8f7us+ilc8eDAwPxQFggb41ERABaEKLmz5+Pd955B++99x6H8uxE5pUQFcwPBmrnHB2UGN3DF6N7+KLqISPi0nLxfcpF7E7Pw0l9KU7uSMfKHeno19ED9/ULwP1hAQj14RV+RO1Vs0PUvn37sGvXLmzfvh19+/a97gbE3333ncWKo9aRzeE8outonBwwvn8AxvcPQElFLbanXsaWI5dw4FwBUi8akHrRgDd+SkcvnTvGhwXgvrAAdPNzk7tsImpFzQ5Rnp6eeOihh6xRC8kki2eiiG5K6+KEycM6Y/Kwzigoq8bPJ3Kw7dhl7D9bUH+GSl+KN2NPoYe/m3SGqoe/G8/WE9m5ZoWouro6jBkzBvfeey90Op21aqJWVFJZi+KKWgD1V+cR0c11cFNjyrDOmDKsM4rKaxB7IgfbUi/jtzP5OJVThlM5p/FO3GkEd3BBZG9/3NPHH0OCveDIZROI7E6z/lY7OjriL3/5C6qrqy1WwJo1axASEgKNRoPw8HAcPHjwpu03bdqEXr16QaPRICwsDNu2bTN7XgiBmJgYBAQEwNnZGZGRkTh9+rRZm8LCQkydOhUeHh7w9PTEzJkzUVZWdt1xVq1ahR49ekCtVqNjx4547bXXLNPpNqRhKM/HTcXFBomayctVhceGBmHdjGFI/Ps9ePPRARjbyw8qRyUyCyrwyb4MTP7oAIa89gvmbUzB9mOXUV5dd+sDE5FNaPZ/jYYNG4bk5GSLvPiGDRswb948LF26FIcPH8aAAQMQFRWF3NzcRtvv378fU6ZMwcyZM5GcnIzo6GhER0cjNTVVarNy5UqsXr0aa9euRUJCAlxdXREVFYWqqiqpzdSpU3H8+HHExsZi69at2Lt3L2bNmmX2Ws899xw+/vhjrFq1CidPnsSWLVswbNgwi/S7LeGVeUSWoXVxwiODO+GTPw1F8pJ78MHUO/DwHR3h6eKE4opafHf4Ip7+8jAGvRqLGZ8dxFcJWcg1VN36wETUZjV7xfKNGzdi4cKF+Nvf/obBgwfD1dX8ypT+/fs3+Vjh4eEYOnQo3nvvPQCAyWRCUFAQnnnmGbz00kvXtZ80aRLKy8uxdetWadvw4cMxcOBArF27FkIIBAYGYv78+Xj++ecBACUlJfD398e6deswefJkpKWloU+fPjh06BCGDBkCANixYwfuv/9+XLhwAYGBgUhLS0P//v2RmpqKnj17NufHY8YWViz/YPdZvL7jJKIHBuLtyYPkLofI7tQZTUjMLELsiRzEnsiR/uPSYECQJ8b28sOYnn7oG+jBxT2J2gCrrVg+efJkAMCzzz4rbVMoFBBCQKFQwGg0Nuk4NTU1SEpKwsKFC6VtSqUSkZGRiI+Pb3Sf+Ph4zJs3z2xbVFQUNm/eDKD+5sh6vR6RkZHS81qtFuHh4YiPj8fkyZMRHx8PT09PKUABQGRkJJRKJRISEvDQQw/hhx9+QJcuXbB161aMGzcOQghERkZi5cqV8Pb2vmGfqqurzYY6DQZDk34WcuKZKCLrcnRQSjdFXjy+N07nliH2RA5+PpGDI9nF0uNfsafg46bC6B5+uLunL+7q7guti9OtX4CIZNPsEJWRkWGRF87Pz4fRaIS/v7/Zdn9/f5w8ebLRffR6faPt9Xq99HzDtpu18fPzM3ve0dER3t7eUptz584hMzMTmzZtwhdffAGj0Yi//e1v+OMf/3jTewMuX74cL7/88q263qZkFZYDADrzbvZEVqdQKNDD3x09/N0xe0w35BqqEHcyF7vTc7HvdD7yy2rw7eEL+PbwBSgVwB2dvTCmlx9G9/BF30APXu1H1MY0O0QFBwdbo442xWQyobq6Gl988QV69OgBAPjkk08wePBgpKen33CIb+HChWZnygwGA4KCglql5pbimSgi+fh5aKQr/WrqTEjMLMTu9DzsTs/FqZwyJGYWITGzCG/8lA5fdzXu7uGLUT18cWfXDujgppa7fKJ2r8WXY504cQJZWVmoqakx2/7ggw82aX8fHx84ODggJyfHbHtOTs4Nl0/Q6XQ3bd/wNScnBwEBAWZtBg4cKLX5/cT1uro6FBYWSvsHBATA0dFRClAA0Lt3bwBAVlbWDUOUWq2GWm07/7DVGk24VFw/sZUhikheKkclRnT1wYiuPlh0f29cKKrAnlN52HUyD7+dyUdeaTU2JV3ApqQLAIA+AR4Y1d0HI7v7YGiINzRODjL3gKj9aXaIOnfuHB566CEcO3ZMmgsFQDrN3NQ5USqVCoMHD0ZcXByio6MB1J8BiouLw5w5cxrdJyIiAnFxcZg7d660LTY2FhEREQCA0NBQ6HQ6xMXFSaHJYDAgISEBTz/9tHSM4uJiJCUlYfDgwQCAnTt3wmQyITw8HABw5513oq6uDmfPnkXXrl0BAKdOnQJgX2fiLhVXwmgSUDsq4eduO+GPqD3o5OWCqeHBmBoejOo6Iw5lFGHPqVz8ejofJ/WlOHHZgBOXDfhw7zmoHJUYGuKFkd18Maq7D/oEcII6UWto9tV5EyZMgIODAz7++GOEhobi4MGDKCgowPz587Fq1SqMGjWqycfasGEDpk+fjg8//BDDhg3D22+/jY0bN+LkyZPw9/fHtGnT0LFjRyxfvhxA/RIHo0ePxooVKzB+/HisX78e//znP3H48GH069cPAPD6669jxYoV+PzzzxEaGoolS5bg6NGjOHHiBDQaDQDgvvvuQ05ODtauXYva2lrMmDEDQ4YMwVdffQWgPswNHToUbm5uePvtt2EymTB79mx4eHjg559/bnL/2vrVeb+ezsMTnxxENz83/DJvtNzlEFET5ZVW47cz+dh3Jh/7TudD/7ulErxcnDCimw9Gdas/sxXk7cz5VETNYLWr8+Lj47Fz5074+PhAqVRCqVRi5MiRWL58OZ599tlmrSE1adIk5OXlISYmBnq9HgMHDsSOHTukieFZWVlQKq8uZTVixAh89dVXWLx4MRYtWoTu3btj8+bNUoACgBdffBHl5eWYNWsWiouLMXLkSOzYsUMKUADw5ZdfYs6cORg7diyUSiUeeeQRrF69WnpeqVTihx9+wDPPPIO77roLrq6uuO+++/Dmm28298fVpnE+FJFt8nVXI3pQR0QP6gghBM7mleHX0/WB6sC5AhRV1OLHo5fx49HLAIBArUa6QnB4lw4MVUQW0uwzUV5eXjh8+DBCQ0PRtWtXfPzxxxgzZgzOnj2LsLAwVFRU3Pog7URbPxO1fFsaPtx7Dn8aEYJ/PNhX7nKIyAJqjSakZBfj19P5+O1MPo5eKEat0fyfeYYqopuz2pmofv364ciRIwgNDUV4eDhWrlwJlUqFjz76CF26dLmtoql18UwUkf1xclBiaIg3hoZ4Y949PVBRU4fDmcU4cK4AB84V4MiFYlwqqcJ3yRfxXfJFAAxVRC3V7BC1ePFilJfXry30yiuv4IEHHsCoUaPQoUMHbNiwweIFkvU0hKjgDgxRRPbKReWIkVeu4gPQpFAVoNVgSIg3hgR7YXCwF3oHeMCBE9WJrtPs4bzGFBYWwsvLi/9z+Z22PJwnhED/f/yM0uo6xP7tLnT3d5e7JCKSQWWNEYeziqRQlZJ9/fCfm9oRgzp7YkiwN4aEeGFgkCdvWE52zWrDeQ3OnDmDs2fP4q677oK3tzcskMWoFRVX1KL0yt3kgzicR9RuOasccGc3H9zZrf5MVWWNEcnZRUg6X4RDmUVIzixCaXUdfj2dj19P5wMAHJQK9AnwwJAQLylY+XtobvYyRHap2SGqoKAAjz32GHbt2gWFQoHTp0+jS5cumDlzJry8vOzuCjZ71TCU5++h5iJ9RCRxVjlIi34CgNEkkK4vRVJmIQ6dL0Li+UJcKqnCsYslOHaxBJ/9dh4AEOTtjEFBXhjU2RMDgzzRJ9ADakf+20L2rdkh6m9/+xucnJyQlZUlreIN1C9XMG/ePIYoG5HJSeVE1AQOSgX6BHqgT6AHnogIAQBcLK5E4vlCJGUWIfF8EdL0BmQXViK7sBJbjlwCAKgclOgT6IGBQZ5SsOrs7cJpH2RXmh2ifv75Z/z000/o1KmT2fbu3bsjMzPTYoWRdWVfCVEcyiOi5uro6YyOAzti4sCOAIDSqlokZxUjJbsYyVlFSMkuRlFFLVKy67et21+/n7erCgODPKVg1b+TJ7TOTjL2hOj2NDtElZeXw8Xl+g/ewsJCm7pvXHuXWVB/hWWwt6vMlRCRrXPXOOGuHr64q4cvgPoLV7IKK66EqmIkZxfjxKUSFJbXYOfJXOw8efX+pV19XTEwyAsDg7To11GL3gEenGJANqPZIWrUqFH44osv8OqrrwKov2eeyWTCypUrMWbMGIsXSNYhrRHVwVnmSojI3igUCgR3cEVwB1fpbFV1nREnLhmkM1Yp2cXIKqzA2bxynM0rx7eH62+s7KhUoLu/O/p31KJfJy36d9Sip86dwYrapGaHqJUrV2Ls2LFITExETU0NXnzxRRw/fhyFhYX47bffrFEjWUF2YSUAzokiotahdnTAoM5eGNTZS9pWUFYtBapjF0tw7EIJCsprkHbZgLTLBmxIzAZQH6x6+Lujf6f6s1X9O9UHK05cJ7m1aMXyU6dO4b333oO7uzvKysrw8MMPY/bs2QgICLBGjWRh1XVGXCppCFEcziMieXRwU2Nsb3+M7V1/v1QhRP2VfxdKkHqxBEcv1n8tLK/BicsGnLhsAA7VBysnh6vBqk+gFn0C3NFL58H1q6hVtei3TavV4u9//7vZtgsXLmDWrFn46KOPLFIYWc/FokoIATg7OcDHTSV3OUREAOqHATt6OqOjpzPG9dMBqA9WF4sr60PVhfplFVIvlqCoohbHLxlw/JIBQPaV/YGQDq7oE1B/NWGfAA/0DvCAv4eaVwWSVVgsshcUFOCTTz5hiLIB194zj/+wEFFbplAo0MnLBZ28XDCuX/1ohxACF4oqpbNVaZcNOHHJgNzSamTklyMjvxw/HrssHcPbVWUWrPoEeqCLjyscHZRydYvsBM97tkNXJ5VzPhQR2R6FQoEgbxcEebvgvrCr00jyy6qlQHXiytezeWUoLK/BvjP52HcmX2qrclSip7/7lbNV7uihqx8O9Hbl2XlqOoaodiirgAttEpH98XFTY1R3X4zq7ittq6o14lROqVmwSrtsQHmNUVp1/ffH6KVzRw9/d/TUuaGnzgPd/dw414oaxd+KdqjhTFQwz0QRkZ3TODmgf6f6hT0bmEwC2UUVOHFlTtVJfSlO5ZQiq7AC+WXV2Hem2uysFVD/n85rg1VPf3d08XWFE4cE27Umh6iHH374ps8XFxffbi3USrK4WjkRtWNK5dV1rK4dDiyvrsPp3DKc0pdKweqkvhT5ZdXIKqxAVmEFfknLkdo7OSjQxccNPXTu6Onvhm5+7ujm54bgDi4MV+1Ek0OUVqu95fPTpk277YLIuhpWEgY4nEdEdC1XtaN0W5prFZRV41ROGdL1BqRf+Xoqpwxl1XVIzylFek4pfrimvZNDfUjr7ueGblceXX3rH84qrm1lT5ocoj777DNr1kGtpKC8BhU1RigUQCcvrlZORHQrHdzUiHBTI6JrB2lbw5pW6XoD0vVlOJ1TijN5ZTiTW4aKGiPO5Nb/+VoN/+52870arhrOXvEegraJc6Lamcwrk8oDPDRc7ZeIqIWuXdPqD738pe0mk8BlQ5UUos7klkp/LqqoRXZhJbILK7ErPc/seL7uanTzdUNXP1d08XFDqK8ruvq4oaOXMxyUXIqmrWKIameyOR+KiMhqlMqr4Wp0D1+z5wrKqnFaCldlOJtXhtM5ZdAbqpBXWo280mrEnysw20floETnDi7o4uMqBatQX1eE+riig6uKa/3JjCGqnWk4E8Ur84iIWlcHNzU6uKkxvEsHs+2lVbU4m1eOM7llOJdXhoz8cpzLK0dGQTlq6kyNDg0CgIfGEaG+bujqUx+qQn2vnMXyceXcq1bCENXOcFI5EVHb4q5xanRCu8kkcKmksj5QXVmJ/eyVkHWxuBKGqjocyS7Gkezi644ZqNUguIMrQnxc0NnbFSEdXNC5gwuCO7jCjWteWQx/ku0Mh/OIiGyDUnn1ljd3/W5osKrWiMyCCmTkl+HsNSHrXF793KtLJVW4VFJ13fAgAPi4qeqXePB2ubLUg8uVhyu8XJw4RNgMDFHtTGZhOQAguIOrzJUQEVFLaZwc0FPnjp469+ueKyqvwbn8cmQVluN8fv36VucLypFVUIGC8hrkl9U/kjKLrtvXXeMoBapgbxeEdHBF5w71X/3c1VBykrsZhqh2pKrWiBxDNQAO5xER2SsvVxUGu6owONjruucMVbXIKqhAZsHVYHW+oBxZhRW4XFKF0qo6pF40IPWi4bp91Y5KdL5yz8IgL2cEebugk5czOnnVb2uPyzQwRLUjF4rqh/Lc1I7wcml/v+xERO2dh8YJ/Tpq0a/j9QtoV9UakVVYH7AyC8qvBq3CClwoqkR1nQmnc8twupFJ7vXHdrwSsFwQ5H01ZAVdGZK0x8nuDFHtSOY1Nx7mmDcREV1L4+SAHv71N1/+vVqjCZeKK5FZUIHsogpkF1biQlEFsosqcaGwfpjQUFWH41fuR9gYHzd1fbi6ErI6eV0NXIGezjZ5qxyGqHaEV+YREVFLODkopfsNNqa8ug4XiiqRXVghhavswqshq7S6Dvll1cgvq0ZyVvF1+ysUgL+7Bh296tfYuu6rpzNc2+BVhW2vIrIaKURxjSgiIrIgV7XjDSe6CyFQUlkrhayGM1n1X68OFeoNVdAbqhqd8A4Ani5OUqC6NlyN6eUHjZM8Q4UMUe1IVgHPRBERUetSKBTwdFHB00XV6FwsIQTyy2pwsbgSF4sqcbG44srXSly48rW0qg7FFbUorqi9brgw9eWo1urKdRii2hEO5xERUVujUCjg666Gr7v6ugVHGxiqanFJCln1Xy8UV6KkolbWxUMZotoJIYQUonjLFyIisiUeGid46JzQS+chdylmbG8qPLVIbmk1qutMUCqAQE9nucshIiKyeQxR7UTDWShbvYyUiIioreGnaTvRsEYUh/KIiIgsgyGqneCkciIiIstqEyFqzZo1CAkJgUajQXh4OA4ePHjT9ps2bUKvXr2g0WgQFhaGbdu2mT0vhEBMTAwCAgLg7OyMyMhInD592qxNYWEhpk6dCg8PD3h6emLmzJkoK2t8KfszZ87A3d0dnp6et9VPOWVfCVFBDFFEREQWIXuI2rBhA+bNm4elS5fi8OHDGDBgAKKiopCbm9to+/3792PKlCmYOXMmkpOTER0djejoaKSmpkptVq5cidWrV2Pt2rVISEiAq6sroqKiUFVVJbWZOnUqjh8/jtjYWGzduhV79+7FrFmzrnu92tpaTJkyBaNGjbJ851tRZkE5ACDYu/HVZomIiKh5FEIIIWcB4eHhGDp0KN577z0AgMlkQlBQEJ555hm89NJL17WfNGkSysvLsXXrVmnb8OHDMXDgQKxduxZCCAQGBmL+/Pl4/vnnAQAlJSXw9/fHunXrMHnyZKSlpaFPnz44dOgQhgwZAgDYsWMH7r//fly4cAGBgYHSsRcsWIBLly5h7NixmDt3LoqLi5vcN4PBAK1Wi5KSEnh4yHtZ5pBlvyC/rBo/zBmJsE7XL3ZGRERE9Zr6+S3rmaiamhokJSUhMjJS2qZUKhEZGYn4+PhG94mPjzdrDwBRUVFS+4yMDOj1erM2Wq0W4eHhUpv4+Hh4enpKAQoAIiMjoVQqkZCQIG3buXMnNm3ahDVr1jSpP9XV1TAYDGaPtqCipv6eRQDnRBEREVmKrCEqPz8fRqMR/v7+Ztv9/f2h1+sb3Uev19+0fcPXW7Xx8/Mze97R0RHe3t5Sm4KCAvzpT3/CunXrmnwWafny5dBqtdIjKCioSftZW8Okcq2zE7QuTjJXQ0REZB9knxPVVj355JN4/PHHcddddzV5n4ULF6KkpER6ZGdnW7HCpuM984iIiCxP1hDl4+MDBwcH5OTkmG3PycmBTqdrdB+dTnfT9g1fb9Xm9xPX6+rqUFhYKLXZuXMnVq1aBUdHRzg6OmLmzJkoKSmBo6MjPv3000ZrU6vV8PDwMHu0BVzegIiIyPJkDVEqlQqDBw9GXFyctM1kMiEuLg4RERGN7hMREWHWHgBiY2Ol9qGhodDpdGZtDAYDEhISpDYREREoLi5GUlKS1Gbnzp0wmUwIDw8HUD9vKiUlRXq88sorcHd3R0pKCh566CHL/ABaiRSiuNAmERGRxch+A+J58+Zh+vTpGDJkCIYNG4a3334b5eXlmDFjBgBg2rRp6NixI5YvXw4AeO655zB69Gi8+eabGD9+PNavX4/ExER89NFHAOrvBj137lwsW7YM3bt3R2hoKJYsWYLAwEBER0cDAHr37o1x48bhySefxNq1a1FbW4s5c+Zg8uTJ0pV5vXv3NqszMTERSqUS/fr1a6WfjOXwTBQREZHlyR6iJk2ahLy8PMTExECv12PgwIHYsWOHNDE8KysLSuXVE2YjRozAV199hcWLF2PRokXo3r07Nm/ebBZuXnzxRZSXl2PWrFkoLi7GyJEjsWPHDmg0GqnNl19+iTlz5mDs2LFQKpV45JFHsHr16tbreCtqCFHBDFFEREQWI/s6UfasLawTZTQJ9F6yAzVGE359cQxXLCciIroFm1gniqwvx1CFGqMJjkoFArSaW+9ARERETcIQZecahvI6eTnD0YFvNxERkaXwU9XONawRxWE8IiIiy2KIsnO8Mo+IiMg6GKLsXGbDlXlcI4qIiMiiGKLsHM9EERERWQdDlJ3LLuScKCIiImtgiLJjpVW1KCyvAcAzUURERJbGEGXHGobyvF1VcNc4yVwNERGRfWGIsmMcyiMiIrIehig7llnAe+YRERFZC0OUHeOVeURERNbDEGXHpBDFNaKIiIgsjiHKjvFMFBERkfUwRNmpOqMJF4sqATBEERERWQNDlJ26XFKFOpOAykEJnYdG7nKIiIjsDkOUnWoYyuvk7QylUiFzNURERPaHIcpOcT4UERGRdTFE2SmuEUVERGRdDFF2iquVExERWRdDlJ3icB4REZF1MUTZqcyCcgBAcAdXmSshIiKyTwxRdqikohaGqjoAQJC3s8zVEBER2SeGKDvUMJTn46aGi8pR5mqIiIjsE0OUHcosbBjK43woIiIia2GIskOcVE5ERGR9DFF2KJshioiIyOoYouxQw0KbDFFERETWwxBlh6ThPM6JIiIishqGKDtTazThUnElAN7yhYiIyJoYouzMxaJKmASgdlTC110tdzlERER2iyHKzlx7ZZ5CoZC5GiIiIvvFEGVnMq+EKK4RRUREZF0MUXamYXmDIM6HIiIisiqGKDuTxeUNiIiIWgVDlJ3hcB4REVHrYIiyI0IIrlZORETUStpEiFqzZg1CQkKg0WgQHh6OgwcP3rT9pk2b0KtXL2g0GoSFhWHbtm1mzwshEBMTg4CAADg7OyMyMhKnT582a1NYWIipU6fCw8MDnp6emDlzJsrKyqTnd+/ejYkTJyIgIACurq4YOHAgvvzyS8t12gqKKmpRVl0HAOjkxRBFRERkTbKHqA0bNmDevHlYunQpDh8+jAEDBiAqKgq5ubmNtt+/fz+mTJmCmTNnIjk5GdHR0YiOjkZqaqrUZuXKlVi9ejXWrl2LhIQEuLq6IioqClVVVVKbqVOn4vjx44iNjcXWrVuxd+9ezJo1y+x1+vfvj2+//RZHjx7FjBkzMG3aNGzdutV6P4zblFlQDgDQeWigcXKQuRoiIiI7J2Q2bNgwMXv2bOl7o9EoAgMDxfLlyxtt/9hjj4nx48ebbQsPDxdPPfWUEEIIk8kkdDqdeOONN6Tni4uLhVqtFl9//bUQQogTJ04IAOLQoUNSm+3btwuFQiEuXrx4w1rvv/9+MWPGjCb3raSkRAAQJSUlTd7ndmxOviCCF2wVj36wv1Vej4iIyB419fNb1jNRNTU1SEpKQmRkpLRNqVQiMjIS8fHxje4THx9v1h4AoqKipPYZGRnQ6/VmbbRaLcLDw6U28fHx8PT0xJAhQ6Q2kZGRUCqVSEhIuGG9JSUl8Pb2vuHz1dXVMBgMZo/WxOUNiIiIWo+sISo/Px9GoxH+/v5m2/39/aHX6xvdR6/X37R9w9dbtfHz8zN73tHREd7e3jd83Y0bN+LQoUOYMWPGDfuzfPlyaLVa6REUFHTDttaQWcAr84iIiFqL7HOibMGuXbswY8YM/Pvf/0bfvn1v2G7hwoUoKSmRHtnZ2a1YpfktX4iIiMi6ZA1RPj4+cHBwQE5Ojtn2nJwc6HS6RvfR6XQ3bd/w9VZtfj9xva6uDoWFhde97p49ezBhwgS89dZbmDZt2k37o1ar4eHhYfZoTdLyBjwTRUREZHWyhiiVSoXBgwcjLi5O2mYymRAXF4eIiIhG94mIiDBrDwCxsbFS+9DQUOh0OrM2BoMBCQkJUpuIiAgUFxcjKSlJarNz506YTCaEh4dL23bv3o3x48fj9ddfN7tyry2qrjPisqH+6kOeiSIiIrI+R7kLmDdvHqZPn44hQ4Zg2LBhePvtt1FeXi7NPZo2bRo6duyI5cuXAwCee+45jB49Gm+++SbGjx+P9evXIzExER999BEAQKFQYO7cuVi2bBm6d++O0NBQLFmyBIGBgYiOjgYA9O7dG+PGjcOTTz6JtWvXora2FnPmzMHkyZMRGBgIoH4I74EHHsBzzz2HRx55RJorpVKpbjq5XC4XiiohBOCickAHV5Xc5RAREdm/Vrpa8Kbeffdd0blzZ6FSqcSwYcPEgQMHpOdGjx4tpk+fbtZ+48aNokePHkKlUom+ffuKH3/80ex5k8kklixZIvz9/YVarRZjx44V6enpZm0KCgrElClThJubm/Dw8BAzZswQpaWl0vPTp08XAK57jB49usn9as0lDnam5YjgBVtF1Ft7rP5aRERE9qypn98KIYSQMcPZNYPBAK1Wi5KSEqvPj/p8/3ks3XIc9/bxx0fThtx6ByIiImpUUz+/eXWeneCVeURERK2LIcpOcI0oIiKi1sUQZSe4WjkREVHrYoiyA0IIDucRERG1MoYoO5BXVo3KWiMUCqCTF0MUERFRa2CIsgMNQ3mBWmeoHPmWEhERtQZ+4tqBLGk+lLPMlRAREbUfDFF2QLoyz9tV5kqIiIjaD4YoO5DFGw8TERG1OoYoO5DNK/OIiIhaHUOUHWgYzmOIIiIiaj0MUTaussaI3NJqAAxRRERErYkhysZdKKo/C+WucYSni5PM1RAREbUfDFE27tqhPIVCIXM1RERE7QdDlI3j7V6IiIjkwRBl47i8ARERkTwYomwcz0QRERHJgyHKxjFEERERyYMhyoaZTEIKUbzlCxERUetiiLJhuaXVqKkzwUGpQICnRu5yiIiI2hWGKBvWcBYq0FMDJwe+lURERK2Jn7w2LLOgHACH8oiIiOTAEGXDGm48HMRJ5URERK2OIcqGSZPKuUYUERFRq2OIsmGZXN6AiIhINgxRNiybIYqIiEg2DFE2qry6DvllNQB4yxciIiI5METZqIb5UJ4uTvDQOMlcDRERUfvDEGWjeLsXIiIieTFE2aisAoYoIiIiOTFE2SieiSIiIpIXQ5SNYogiIiKSF0OUjZJCFK/MIyIikgVDlA0ymgQuFPFMFBERkZwYomyQ3lCFWqOAk4MCAVpnucshIiJqlxiibFBmQTkAoJOXCxyUCpmrISIiap8YomxQw+1egjiUR0REJJs2EaLWrFmDkJAQaDQahIeH4+DBgzdtv2nTJvTq1QsajQZhYWHYtm2b2fNCCMTExCAgIADOzs6IjIzE6dOnzdoUFhZi6tSp8PDwgKenJ2bOnImysjKzNkePHsWoUaOg0WgQFBSElStXWqbDt+nqlXkcyiMiIpKL7CFqw4YNmDdvHpYuXYrDhw9jwIABiIqKQm5ubqPt9+/fjylTpmDmzJlITk5GdHQ0oqOjkZqaKrVZuXIlVq9ejbVr1yIhIQGurq6IiopCVVWV1Gbq1Kk4fvw4YmNjsXXrVuzduxezZs2SnjcYDLj33nsRHByMpKQkvPHGG/jHP/6Bjz76yHo/jCbKvLLQZrC3q8yVEBERtWNCZsOGDROzZ8+WvjcajSIwMFAsX7680faPPfaYGD9+vNm28PBw8dRTTwkhhDCZTEKn04k33nhDer64uFio1Wrx9ddfCyGEOHHihAAgDh06JLXZvn27UCgU4uLFi0IIId5//33h5eUlqqurpTYLFiwQPXv2bHLfSkpKBABRUlLS5H2a4sF3fxXBC7aK7ccuW/S4RERE1PTPb1nPRNXU1CApKQmRkZHSNqVSicjISMTHxze6T3x8vFl7AIiKipLaZ2RkQK/Xm7XRarUIDw+X2sTHx8PT0xNDhgyR2kRGRkKpVCIhIUFqc9ddd0GlUpm9Tnp6OoqKihqtrbq6GgaDwexhDQ3DecFcI4qIiEg2soao/Px8GI1G+Pv7m2339/eHXq9vdB+9Xn/T9g1fb9XGz8/P7HlHR0d4e3ubtWnsGNe+xu8tX74cWq1WegQFBTXe8dtQWWOEyrH+bePEciIiIvnIPifKnixcuBAlJSXSIzs72+Kv4axyQMKiSJx8dRzc1I4WPz4RERE1jawhysfHBw4ODsjJyTHbnpOTA51O1+g+Op3upu0bvt6qze8nrtfV1aGwsNCsTWPHuPY1fk+tVsPDw8PsYS0aJwerHZuIiIhuTdYQpVKpMHjwYMTFxUnbTCYT4uLiEBER0eg+ERERZu0BIDY2VmofGhoKnU5n1sZgMCAhIUFqExERgeLiYiQlJUltdu7cCZPJhPDwcKnN3r17UVtba/Y6PXv2hJeX1232nIiIiGxeK010v6H169cLtVot1q1bJ06cOCFmzZolPD09hV6vF0II8cQTT4iXXnpJav/bb78JR0dHsWrVKpGWliaWLl0qnJycxLFjx6Q2K1asEJ6enuL7778XR48eFRMnThShoaGisrJSajNu3DgxaNAgkZCQIPbt2ye6d+8upkyZIj1fXFws/P39xRNPPCFSU1PF+vXrhYuLi/jwww+b3DdrXZ1HRERE1tPUz2/ZQ5QQQrz77ruic+fOQqVSiWHDhokDBw5Iz40ePVpMnz7drP3GjRtFjx49hEqlEn379hU//vij2fMmk0ksWbJE+Pv7C7VaLcaOHSvS09PN2hQUFIgpU6YINzc34eHhIWbMmCFKS0vN2hw5ckSMHDlSqNVq0bFjR7FixYpm9YshioiIyPY09fNbIYQQ8p4Ls18GgwFarRYlJSVWnR9FREREltPUz29enUdERETUAgxRRERERC3AEEVERETUAgxRRERERC3AEEVERETUAgxRRERERC3AEEVERETUAgxRRERERC3AEEVERETUAo5yF2DPGhaDNxgMMldCRERETdXwuX2rm7owRFlRaWkpACAoKEjmSoiIiKi5SktLodVqb/g8751nRSaTCZcuXYK7uzsUCoXFjmswGBAUFITs7Gy7vCefvfcPsP8+2nv/APvvI/tn++y9j9bsnxACpaWlCAwMhFJ545lPPBNlRUqlEp06dbLa8T08POzyL0YDe+8fYP99tPf+AfbfR/bP9tl7H63Vv5udgWrAieVERERELcAQRURERNQCDFE2SK1WY+nSpVCr1XKXYhX23j/A/vto7/0D7L+P7J/ts/c+toX+cWI5ERERUQvwTBQRERFRCzBEEREREbUAQxQRERFRCzBEEREREbUAQ5QNWrNmDUJCQqDRaBAeHo6DBw/KXdJ1/vGPf0ChUJg9evXqJT1fVVWF2bNno0OHDnBzc8MjjzyCnJwcs2NkZWVh/PjxcHFxgZ+fH1544QXU1dWZtdm9ezfuuOMOqNVqdOvWDevWrbNKf/bu3YsJEyYgMDAQCoUCmzdvNnteCIGYmBgEBATA2dkZkZGROH36tFmbwsJCTJ06FR4eHvD09MTMmTNRVlZm1ubo0aMYNWoUNBoNgoKCsHLlyutq2bRpE3r16gWNRoOwsDBs27atVfr4pz/96br3dNy4cTbTx+XLl2Po0KFwd3eHn58foqOjkZ6ebtamNX8vLf33uCn9u/vuu697D//yl7/YRP8++OAD9O/fX1pYMSIiAtu3b5eet+X3rql9tOX3rzErVqyAQqHA3LlzpW029z4Ksinr168XKpVKfPrpp+L48ePiySefFJ6eniInJ0fu0swsXbpU9O3bV1y+fFl65OXlSc//5S9/EUFBQSIuLk4kJiaK4cOHixEjRkjP19XViX79+onIyEiRnJwstm3bJnx8fMTChQulNufOnRMuLi5i3rx54sSJE+Ldd98VDg4OYseOHRbvz7Zt28Tf//538d133wkA4n//+5/Z8ytWrBBarVZs3rxZHDlyRDz44IMiNDRUVFZWSm3GjRsnBgwYIA4cOCB+/fVX0a1bNzFlyhTp+ZKSEuHv7y+mTp0qUlNTxddffy2cnZ3Fhx9+KLX57bffhIODg1i5cqU4ceKEWLx4sXBychLHjh2zeh+nT58uxo0bZ/aeFhYWmrVpy32MiooSn332mUhNTRUpKSni/vvvF507dxZlZWVSm9b6vbTG3+Om9G/06NHiySefNHsPS0pKbKJ/W7ZsET/++KM4deqUSE9PF4sWLRJOTk4iNTVVCGHb711T+2jL79/vHTx4UISEhIj+/fuL5557Ttpua+8jQ5SNGTZsmJg9e7b0vdFoFIGBgWL58uUyVnW9pUuXigEDBjT6XHFxsXBychKbNm2StqWlpQkAIj4+XghR/4GuVCqFXq+X2nzwwQfCw8NDVFdXCyGEePHFF0Xfvn3Njj1p0iQRFRVl4d6Y+33AMJlMQqfTiTfeeEPaVlxcLNRqtfj666+FEEKcOHFCABCHDh2S2mzfvl0oFApx8eJFIYQQ77//vvDy8pL6J4QQCxYsED179pS+f+yxx8T48ePN6gkPDxdPPfWUVfsoRH2Imjhx4g33sbU+5ubmCgBiz549QojW/b1sjb/Hv++fEPUfwtd+YP2eLfVPCCG8vLzExx9/bHfvXWN9FMJ+3r/S0lLRvXt3ERsba9YnW3wfOZxnQ2pqapCUlITIyEhpm1KpRGRkJOLj42WsrHGnT59GYGAgunTpgqlTpyIrKwsAkJSUhNraWrN+9OrVC507d5b6ER8fj7CwMPj7+0ttoqKiYDAYcPz4canNtcdoaNPaP4uMjAzo9XqzWrRaLcLDw8364+npiSFDhkhtIiMjoVQqkZCQILW56667oFKppDZRUVFIT09HUVGR1EbOPu/evRt+fn7o2bMnnn76aRQUFEjP2VofS0pKAADe3t4AWu/3srX+Hv++fw2+/PJL+Pj4oF+/fli4cCEqKiqk52ylf0ajEevXr0d5eTkiIiLs7r1rrI8N7OH9mz17NsaPH39dHbb4PvIGxDYkPz8fRqPR7JcHAPz9/XHy5EmZqmpceHg41q1bh549e+Ly5ct4+eWXMWrUKKSmpkKv10OlUsHT09NsH39/f+j1egCAXq9vtJ8Nz92sjcFgQGVlJZydna3UO3MN9TRWy7W1+vn5mT3v6OgIb29vszahoaHXHaPhOS8vrxv2ueEY1jRu3Dg8/PDDCA0NxdmzZ7Fo0SLcd999iI+Ph4ODg0310WQyYe7cubjzzjvRr18/6fVb4/eyqKjI6n+PG+sfADz++OMIDg5GYGAgjh49igULFiA9PR3fffedTfTv2LFjiIiIQFVVFdzc3PC///0Pffr0QUpKit28dzfqI2D77x8ArF+/HocPH8ahQ4eue84W/w4yRJFV3HfffdKf+/fvj/DwcAQHB2Pjxo2tFm7IsiZPniz9OSwsDP3790fXrl2xe/dujB07VsbKmm/27NlITU3Fvn375C7FKm7Uv1mzZkl/DgsLQ0BAAMaOHYuzZ8+ia9eurV1ms/Xs2RMpKSkoKSnBN998g+nTp2PPnj1yl2VRN+pjnz59bP79y87OxnPPPYfY2FhoNBq5y7EIDufZEB8fHzg4OFx3pUJOTg50Op1MVTWNp6cnevTogTNnzkCn06GmpgbFxcVmba7th06na7SfDc/drI2Hh0erBrWGem72vuh0OuTm5po9X1dXh8LCQov0WY73v0uXLvDx8cGZM2ek2myhj3PmzMHWrVuxa9cudOrUSdreWr+X1v57fKP+NSY8PBwAzN7Dttw/lUqFbt26YfDgwVi+fDkGDBiAd955x27eu5v1sTG29v4lJSUhNzcXd9xxBxwdHeHo6Ig9e/Zg9erVcHR0hL+/v829jwxRNkSlUmHw4MGIi4uTtplMJsTFxZmNmbdFZWVlOHv2LAICAjB48GA4OTmZ9SM9PR1ZWVlSPyIiInDs2DGzD+XY2Fh4eHhIp7YjIiLMjtHQprV/FqGhodDpdGa1GAwGJCQkmPWnuLgYSUlJUpudO3fCZDJJ/xBGRERg7969qK2tldrExsaiZ8+e8PLyktq0hT4DwIULF1BQUICAgACptrbcRyEE5syZg//973/YuXPndcOKrfV7aa2/x7fqX2NSUlIAwOw9bKv9a4zJZEJ1dbXNv3dN6WNjbO39Gzt2LI4dO4aUlBTpMWTIEEydOlX6s829j82ahk6yW79+vVCr1WLdunXixIkTYtasWcLT09PsSoW2YP78+WL37t0iIyND/PbbbyIyMlL4+PiI3NxcIUT9ZaydO3cWO3fuFImJiSIiIkJERERI+zdcxnrvvfeKlJQUsWPHDuHr69voZawvvPCCSEtLE2vWrLHaEgelpaUiOTlZJCcnCwDiX//6l0hOThaZmZlCiPolDjw9PcX3338vjh49KiZOnNjoEgeDBg0SCQkJYt++faJ79+5ml/8XFxcLf39/8cQTT4jU1FSxfv164eLict3l/46OjmLVqlUiLS1NLF261GJLHNysj6WlpeL5558X8fHxIiMjQ/zyyy/ijjvuEN27dxdVVVU20cenn35aaLVasXv3brNLxCsqKqQ2rfV7aY2/x7fq35kzZ8Qrr7wiEhMTRUZGhvj+++9Fly5dxF133WUT/XvppZfEnj17REZGhjh69Kh46aWXhEKhED///LMQwrbfu6b00dbfvxv5/RWHtvY+MkTZoHfffVd07txZqFQqMWzYMHHgwAG5S7rOpEmTREBAgFCpVKJjx45i0qRJ4syZM9LzlZWV4q9//avw8vISLi4u4qGHHhKXL182O8b58+fFfffdJ5ydnYWPj4+YP3++qK2tNWuza9cuMXDgQKFSqUSXLl3EZ599ZpX+7Nq1SwC47jF9+nQhRP0yB0uWLBH+/v5CrVaLsWPHivT0dLNjFBQUiClTpgg3Nzfh4eEhZsyYIUpLS83aHDlyRIwcOVKo1WrRsWNHsWLFiutq2bhxo+jRo4dQqVSib9++4scff7R6HysqKsS9994rfH19hZOTkwgODhZPPvnkdf/gtOU+NtY3AGa/M635e2npv8e36l9WVpa46667hLe3t1Cr1aJbt27ihRdeMFtnqC33789//rMIDg4WKpVK+Pr6irFjx0oBSgjbfu+a0kdbf/9u5PchytbeR4UQQjTv3BURERERcU4UERERUQswRBERERG1AEMUERERUQswRBERERG1AEMUERERUQswRBERERG1AEMUERERUQswRBERERG1AEMUERGAkJAQvP3223KXQUQ2hCGKiGyKQqG46eMf//hHi4576NAhzJo167Zqy8jIwOOPP47AwEBoNBp06tQJEydOxMmTJwEA58+fh0KhkG4cS0S2zVHuAoiImuPy5cvSnzds2ICYmBikp6dL29zc3KQ/CyFgNBrh6Hjrf+p8fX1vq67a2lrcc8896NmzJ7777jsEBATgwoUL2L59O4qLi2/r2ETUNvFMFBHZFJ1OJz20Wi0UCoX0/cmTJ+Hu7o7t27dj8ODBUKvV2LdvH86ePYuJEyfC398fbm5uGDp0KH755Rez4/5+OE+hUODjjz/GQw89BBcXF3Tv3h1btmy5YV3Hjx/H2bNn8f7772P48OEIDg7GnXfeiWXLlmH48OEAgNDQUADAoEGDoFAocPfdd0v7f/zxx+jduzc0Gg169eqF999/X3qu4QzW+vXrMWLECGg0GvTr1w979uyxwE+UiFqKIYqI7M5LL72EFStWIC0tDf3790dZWRnuv/9+xMXFITk5GePGjcOECROQlZV10+O8/PLLeOyxx3D06FHcf//9mDp1KgoLCxtt6+vrC6VSiW+++QZGo7HRNgcPHgQA/PLLL7h8+TK+++47AMCXX36JmJgYvPbaa0hLS8M///lPLFmyBJ9//rnZ/i+88ALmz5+P5ORkREREYMKECSgoKGjuj4eILEUQEdmozz77TGi1Wun7Xbt2CQBi8+bNt9y3b9++4t1335W+Dw4OFm+99Zb0PQCxePFi6fuysjIBQGzfvv2Gx3zvvfeEi4uLcHd3F2PGjBGvvPKKOHv2rPR8RkaGACCSk5PN9uvatav46quvzLa9+uqrIiIiwmy/FStWSM/X1taKTp06iddff/2WfSUi6+CZKCKyO0OGDDH7vqysDM8//zx69+4NT09PuLm5IS0t7ZZnovr37y/92dXVFR4eHsjNzb1h+9mzZ0Ov1+PLL79EREQENm3ahL59+yI2NvaG+5SXl+Ps2bOYOXMm3NzcpMeyZctw9uxZs7YRERHSnx0dHTFkyBCkpaXdtA9EZD2cWE5EdsfV1dXs++effx6xsbFYtWoVunXrBmdnZ/zxj39ETU3NTY/j5ORk9r1CoYDJZLrpPu7u7pgwYQImTJiAZcuWISoqCsuWLcM999zTaPuysjIAwL///W+Eh4ebPefg4HDT1yIiefFMFBHZvd9++w1/+tOf8NBDDyEsLAw6nQ7nz5+3+usqFAr06tUL5eXlAACVSgUAZnOm/P39ERgYiHPnzqFbt25mj4aJ6A0OHDgg/bmurg5JSUno3bu31ftBRI3jmSgisnvdu3fHd999hwkTJkChUGDJkiW3PKPUXCkpKVi6dCmeeOIJ9OnTByqVCnv27MGnn36KBQsWAAD8/Pzg7OyMHTt2oFOnTtBoNNBqtXj55Zfx7LPPQqvVYty4caiurkZiYiKKioowb9486TXWrFmD7t27o3fv3njrrbdQVFSEP//5zxbtBxE1HUMUEdm9f/3rX/jzn/+MESNGwMfHBwsWLIDBYLDoa3Tq1AkhISF4+eWXpSUJGr7/29/+BqB+HtPq1avxyiuvICYmBqNGjcLu3bvxf//3f3BxccEbb7yBF154Aa6urggLC8PcuXPNXmPFihVYsWIFUlJS0K1bN2zZsgU+Pj4W7QcRNZ1CCCHkLoKIiG7s/PnzCA0NRXJyMgYOHCh3OUR0BedEEREREbUAQxQRERFRC3A4j4iIiKgFeCaKiIiIqAUYooiIiIhagCGKiIiIqAUYooiIiIhagCGKiIiIqAUYooiIiIhagCGKiIiIqAUYooiIiIha4P8DAa3C4Nz+Mz4AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.plot(learning_rate(tf.range(40000, dtype=tf.float32)))\n", "plt.ylabel('Learning Rate')\n", "plt.xlabel('Train Step')" ] }, { "cell_type": "markdown", "id": "4cfba386", "metadata": {}, "source": [ "Next, you set up the loss. Since the target sequences are padded, it is important to apply a padding mask when calculating the loss.\n", "\n", "You will use the sparse categorical cross-entropy loss function (`tf.keras.losses.SparseCategoricalCrossentropy`) and set the parameter `from_logits` to False since the Transformer does not output raw logits since the last layer has a softmax activation:" ] }, { "cell_type": "code", "execution_count": 27, "id": "99fc8885", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False, reduction='none')\n", "\n", "def masked_loss(real, pred):\n", " mask = tf.math.logical_not(tf.math.equal(real, 0))\n", " loss_ = loss_object(real, pred)\n", "\n", " mask = tf.cast(mask, dtype=loss_.dtype)\n", " loss_ *= mask\n", "\n", " return tf.reduce_sum(loss_)/tf.reduce_sum(mask)\n", "\n", "\n", "train_loss = tf.keras.metrics.Mean(name='train_loss')\n", "\n", "# Here you will store the losses, so you can later plot them\n", "losses = []" ] }, { "cell_type": "markdown", "id": "33db3f0b", "metadata": {}, "source": [ "Now you can define your custom training function. If you are not very advanced with tensorflow, you can understand this function as an alternative to using `model.compile()` and `model.fit()`, but with added extra flexibility." ] }, { "cell_type": "code", "execution_count": 28, "id": "79092091", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "@tf.function\n", "def train_step(model, inp, tar):\n", " \"\"\"\n", " One training step for the transformer\n", " Arguments:\n", " inp (tf.Tensor): Input data to summarize\n", " tar (tf.Tensor): Target (summary)\n", " Returns:\n", " None\n", " \"\"\"\n", " tar_inp = tar[:, :-1]\n", " tar_real = tar[:, 1:]\n", "\n", " # Create masks\n", " enc_padding_mask = create_padding_mask(inp)\n", " look_ahead_mask = create_look_ahead_mask(tf.shape(tar_inp)[1])\n", " dec_padding_mask = create_padding_mask(inp) # Notice that both encoder and decoder padding masks are equal\n", "\n", " with tf.GradientTape() as tape:\n", " predictions, _ = model(\n", " inp,\n", " tar_inp, \n", " True, \n", " enc_padding_mask, \n", " look_ahead_mask, \n", " dec_padding_mask\n", " )\n", " loss = masked_loss(tar_real, predictions)\n", "\n", " gradients = tape.gradient(loss, transformer.trainable_variables) \n", " optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n", "\n", " train_loss(loss)" ] }, { "cell_type": "markdown", "id": "1480d5fd", "metadata": {}, "source": [ "Now you are ready for training the model. But before starting the training, you can also define one more set of functions to perform the inference. Because you are using a custom training loop, you can do whatever you want between the training steps. And wouldnt't it be fun to see after each epoch some examples of how the model performs?" ] }, { "cell_type": "markdown", "id": "79e05c54", "metadata": {}, "source": [ "\n", "## 11 - Summarization\n", "\n", "The last thing you will implement is inference. With this, you will be able to produce actual summaries of the documents. You will use a simple method called greedy decoding, which means you will predict one word at a time and append it to the output. You will start with an `[SOS]` token and repeat the word by word inference until the model returns you the `[EOS]` token or until you reach the maximum length of the sentence (you need to add this limit, otherwise a poorly trained model could give you infinite sentences without ever producing the `[EOS]` token.\n", "\n", " \n", "### Exercise 5 - next_word\n", "Write a helper function that predicts the next word, so you can use it to write the whole sentences. Hint: this is very similar to what happens in the train_step, but you have to set the training of the model to False." ] }, { "cell_type": "code", "execution_count": 29, "id": "175fae70", "metadata": { "deletable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "# GRADED FUNCTION: next_word\n", "def next_word(model, encoder_input, output):\n", " \"\"\"\n", " Helper function for summarization that uses the model to predict just the next word.\n", " Arguments:\n", " encoder_input (tf.Tensor): Input data to summarize\n", " output (tf.Tensor): (incomplete) target (summary)\n", " Returns:\n", " predicted_id (tf.Tensor): The id of the predicted word\n", " \"\"\"\n", " ### START CODE HERE ###\n", " # Create a padding mask for the input (encoder)\n", " enc_padding_mask = create_padding_mask(encoder_input)\n", " # Create a look-ahead mask for the output\n", " look_ahead_mask = create_look_ahead_mask(tf.shape(output)[1])\n", " # Create a padding mask for the input (decoder)\n", " dec_padding_mask = create_padding_mask(encoder_input)\n", "\n", " # Run the prediction of the next word with the transformer model\n", " predictions, attention_weights = model(\n", " encoder_input, output, False, enc_padding_mask, look_ahead_mask, dec_padding_mask\n", " )\n", " ### END CODE HERE ###\n", "\n", " predictions = predictions[: ,-1:, :]\n", " predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n", " \n", " return predicted_id" ] }, { "cell_type": "markdown", "id": "29af50d0", "metadata": {}, "source": [ "Check if your function works." ] }, { "cell_type": "code", "execution_count": 30, "id": "3e97ba77", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Predicted token: [[14859]]\n", "Predicted word: masses\n" ] } ], "source": [ "# Take a random sentence as an input\n", "input_document = tokenizer.texts_to_sequences([\"a random sentence\"])\n", "input_document = tf.keras.preprocessing.sequence.pad_sequences(input_document, maxlen=encoder_maxlen, padding='post', truncating='post')\n", "encoder_input = tf.expand_dims(input_document[0], 0)\n", "\n", "# Take the start of sentence token as the only token in the output to predict the next word\n", "output = tf.expand_dims([tokenizer.word_index[\"[SOS]\"]], 0)\n", "\n", "# predict the next word with your function\n", "predicted_token = next_word(transformer, encoder_input, output)\n", "print(f\"Predicted token: {predicted_token}\")\n", "\n", "predicted_word = tokenizer.sequences_to_texts(predicted_token.numpy())[0]\n", "print(f\"Predicted word: {predicted_word}\")" ] }, { "cell_type": "markdown", "id": "7157031c", "metadata": {}, "source": [ "##### __Expected Output__\n", "\n", "```\n", "Predicted token: [[14859]]\n", "Predicted word: masses\n", "```" ] }, { "cell_type": "code", "execution_count": 31, "id": "6bd98959", "metadata": { "deletable": false, "editable": false, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[92m All tests passed!\n" ] } ], "source": [ "# UNIT TEST\n", "w2_unittest.test_next_word(next_word, transformer, encoder_input, output)" ] }, { "cell_type": "code", "execution_count": 32, "id": "6177dc6a", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [], "source": [ "def summarize(model, input_document):\n", " \"\"\"\n", " A function for summarization using the transformer model\n", " Arguments:\n", " input_document (tf.Tensor): Input data to summarize\n", " Returns:\n", " _ (str): The summary of the input_document\n", " \"\"\" \n", " input_document = tokenizer.texts_to_sequences([input_document])\n", " input_document = tf.keras.preprocessing.sequence.pad_sequences(input_document, maxlen=encoder_maxlen, padding='post', truncating='post')\n", " encoder_input = tf.expand_dims(input_document[0], 0)\n", " \n", " output = tf.expand_dims([tokenizer.word_index[\"[SOS]\"]], 0)\n", " \n", " for i in range(decoder_maxlen):\n", " predicted_id = next_word(model, encoder_input, output)\n", " output = tf.concat([output, predicted_id], axis=-1)\n", " \n", " if predicted_id == tokenizer.word_index[\"[EOS]\"]:\n", " break\n", "\n", " return tokenizer.sequences_to_texts(output.numpy())[0] # since there is just one translated document" ] }, { "cell_type": "markdown", "id": "d3b15117", "metadata": {}, "source": [ "Now you can already summarize a sentence! But beware, since the model was not yet trained at all, it will just produce nonsense." ] }, { "cell_type": "code", "execution_count": 33, "id": "bae4d5f1", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training set example:\n", "[SOS] amanda: i baked cookies. do you want some? jerry: sure! amanda: i'll bring you tomorrow :-) [EOS]\n", "\n", "Human written summary:\n", "[SOS] amanda baked cookies and will bring jerry some tomorrow. [EOS]\n", "\n", "Model written summary:\n" ] }, { "data": { "text/plain": [ "\"[SOS] masses kindergarten concept kindergarten concept bloomer wilingness sux sam kindergarten lisabeth kindergarten sawyer's sawyer's masses concept bloomer lisabeth bloomer wilingness 80000 bt hotsummer hoax hoax kieslowski wilingness 80000 dont't elis' 🐶❤️👍 cots saaaad evelynn inexperienced suji zubac forthcoming callum farmers extraordinary callum kindergarten worthy extraordinary readable 🐶❤️👍 thinkgn 🐶❤️👍 cots\"" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_set_example = 0\n", "\n", "# Check a summary of a document from the training set\n", "print('Training set example:')\n", "print(document[training_set_example])\n", "print('\\nHuman written summary:')\n", "print(summary[training_set_example])\n", "print('\\nModel written summary:')\n", "summarize(transformer, document[training_set_example])" ] }, { "cell_type": "markdown", "id": "90d6f836", "metadata": {}, "source": [ "\n", "# 12 - Train the model\n", "\n", "Now you can finally train the model. Below is a loop that will train your model for 20 epochs. note that it should take about 30 seconds per epoch (with the exception of the first few epochs which can take a few minutes each).\n", "\n", "Note that after each epoch you perform the summarization on one of the sentences in the test set and print it out, so you can see how your model is improving." ] }, { "cell_type": "code", "execution_count": 34, "id": "ebe2bf5f", "metadata": { "deletable": false, "editable": false, "scrolled": true, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1, Batch 1/231\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-06-12 13:11:54.587111: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7bcf29ee4ae0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", "2025-06-12 13:11:54.587172: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA A10G, Compute Capability 8.6\n", "2025-06-12 13:11:54.636169: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:255] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n", "2025-06-12 13:11:54.723376: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8600\n", "2025-06-12 13:11:55.021293: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1, Loss 7.886631\n", "Time taken for one epoch: 65.2928831577301 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] [EOS]\n", "\n", "Epoch 2, Loss 6.600031\n", "Time taken for one epoch: 24.169685125350952 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] is going to the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the\n", "\n", "Epoch 3, Loss 6.029531\n", "Time taken for one epoch: 16.651750326156616 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] tom is going to the new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new new\n", "\n", "Epoch 4, Loss 5.683831\n", "Time taken for one epoch: 12.880155563354492 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] tom is going to the new new new new new new new new new new new new new new new job [EOS]\n", "\n", "Epoch 5, Loss 5.475431\n", "Time taken for one epoch: 13.034060716629028 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] the new new new new new new new job and she will be at the weekend [EOS]\n", "\n", "Epoch 6, Loss 5.322731\n", "Time taken for one epoch: 10.890370845794678 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] tom is going to the new job [EOS]\n", "\n", "Epoch 7, Loss 5.195131\n", "Time taken for one epoch: 10.396798133850098 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] tom is going to the party with her [EOS]\n", "\n", "Epoch 8, Loss 5.083531\n", "Time taken for one epoch: 11.252682209014893 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] the new year's eve is going to the party [EOS]\n", "\n", "Epoch 9, Loss 4.977131\n", "Time taken for one epoch: 9.915740489959717 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] the new year's eve is going to the party [EOS]\n", "\n", "Epoch 10, Loss 4.876331\n", "Time taken for one epoch: 9.256117343902588 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] the car is going to the party with her [EOS]\n", "\n", "Epoch 11, Loss 4.776131\n", "Time taken for one epoch: 9.405097723007202 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] mark has just arrived to the office today [EOS]\n", "\n", "Epoch 12, Loss 4.674431\n", "Time taken for one epoch: 9.421130657196045 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] the car is going to the cinema with her [EOS]\n", "\n", "Epoch 13, Loss 4.571131\n", "Time taken for one epoch: 9.543477296829224 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] ben will buy the red dress for the movie [EOS]\n", "\n", "Epoch 14, Loss 4.470531\n", "Time taken for one epoch: 9.424731492996216 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] alice has just arrived at the cinema with her [EOS]\n", "\n", "Epoch 15, Loss 4.368131\n", "Time taken for one epoch: 9.076642751693726 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] alice has just arrived to the cinema with her [EOS]\n", "\n", "Epoch 16, Loss 4.267731\n", "Time taken for one epoch: 9.078428030014038 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] hannah has just arrived to the cinema with her [EOS]\n", "\n", "Epoch 17, Loss 4.164831\n", "Time taken for one epoch: 10.068711042404175 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] alice has just arrived to the cinema with her [EOS]\n", "\n", "Epoch 18, Loss 4.069731\n", "Time taken for one epoch: 8.947221755981445 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] hannah has just arrived to the cinema with amanda and sara will go to the cinema [EOS]\n", "\n", "Epoch 19, Loss 3.967131\n", "Time taken for one epoch: 9.104878425598145 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] alice has just finished the book and he will be at the cinema with amanda [EOS]\n", "\n", "Epoch 20, Loss 3.876331\n", "Time taken for one epoch: 8.590877771377563 sec\n", "Example summarization on the test set:\n", " True summarization:\n", " [SOS] hannah needs betty's number but amanda doesn't have it. she needs to contact larry. [EOS]\n", " Predicted summarization:\n", " [SOS] alice and hannah are going to the cinema with amanda and sara will see him [EOS]\n", "\n" ] } ], "source": [ "# Take an example from the test set, to monitor it during training\n", "test_example = 0\n", "true_summary = summary_test[test_example]\n", "true_document = document_test[test_example]\n", "\n", "# Define the number of epochs\n", "epochs = 20\n", "\n", "# Training loop\n", "for epoch in range(epochs):\n", " \n", " start = time.time()\n", " train_loss.reset_states()\n", " number_of_batches=len(list(enumerate(dataset)))\n", "\n", " for (batch, (inp, tar)) in enumerate(dataset):\n", " print(f'Epoch {epoch+1}, Batch {batch+1}/{number_of_batches}', end='\\r')\n", " train_step(transformer, inp, tar)\n", " \n", " print (f'Epoch {epoch+1}, Loss {train_loss.result():.4f}')\n", " losses.append(train_loss.result())\n", " \n", " print (f'Time taken for one epoch: {time.time() - start} sec')\n", " print('Example summarization on the test set:')\n", " print(' True summarization:')\n", " print(f' {true_summary}')\n", " print(' Predicted summarization:')\n", " print(f' {summarize(transformer, true_document)}\\n')" ] }, { "cell_type": "markdown", "id": "35687ddc", "metadata": {}, "source": [ "Plot the loss funtion." ] }, { "cell_type": "code", "execution_count": 35, "id": "eb3d5335", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "data": { "text/plain": [ "Text(0.5, 0, 'Epoch')" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABLsElEQVR4nO3deVxU9f4/8NcZlmGRGZRtGBgRV5AdVzSvlRaapmiL+rOrVrZdW6zb95a3rMwKu9263Vs3TW9qZdq1BSwtFSzNFHMB3DcE2RdFYVhkgJnz+4MY5QojIMyZ5fV8PObxcM58zuF9Oo28/JzP+XwEURRFEBEREdkImdQFEBEREXUlhhsiIiKyKQw3REREZFMYboiIiMimMNwQERGRTWG4ISIiIpvCcENEREQ2xVHqAszNYDCgqKgIHh4eEARB6nKIiIioHURRRFVVFdRqNWQy030zdhduioqKoNFopC6DiIiIOiE/Px+BgYEm29hduPHw8ADQ9B9HoVBIXA0RERG1h1arhUajMf4eN8Xuwk3zrSiFQsFwQ0REZGXaM6SEA4qJiIjIpkgabvR6PRYvXozg4GC4urqiX79+WLp0KW60lufOnTsRGxsLuVyO/v37Y+3ateYpmIiIiCyepLel3n77bSxfvhyffvopwsLCcPDgQTz44INQKpV4+umnW90nJycHkyZNwuOPP44vvvgCO3bswPz58+Hv74/4+HgznwERERFZGkG8UTdJN5o8eTL8/PzwySefGLfdc889cHV1xbp161rd54UXXsCWLVtw7Ngx47aZM2eioqICW7duva69TqeDTqczvm8ekFRZWckxN0RERFZCq9VCqVS26/e3pLelRo0ahR07duDMmTMAgMOHD+PXX3/FxIkT29wnLS0N48ePb7EtPj4eaWlprbZPTEyEUqk0vvgYOBERkW2T9LbUiy++CK1Wi5CQEDg4OECv1+PNN9/E7Nmz29ynpKQEfn5+Lbb5+flBq9XiypUrcHV1bfHZokWL8NxzzxnfN/fcEBERkW2SNNxs3LgRX3zxBdavX4+wsDBkZmZi4cKFUKvVmDt3bpf8DLlcDrlc3iXHIiIiIssnabj5v//7P7z44ouYOXMmACAiIgK5ublITExsM9yoVCqUlpa22FZaWgqFQnFdrw0RERHZH0nH3NTW1l63PoSDgwMMBkOb+8TFxWHHjh0ttqWkpCAuLq5baiQiIiLrImm4ufvuu/Hmm29iy5YtOH/+PJKSkvDee+9h2rRpxjaLFi3CnDlzjO8ff/xxZGdn4y9/+QtOnTqFjz76CBs3bsSzzz4rxSkQERGRhZH0ttQHH3yAxYsX409/+hPKysqgVqvx2GOP4ZVXXjG2KS4uRl5envF9cHAwtmzZgmeffRb//Oc/ERgYiP/85z+c44aIiIgASDzPjRQ68pw8ERERWQarmefG1pRX63CmtErqMoiIiOwaw00XST1RiiFvpOLPGw9LXQoREZFdY7jpIqHqpi6yk8VaXKnXS1wNERGR/WK46SJqpQt8PeRoNIg4WlgpdTlERER2i+GmiwiCgJjengCAjLzL0hZDRERkxxhuulBs754AgIy8CmkLISIismMMN10o5vdwk553GXb2hD0REZHFYLjpQhEBSjjKBJRV6VBUWSd1OURERHaJ4aYLuTo7INS/6akpjrshIiKSBsNNF2seVJyeWyFpHURERPaK4aaLGQcV57PnhoiISAoMN12suefmeKEWukZO5kdERGRuDDddrHcvN/Ryd0a93oDjRVqpyyEiIrI7DDddTBAExBon86uQtBYiIiJ7xHDTDa6d74aIiIjMi+GmG8RoPAEAmey5ISIiMjuGm24QqfGETAAKK66gVMvJ/IiIiMyJ4aYb9JA7YqCfBwBO5kdERGRuDDfdJIaLaBIREUmC4aab8IkpIiIiaTDcdJPmnpsjhRVo0BskroaIiMh+MNx0k77e7lC4OKKuwYBTxVVSl0NERGQ3GG66iUwmcL4bIiIiCTDcdKMY47gbhhsiIiJzYbjpRldXCK+QthAiIiI7wnDTjaJ+n6k4t7wWF6t10hZDRERkJxhuupHS1Qn9fXsA4FIMRERE5sJw082M893kc9wNERGROTDcdDPjE1O5FdIWQkREZCcYbrpZ8xNThwsqoDeI0hZDRERkBxhuutkAXw/0kDuitl6PM6WczI+IiKi7SRpu+vTpA0EQrnstWLCg1fZr1669rq2Li4uZq+4YB5mAKI0SACfzIyIiMgdJw82BAwdQXFxsfKWkpAAA7rvvvjb3USgULfbJzc01V7mdFqPhCuFERETm4ijlD/fx8WnxftmyZejXrx/Gjh3b5j6CIEClUrX7Z+h0Ouh0V+eY0Wq1HS/0JsUGeQJgzw0REZE5WMyYm/r6eqxbtw4PPfQQBEFos111dTWCgoKg0WgwdepUHD9+3ORxExMToVQqjS+NRtPVpd9Q9O89N9kXalBRW2/2n09ERGRPLCbcJCcno6KiAvPmzWuzzaBBg7B69Wps2rQJ69atg8FgwKhRo1BQUNDmPosWLUJlZaXxlZ+f3w3Vm9bL3RnB3u4AgEwuxUBERNStJL0tda1PPvkEEydOhFqtbrNNXFwc4uLijO9HjRqF0NBQfPzxx1i6dGmr+8jlcsjl8i6vt6NiNJ7IuViD9LwK3DrIV+pyiIiIbJZF9Nzk5uYiNTUV8+fP79B+Tk5OiImJQVZWVjdV1nW4QjgREZF5WES4WbNmDXx9fTFp0qQO7afX63H06FH4+/t3U2Vdp3mm4sz8Chg4mR8REVG3kTzcGAwGrFmzBnPnzoWjY8u7ZHPmzMGiRYuM719//XVs374d2dnZSE9PxwMPPIDc3NwO9/hIIUTlARcnGarqGnHuQrXU5RAREdksycfcpKamIi8vDw899NB1n+Xl5UEmu5q/Ll++jEceeQQlJSXo2bMnhgwZgr1792Lw4MHmLLlTHB1kiAz0xP6cS8jIq8AAPw+pSyIiIrJJgiiKdnWPRKvVQqlUorKyEgqFwqw/e9mPp7Bi1znMGq5B4vRIs/5sIiIia9aR39+S35ayJ82DirlCOBERUfdhuDGj5nBzpqwKVXUN0hZDRERkoxhuzMjXwwWBPV0hisDh/EqpyyEiIrJJDDdm1vxIOOe7ISIi6h4MN2YW2zyZH5dhICIi6hYMN2Z2bc+NnT2oRkREZBYMN2Y22F8BZ0cZLtc24Hx5rdTlEBER2RyGGzNzdpQhIkAJgONuiIiIugPDjQRiNJ4AgHSGGyIioi7HcCOBq+NuKqQthIiIyAYx3EggNsgTAHCqpAq19Y3SFkNERGRjGG4k4K90hUrhAr1BxJECTuZHRETUlRhuJNK8FANvTREREXUthhuJxP4+7oaDiomIiLoWw41Eru254WR+REREXYfhRiLhAUo4OQi4WK1DweUrUpdDRERkMxhuJOLi5IDB/goAvDVFRETUlRhuJMT5boiIiLoew42EYrhCOBERUZdjuJFQ8xNTJ4oqUdegl7gaIiIi28BwI6HAnq7w7uGMBr2I40WczI+IiKgrMNxISBAEjrshIiLqYgw3Emsed8MnpoiIiLoGw43EYjTsuSEiIupKDDcSi9IoIROA4so6FFdyMj8iIqKbxXAjMTdnR4SomibzY+8NERHRzWO4sQCxQZ4AgAyOuyEiIrppDDcWoHncTTp7boiIiG4aw40FaH5i6mhhJeobDdIWQ0REZOUYbixAsLc7PN2cUN9owMlirdTlEBERWTWGGwsgCAJiNJ4AON8NERHRzZI03PTp0weCIFz3WrBgQZv7fPXVVwgJCYGLiwsiIiLwww8/mLHi7sOZiomIiLqGpOHmwIEDKC4uNr5SUlIAAPfdd1+r7ffu3YtZs2bh4YcfRkZGBhISEpCQkIBjx46Zs+xu0byIZkY+e26IiIhuhiCKoih1Ec0WLlyIzZs34+zZsxAE4brPZ8yYgZqaGmzevNm4beTIkYiOjsaKFStaPaZOp4NOpzO+12q10Gg0qKyshEKh6PqT6CRtXQOilmyHKAIHXhoPHw+51CURERFZDK1WC6VS2a7f3xYz5qa+vh7r1q3DQw891GqwAYC0tDSMHz++xbb4+HikpaW1edzExEQolUrjS6PRdGndXUXh4oQBvj0AcL4bIiKim2Ex4SY5ORkVFRWYN29em21KSkrg5+fXYpufnx9KSkra3GfRokWorKw0vvLz87uq5C7XfGuK890QERF1nsWEm08++QQTJ06EWq3u0uPK5XIoFIoWL0vVPN8Ne26IiIg6z1HqAgAgNzcXqamp+Pbbb022U6lUKC0tbbGttLQUKpWqO8szm+aemyMFlWjUG+DoYDHZk4iIyGpYxG/PNWvWwNfXF5MmTTLZLi4uDjt27GixLSUlBXFxcd1Zntn08+kBD7kjrjTocaqkSupyiIiIrJLk4cZgMGDNmjWYO3cuHB1bdiTNmTMHixYtMr5/5plnsHXrVrz77rs4deoUXnvtNRw8eBBPPvmkucvuFjKZgOjmW1P5FZLWQkREZK0kDzepqanIy8vDQw89dN1neXl5KC4uNr4fNWoU1q9fj5UrVyIqKgpff/01kpOTER4ebs6Su9XVyfw47oaIiKgzLGqeG3PoyHPyUvj5dBkeXHMAwd7u+Pn5W6Uuh4iIyCJY5Tw31KR5jamcizW4XFMvbTFERERWiOHGwni6OaOvjzsAIJPjboiIiDqM4cYCxWiaJ/PjuBsiIqKOYrixQFcn86uQtA4iIiJrxHBjgZon88vMr4DeYFfjvYmIiG4aw40FGujXA27ODqjWNSKrrFrqcoiIiKwKw40FcnSQISrQEwDnuyEiIuoohhsL1TzuhoOKiYiIOobhxkJdnam4QtpCiIiIrAzDjYVq7rk5W1aNyisN0hZDRERkRRhuLJR3Dzl693IDABzmZH5ERETtxnBjwTjfDRERUccx3Fiw5vluMvI5qJiIiKi9GG4s2LU9NwZO5kdERNQuDDcWLESlgNxRhsorDcgpr5G6HCIiIqvAcGPBnB1liAxUAgDSc3lrioiIqD0Ybiyccb4bPjFFRETULgw3Fi6WT0wRERF1CMONhWvuuTldokW1rlHiaoiIiCwfw42F81O4QK10gUEEjhRUSF0OERGRxWO4sQIxQVxnioiIqL0YbqxAjMYTAJDBFcKJiIhuiOHGCly7QrgocjI/IiIiUxhurEB4gALODjKU19Qj/9IVqcshIiKyaAw3VkDu6IDBagUAIJ23poiIiExiuLESzetM/XLmgrSFEBERWTiGGytxd5QaAPDd4SLkX6qVuBoiIiLLxXBjJWJ798SYAd5oNIj4989ZUpdDRERksRhurMgz4wYAAL4+VMDeGyIiojYw3FiRoX16YXR/LzQaRHy085zU5RAREVkkycNNYWEhHnjgAXh5ecHV1RURERE4ePBgm+137twJQRCue5WUlJixauk8M24gAODrQ/korOBj4URERP9L0nBz+fJljB49Gk5OTvjxxx9x4sQJvPvuu+jZs+cN9z19+jSKi4uNL19fXzNULL3hwb0Q19cLDXoRH3HsDRER0XUcpfzhb7/9NjQaDdasWWPcFhwc3K59fX194enp2U2VWbZnxg9A2spybDyYjwW39Yfa01XqkoiIiCyGpD033333HYYOHYr77rsPvr6+iImJwapVq9q1b3R0NPz9/XHHHXdgz549bbbT6XTQarUtXtZuZF8vjAjuhQa9iOUce0NERNSCpOEmOzsby5cvx4ABA7Bt2zY88cQTePrpp/Hpp5+2uY+/vz9WrFiBb775Bt988w00Gg1uvfVWpKent9o+MTERSqXS+NJoNN11Oma1cHzT2Jv/HshHcSXH3hARETUTRAlXYnR2dsbQoUOxd+9e47ann34aBw4cQFpaWruPM3bsWPTu3Ruff/75dZ/pdDrodDrje61WC41Gg8rKSigUips7AYnd/3Ea9udcwpy4ILw+NVzqcoiIiLqNVquFUqls1+9vSXtu/P39MXjw4BbbQkNDkZeX16HjDB8+HFlZrQ+ulcvlUCgULV62YuHv8958uT8fJZV1EldDRERkGSQNN6NHj8bp06dbbDtz5gyCgoI6dJzMzEz4+/t3ZWlWIa6fF4b16Yl6vQErdnHsDRERESBxuHn22Wexb98+vPXWW8jKysL69euxcuVKLFiwwNhm0aJFmDNnjvH9+++/j02bNiErKwvHjh3DwoUL8dNPP7XYx14IgmCc92b9/jyUatl7Q0REJGm4GTZsGJKSkrBhwwaEh4dj6dKleP/99zF79mxjm+Li4ha3qerr6/HnP/8ZERERGDt2LA4fPozU1FSMGzdOilOQ3Oj+XhgS1BP1jey9ISIiAiQeUCyFjgxIsha/nLmAOav3Q+4ow+6/3AZfhYvUJREREXUpqxlQTF1jzABvxPT2hK7RgI9/yZa6HCIiIkkx3NiAprE3TU9OffFbLi5U6W6wBxERke1iuLERYwf6IErjiboGA1b+wrE3RERkvxhubIQgCFg4vqn35vN9ubhYzd4bIiKyTww3NuTWgT6IClSirsGAVRx7Q0REdorhxoYIgoBnfu+9+SwtF+XsvSEiIjvEcGNjbhvki8hAJa406LFqd47U5RAREZkdw42NEQQBT9/e3HtzHpdq6iWuiIiIyLwYbmzQuFBfhAcoUFuvx6rdHHtDRET2heHGBrXovdl7HpfZe0NERHaE4cZG3THYD4P9Faip1+M/v7L3hoiI7AfDjY0SBAFP/z5r8ad7c1FRy94bIiKyDww3NuzOwX4IUXmgWteIT37lk1NERGQfGG5smEx2ddbitXvOo7K2QeKKiIiIuh/DjY27c7AKISoPVOka8cke9t4QEZHtY7ixcTLZ1bE3a/bkoPIKe2+IiMi2MdzYgQlhKgzy80BVXSPWsPeGiIhsHMONHZDJBDw1rj8AYPWvOdDWsfeGiIhsF8ONnbgr3B8DfHtAW9eItXvOS10OERFRt2G4sRNNvTdNY28+Ye8NERHZMIYbOzIpwh/9fNxReaUBn7L3hoiIbBTDjR1xuObJqf/8moMq9t4QEZENYrixM5Mj1ej7e+/NZ2m5UpdDRETU5Rhu7IyD7OqK4at2Z6Na1yhxRURERF2L4cYO3R2lRl9vd1TUNuCztPNSl0NERNSlGG7skINMwJO3N817s+qXbNSw94aIiGwIw42dmhKlRh8vN1yubcDn+zj2hoiIbAfDjZ1ydJDhyeaxN79ko7aevTdERGQbGG7sWEK0GkFebiivqcc69t4QEZGNYLixY44OMiy4rWnszcpfsnGlXi9xRURERDeP4cbOTYsJgKaXKy5W1+OL39h7Q0RE1k/ycFNYWIgHHngAXl5ecHV1RUREBA4ePGhyn507dyI2NhZyuRz9+/fH2rVrzVOsDXJykOHJ33tvVuxi7w0REVk/ScPN5cuXMXr0aDg5OeHHH3/EiRMn8O6776Jnz55t7pOTk4NJkybhtttuQ2ZmJhYuXIj58+dj27ZtZqzctkyPDURgT1dcrNbhlU3HIIqi1CURERF1miBK+JvsxRdfxJ49e7B79+527/PCCy9gy5YtOHbsmHHbzJkzUVFRga1bt95wf61WC6VSicrKSigUik7VbYt2n72Auav3wyACr0wejIduCZa6JCIiIqOO/P7uVM9Nfn4+CgoKjO/379+PhQsXYuXKlR06znfffYehQ4fivvvug6+vL2JiYrBq1SqT+6SlpWH8+PEttsXHxyMtLa3V9jqdDlqttsWLrjdmgA/+elcoAOCNLSfwy5kLEldERETUOZ0KN//v//0//PzzzwCAkpIS3HHHHdi/fz9eeuklvP766+0+TnZ2NpYvX44BAwZg27ZteOKJJ/D000/j008/bXOfkpIS+Pn5tdjm5+cHrVaLK1euXNc+MTERSqXS+NJoNO2uz948fEsw7h0SCIMIPLk+HTkXa6QuiYiIqMM6FW6OHTuG4cOHAwA2btyI8PBw7N27F1988UWHBvcaDAbExsbirbfeQkxMDB599FE88sgjWLFiRWfKatWiRYtQWVlpfOXn53fZsW2NIAh4c1o4Ynt7QlvXiEc+OwhtXYPUZREREXVIp8JNQ0MD5HI5ACA1NRVTpkwBAISEhKC4uLjdx/H398fgwYNbbAsNDUVeXl6b+6hUKpSWlrbYVlpaCoVCAVdX1+vay+VyKBSKFi9qm9zRASv+OAT+ShdklVVj4ZeZ0Bs4wJiIiKxHp8JNWFgYVqxYgd27dyMlJQUTJkwAABQVFcHLy6vdxxk9ejROnz7dYtuZM2cQFBTU5j5xcXHYsWNHi20pKSmIi4vrwBmQKb4eLlj5x6GQO8rw06kyvLPt9I13IiIishCdCjdvv/02Pv74Y9x6662YNWsWoqKiADQNEG6+XdUezz77LPbt24e33noLWVlZWL9+PVauXIkFCxYY2yxatAhz5swxvn/88ceRnZ2Nv/zlLzh16hQ++ugjbNy4Ec8++2xnToXaEBGoxN/ujQQArNh1DskZhRJXRERE1D6dfhRcr9dDq9W2mJPm/PnzcHNzg6+vb7uPs3nzZixatAhnz55FcHAwnnvuOTzyyCPGz+fNm4fz589j586dxm07d+7Es88+ixMnTiAwMBCLFy/GvHnz2vXz+Ch4x/xt6yl8tPMc5I4ybHwsDlEaT6lLIiIiO9SR39+dCjdXrlyBKIpwc3MDAOTm5iIpKQmhoaGIj4/vXNVmwnDTMQaDiEc/P4jUk2XwU8jx/ZO3wFfhInVZRERkZ7p9npupU6fis88+AwBUVFRgxIgRePfdd5GQkIDly5d35pBkoWQyAf+YEY0Bvj1QqtXh0c8Poa6BSzQQEZHl6lS4SU9Px5gxYwAAX3/9Nfz8/JCbm4vPPvsM//rXv7q0QJKeh4sTVs0ZCqWrEzLzK/BSEpdoICIiy9WpcFNbWwsPDw8AwPbt2zF9+nTIZDKMHDkSublcWdoW9fF2x0ezY+EgE/BNegE++TVH6pKIiIha1alw079/fyQnJyM/Px/btm3DnXfeCQAoKyvjOBYbNrq/N16e1LREw1s/nMTO02USV0RERHS9ToWbV155Bc8//zz69OmD4cOHG+eY2b59O2JiYrq0QLIs80b1wYyhGhhE4KkNGTh3oVrqkoiIiFro9KPgJSUlKC4uRlRUFGSypoy0f/9+KBQKhISEdGmRXYlPS908XaMes1f9hoO5l9HXxx1JfxoNpauT1GUREZEN6/ZHwa/VvDp4YGDgzRzGbBhuusaFKh2mfvgriirrMHagD1bPGwYHmSB1WUREZKO6/VFwg8GA119/HUqlEkFBQQgKCoKnpyeWLl0Kg8HQqaLJuvh4yLFyzlC4OMmw68wFvL31lNQlERERAehkuHnppZfw4YcfYtmyZcjIyEBGRgbeeustfPDBB1i8eHFX10gWKjxAib/f17T0xspfsvHNoQKJKyIiIurkbSm1Wo0VK1YYVwNvtmnTJvzpT39CYaHlrkPE21Jd793tp/HBT1lwdpThv4+OREzvnjfeiYiIqAO6/bbUpUuXWh00HBISgkuXLnXmkGTFnh0/EHcM9kN9owGPfX4IJZV1UpdERER2rFPhJioqCh9++OF12z/88ENERkbedFFkXZqXaBjo1wNlVTo89vlBLtFARESS6dRtqV27dmHSpEno3bu3cY6btLQ05Ofn44cffjAuzWCJeFuq++SV12LKv39FRW0DEqLV+MeMaAgCn6AiIqKb1+23pcaOHYszZ85g2rRpqKioQEVFBaZPn47jx4/j888/71TRZP16e7kZl2hIzizCyl+ypS6JiIjs0E3Pc3Otw4cPIzY2Fnq95d6SYM9N9/s87TwWbzoOQQBWzx2G20J8pS6JiIisXLf33BCZ8sDIIMwa3huiCDy9IQNZZVVSl0RERHaE4Ya6nCAIWDIlDMP79EKVrhHzPz2IytoGqcsiIiI7wXBD3cLZUYblD8QiwNMV58tr8eSGdDTqOXs1ERF1P8eONJ4+fbrJzysqKm6mFrIxXj3kWDVnKO5Zvhe7z17E0s0n8OrdYZBxDSoiIupGHQo3SqXyhp/PmTPnpgoi2zJYrcB790fhiS/S8WlaLs6X1+Ld+6Pg3UMudWlERGSjuvRpKWvAp6WksfFgPhYnH4Ou0QBfDznenxGNUf29pS6LiIisBJ+WIotz/1ANvnvyFgzwbZrFePYnv+Hd7ac5DoeIiLocww2ZzSCVB7578hbMHKaBKAIf/JSFWav2oajiitSlERGRDWG4IbNydXbAsnsi8a9ZMeghd8SB85cx8Z+7sf14idSlERGRjWC4IUlMiVJjy9O3IDJQicorDXj080N47bvj0DVa7uzWRERkHRhuSDJBXu74+vFReGRMMABg7d7zmP7RXmRfqJa4MiIismYMNyQpZ0cZXpo0GGvmDUMvd2ccL9Ji8ge/4tv0AqlLIyIiK8VwQxbhthBf/PD0GIzs2wu19Xo8t/Ew/rzxMGp0jVKXRkREVobhhiyGSumCL+aPxHN3DIRMAL5JL8DdH/yK40WVUpdGRERWhOGGLIqDTMDT4wZgwyMjoVK4IPtiDaZ9tBefpZ2Hnc03SUREncRwQxZpRF8v/PjMGIwP9UV9owGvbDqOxz4/hIraeqlLIyIiCydpuHnttdcgCEKLV0hISJvt165de117FxcXM1ZM5tTT3Rmr5gzFK5MHw8lBwPYTpZj0r19x8PwlqUsjIiIL1qGFM7tDWFgYUlNTje8dHU2XpFAocPr0aeN7QeAK07ZMEAQ8dEswhvXphac2pON8eS1mrNyH5+4YiMfH9oMDVxgnIqL/IXm4cXR0hEqland7QRA61J5sQ0SgEpufHoOXk44iObMI72w7jb3nLuIfM6Lh68HeOyIiukryMTdnz56FWq1G3759MXv2bOTl5ZlsX11djaCgIGg0GkydOhXHjx832V6n00Gr1bZ4kXXqIXfEP2ZE42/3RsLVyQF7sspx1z93Y9eZC1KXRkREFkTScDNixAisXbsWW7duxfLly5GTk4MxY8agqqqq1faDBg3C6tWrsWnTJqxbtw4GgwGjRo1CQUHbE74lJiZCqVQaXxqNprtOh8xAEATcP1SD758ajRCVBy5W12Pu6v1I/PEkGrjCOBERARBEC3q+tqKiAkFBQXjvvffw8MMP37B9Q0MDQkNDMWvWLCxdurTVNjqdDjqdzvheq9VCo9GgsrISCoWiy2on86tr0OONLSewbl9Tb99Avx54YUIIbg/x5VgsIiIbo9VqoVQq2/X7W/LbUtfy9PTEwIEDkZWV1a72Tk5OiImJMdleLpdDoVC0eJFtcHFywBsJEVg+OxZKVyecKa3Gw58exIyP9+FQ7mWpyyMiIolYVLiprq7GuXPn4O/v3672er0eR48ebXd7sk0TI/zxy//dhsfH9oPcUYb95y/hnuV78djnB5FVxkU4iYjsjaTh5vnnn8euXbtw/vx57N27F9OmTYODgwNmzZoFAJgzZw4WLVpkbP/6669j+/btyM7ORnp6Oh544AHk5uZi/vz5Up0CWQilmxNenBiCnf93K2YM1UAmANuOlyL+/V+w6NsjKNXWSV0iERGZiaSPghcUFGDWrFkoLy+Hj48PbrnlFuzbtw8+Pj4AgLy8PMhkV/PX5cuX8cgjj6CkpAQ9e/bEkCFDsHfvXgwePFiqUyAL4690xdv3RmL+mGD8bdtppJwoxYb9+UjKKMRDo4Px2Nh+ULo6SV0mERF1I4saUGwOHRmQRNbv4PlLSPzxlHEMjqebE568rT8eGBkEFycHiasjIqL26sjvb4YbsnmiKCL1ZBne3nrKOAYnwNMVz90xEAkxAZzlmIjICjDcmMBwY78a9QZ8m16I91LOoOT3MTghKg+8MCEEtw7y4ePjREQWjOHGBIYbqmvQY+3e8/j3z1moqmsEAIwI7oUXJ4YgpndPiasjIqLWMNyYwHBDzSpq6/HRznNYu/c86hubZjeeGK7C8/GD0M+nh8TVERHRtRhuTGC4of9VWHEF/0g5g2/SCyCKgINMwIxhGiwcNwC+Ci7KSURkCRhuTGC4obacLqnCO9tOIfVkGQDA1ckB88cE49E/9IWHCx8fJyKSEsONCQw3dCP7cy5h2Y8nkZ5XAQDo6eaEJ28fgNkjevPxcSIiiTDcmMBwQ+0hiiK2nyjF37aewrkLNQAAL3dn/DEuCH8cGQSvHnKJKyQisi8MNyYw3FBHNOoN+PpQAT74KQuFFVcAAHJHGabHBmL+mGAOPCYiMhOGGxMYbqgzGvUG/HisBKt2Z+NIQaVx+/hQX8wf0xcjgntxnhwiom7EcGMCww3dDFEUsT/nElbtzkHqyVLj9ogAJR75Q1/cFa6Co4Ok69ESEdkkhhsTGG6oq5y7UI1Pfs3BN4cKoPt9npwAT1c8OLoPZgzT8AkrIqIuxHBjAsMNdbXyah3W7cvDZ2nnUV5TDwDwkDti1ojemDeqD9SerhJXSERk/RhuTGC4oe5S16BHUkYhVu3ORvbvT1g5ygRMjvTH/DF9ER6glLhCIiLrxXBjAsMNdTeDQcTPp8uwanc29mVfMm6P6+uFR//QF2MH+kDGlciJiDqE4cYEhhsyp6MFlVi1OxtbjhZDb2j6qvX37YH5twQjISaAkwISEbUTw40JDDckhcKKK1i7Jwcb9uejWte0Erl3D2fMieuDB0YGoZe7s8QVEhFZNoYbExhuSEraugb8d38+Vu/JQXFlHQDAxalpUsDZI3ojTM1xOURErWG4MYHhhixBg96AH44WY9XubBwr1Bq3RwYqMXNYb0yJVqOH3FHCComILAvDjQkMN2RJRFHEvuxLWLcvF9tPlKBB3/R1dHN2wN2RaswYrkGMxpOzHxOR3WO4MYHhhixVebUO36YXYsOBPOOj5AAwyM8DM4drMC0mAJ5uHJtDRPaJ4cYEhhuydKIo4sD5y/hyfx62HC02zn7s7CjDXeEqzBzem2tZEZHdYbgxgeGGrEllbQM2HS7Ehv35OFl8dWxOX293zBimwT1DAuHdQy5hhURE5sFwYwLDDVkjURRxpKASXx7Iw3eZRaip1wNomgH5jsF+mDm8N8b09+bkgERksxhuTGC4IWtXo2vE94eL8OWBfGTmVxi3B3i6YsYwDe4bGgh/JdezIiLbwnBjAsMN2ZKTxVr890A+vk0vgLauaXJAmQDcNsgXM4f3xm2DfODoIJO4SiKim8dwYwLDDdmiugY9fjxWjA3787E/5+p6Vr4ectw3NBAzhvZGby83CSskIro5DDcmMNyQrcu+UI3/HsjH14cKUF5Tb9w+IrgX7h0SiLsi/OHOCQKJyMow3JjAcEP2or7RgNSTpdiwPw+/Zl1E8zfdzdkBE8P9ce+QQIwI7sVByERkFRhuTGC4IXtUXHkF36YX4ptDBci+eHWCQE0vV9wTG4h7YgOh6cXbVkRkuRhuTGC4IXsmiiLS8yrw9aECbD5chKrfVygHgJF9e+HeIRpMDFfxthURWZyO/P6W9DGK1157DYIgtHiFhISY3Oerr75CSEgIXFxcEBERgR9++MFM1RJZP0EQMCSoJxKnR2D/S+Pxz5nRGDPAG4IA7Mu+hOe/Ooxhb6bi+a8O47fschgMdvVvHyKyEZL/8ywsLAypqanG946ObZe0d+9ezJo1C4mJiZg8eTLWr1+PhIQEpKenIzw83BzlEtkMV2cHTI0OwNToABRVXMG36QX4+lABzpfX4utDTX/u3csN98QGYnpsAG9bEZHVkPS21GuvvYbk5GRkZma2q/2MGTNQU1ODzZs3G7eNHDkS0dHRWLFiRbuOwdtSRG0TRRGHci833bY6Uozqa25bxfX1wr1DAjExQgU3Z8n/XUREdsZqbksBwNmzZ6FWq9G3b1/Mnj0beXl5bbZNS0vD+PHjW2yLj49HWlpam/vodDpotdoWLyJqnSAIGNqnF5bdE4kDL43H+zOiMbq/FwQBSMsux5+/Ooxhb6TiL18fxv6cS7CzIXtEZCUk/efXiBEjsHbtWgwaNAjFxcVYsmQJxowZg2PHjsHDw+O69iUlJfDz82uxzc/PDyUlJW3+jMTERCxZsqTLayeyda7ODkiICUBCTAAKLtciKb0QX6cXILe8FhsPFmDjwabbVvcOCcS0GN62IiLLYVFPS1VUVCAoKAjvvfceHn744es+d3Z2xqeffopZs2YZt3300UdYsmQJSktLWz2mTqeDTqczvtdqtdBoNLwtRdQJoijiYO5lfH2wAJuPXF3AEwCGBPVEQrQakyLV6OXuLGGVRGSLOnJbyqJunHt6emLgwIHIyspq9XOVSnVdiCktLYVKpWrzmHK5HHK5vEvrJLJXgiBgWJ9eGNanF16dMhhbj5Xgm/QC7D1XjkO5l3Eo9zKWfH8CYwf6YGpMAO4I9YOrs4PUZRORnZF8zM21qqurce7cOfj7+7f6eVxcHHbs2NFiW0pKCuLi4sxRHhFdw83ZEdNjA/HF/JFIe3EcXrorFGFqBRoNInacKsPTGzIw9I0UPPffTOw6cwGNeoPUJRORnZD0ttTzzz+Pu+++G0FBQSgqKsKrr76KzMxMnDhxAj4+PpgzZw4CAgKQmJgIoOlR8LFjx2LZsmWYNGkSvvzyS7z11lsdehScT0sRda+ssiokZxRh0+FC5F+6Ytzu3UOOyZH+SIgJQFSgEoLAZR+IqP2s5rZUQUEBZs2ahfLycvj4+OCWW27Bvn374OPjAwDIy8uDTHa1c2nUqFFYv349Xn75Zfz1r3/FgAEDkJyczDluiCxIf18PPB8/CH++cyDS8y4jOaMIm48U4WK1Dmv3nsfavecR7O2OKVFqJMQEINjbXeqSicjGWNSAYnNgzw2R+TXoDdh99gKSM4qw/UQJ6hqu3qKK0ngiIVqNyZFq+HhwfBwRtY5rS5nAcEMkrRpdI7afKEFyRhF+zboI/e9LPDjIBIzu742EaDXuDFOhB9e3IqJrMNyYwHBDZDkuVOmw+UgRkjOLcDi/wrjdxUmGOwarkBCtxh8G+sDJwaKefSAiCTDcmMBwQ2SZci7WYFNmITZlFiHnYo1xe083J0yK9EdCdABie/eETMaByET2iOHGBIYbIssmiiKOFFQiObMQ3x8uxsXqq5NwBni6Ymq0GlOjAzBIdf0s5kRkuxhuTGC4IbIejXoD9p4rx6bMImw7XtJiIc8QlQcSYgJwd5QaAZ6uElZJRObAcGMCww2Rdapr0GPHyTIkZxZi5+kyNOiv/tU1PLgXpkarMSnCH55uXPqByBYx3JjAcENk/SprG/DDsWJsyizEbzmX0Py3mJOD0LT0Q3QAxnPpByKbwnBjAsMNkW0pqriC7w8XYVNmEU4Ua43b3Z0dEB+mwtSYAIzu5wVHPnFFZNUYbkxguCGyXWdLq7Aps7WlH5wxOVKNKdFqxGg8ufQDkRViuDGB4YbI9omiiPS8CmzKLMTmI8W4VFNv/Kx3LzfjE1f9fXtIWCURdQTDjQkMN0T2pUFvwK9ZF7EpoxDbT5Sitl5v/Cw8QIGpUQGYEq2Gn8JFwiqJ6EYYbkxguCGyX7X1jUg5UYrvMouw68wFNP6+9IMgAKP7eSMhJgATwrn0A5ElYrgxgeGGiADgUk09thwtxqaMQhzMvWzczqUfiCwTw40JDDdE9L/yL9ViU2Yhvs0oRPaFq0s/9HJ3xuRIfyTEBHAgMpHEGG5MYLghoraIoohjhVokZRTiu8NFLZZ+CPJyQ0J0ABJiAhDs7S5hlUT2ieHGBIYbImqPRr0Be86VIzmjEFuPleBKw9WByNEaT0yLCcDkSH949ZBLWCWR/WC4MYHhhog6qkbXNBA5KaMQu89ewO/jkOEgE/CHAU0Dke8crOKMyETdiOHGBIYbIroZZVV12Hy4GMmZhThSUGnc7u7sgPhwFabFBGBUP284yDg+h6grMdyYwHBDRF0lq6wamzILkZRRiILLV2dE9vWQY0qUGgkxAQhTKzgQmagLMNyYwHBDRF2taUbky0jKaJoRuaK2wfhZf98emB4bgIToAKg9XSWsksi6MdyYwHBDRN2pvtGAXWcuIDmjECknS1HfaADQNFFgXF8vTIsJwMQIf04USNRBDDcmMNwQkblo6xrw49FifJNeiP05l4zbXZxkiA9rGp9zS39vrlhO1A4MNyYw3BCRFPIv1SI5o2l8TvbFqxMF+njIMTVKjemxgRis5t9JRG1huDGB4YaIpCSKIjLzK5CUUYjvDxfh8jXjc0JUHpgW0zRRIBfyJGqJ4cYEhhsishT1jQbsPF2GpIxC7DhZhnp90/gcmQCM7u+NaTEBiA9TwZ3jc4gYbkxhuCEiS1RZ24DNR4uQlN5yIU83ZwdMCFNhWiznzyH7xnBjAsMNEVm63PIaJP0+Pie3vNa43U8hR0J0AKbFBiBExb+/yL4w3JjAcENE1qJp/pwKfJtegM1HilF55er4nMH+CkyPDcCUKDV8OT6H7ADDjQkMN0RkjXSNevx8qgzfphfi59NlaNA3/dUtE4BR/bwxNVqNCeEqeLg4SVwpUfdguDGB4YaIrN3lmnpsPlKEpIxCpOdVGLfLHWUYH+qHqdFq3DrIF86OnD+HbAfDjQkMN0RkS/LKa7EpsxDJmYU4d+Hq/DlKVyfcFeGPhGg1hvXpBRkHIpOV68jvb4uJ9cuWLYMgCFi4cGGbbdauXQtBEFq8XFx4r5mI7FdvLzc8NW4AUp8bi81P3YL5twTD10OOyisN2LA/DzNW7sOYv/2MZT+ewqkSrdTlEpmFRUyecODAAXz88ceIjIy8YVuFQoHTp08b33O1XSKipr8LwwOUCA9QYtFdodiXXY7kjEJsPVaCwoorWLHrHFbsOocQlQemRgdgSrQaAVzIk2yU5OGmuroas2fPxqpVq/DGG2/csL0gCFCpVGaojIjIOjnIBIzu743R/b2xNCEcP50qQ3JG00DkUyVVOLX1FN7eegrDg3shIToAd0Wo4OnmLHXZRF1G8ttSCxYswKRJkzB+/Ph2ta+urkZQUBA0Gg2mTp2K48ePm2yv0+mg1WpbvIiI7IWLkwPuivDHyjlDcfClO5A4PQIjgnsBAPbnXMJfk45i2JupeOSzg9hypBh1DXqJKya6eZL23Hz55ZdIT0/HgQMH2tV+0KBBWL16NSIjI1FZWYm///3vGDVqFI4fP47AwMBW90lMTMSSJUu6smwiIqukdHPCrOG9MWt4bxRVXMF3h4uQnFGIUyVVSDlRipQTpeghd8SEcBUSogMQ18+LMyKTVZLsaan8/HwMHToUKSkpxrE2t956K6Kjo/H++++36xgNDQ0IDQ3FrFmzsHTp0lbb6HQ66HQ643utVguNRsOnpYiIfne6pArJmYX4LrMIhRVXjNt9PeSYEqXGtNgADPZXcIwjScoqHgVPTk7GtGnT4ODgYNym1+shCAJkMhl0Ol2Lz9py3333wdHRERs2bGjXz+Wj4ERErTMYRBzMvYzkzEL8cLQYFdesWD7QrwemxQRiarQaag5EJglYRbipqqpCbm5ui20PPvggQkJC8MILLyA8PPyGx9Dr9QgLC8Ndd92F9957r10/l+GGiOjG6hsN2HXmApIyCpB6sgz1jU0rlgsCMDLYC9NiAjAhQgUFZ0QmM+nI72/Jxtx4eHhcF2Dc3d3h5eVl3D5nzhwEBAQgMTERAPD6669j5MiR6N+/PyoqKvDOO+8gNzcX8+fPN3v9RES2zNlRhjsG++GOwX6ovNKAH48WIymjEL/lXEJadjnSssuxeNMxjB/sh+kxAfjDQB84OUj+jAoRAAt4FNyUvLw8yGRXvyyXL1/GI488gpKSEvTs2RNDhgzB3r17MXjwYAmrJCKybUpXJ8wc3hszh/dGweVabMpsWvohq6waW44UY8uRYvRyd8bdkf5IiAlAtMaT43NIUlx+gYiIOkwURRwv0uLb9EJ8d7gIF6uvPrgR7O2OhOgATIsJQG8vNwmrJFtiFWNupMJwQ0TUtRr1BvyadRHJGYXYdrwUV66ZK2dIUE9MiwnApAh/9HTnRIHUeQw3JjDcEBF1nxpdI7YdL0FSRiH2ZF2E4fffME4OAm4b5ItpMQG4PdQXcscbPw1LdC2GGxMYboiIzKNUW4fvDxfh2/RCnCi+Oju8wsURkyL9kRAdwBXLqd0YbkxguCEiMr/TJVVIyijEpsxCFFfWGbcHeLri7ig1EmLUCFHx72RqG8ONCQw3RETSMRhE7MspR1J604rlVbpG42chKg9MiVZjSpQagT05EJlaYrgxgeGGiMgy1DXo8dOpMmzKLMTPpy6gXm8wfja8Ty9MiVZzIDIZMdyYwHBDRGR5Kmsb8OOxYiRnNk0UKF4zEHnsQB9MjQ7A+FA/uDpzILK9YrgxgeGGiMiyFVdewfeHi5CcUdRiILK7swPiw1SYGhOA0f284MgZke0Kw40JDDdERNbjbGkVNmUWYdPhQuRfurpiuXcPZ0yOVGNqtJozItsJhhsTGG6IiKyPKIpIz7uMTZlF2HykGJdq6o2f9fFyw5ToACREq9HXp4eEVVJ3YrgxgeGGiMi6NegN+PXsRSRnFmL7/8yIHBGgxNTfn7jyVbhIWCV1NYYbExhuiIhsR42uEaknS5GcUYhfzl6E/vcpkWUCMKqfN6bFBGBCuArucoteJ5rageHGBIYbIiLbVF6tw5ajxdiUWYRDuZeN212dHBAf5odpsYG4pb83HDgjslViuDGB4YaIyPblldciObMQ36YX4Hx5rXG7j4ccU6PUmBYbgMH+Cg5EtiIMNyYw3BAR2Q9RFJGRX4HkjEJ8f7gIl2sbjJ8N8vPAtNgAJEQHQKXk+BxLx3BjAsMNEZF9qm80YOfpMiRlFGLHyTLjjMiCAIzq54VpMYGYEK5CD47PsUgMNyYw3BARUWVtA7YcLUZSRgEOnL86PsfFSYb4MBWmxQTglv7enCjQgjDcmMBwQ0RE18q/VIukjEIkZRQi52KNcbuPhxxTotSYFhOAMDXH50iN4cYEhhsiImqNKIrIzK9AUivjcwb69cC0mEAkxKjhr3SVsEr7xXBjAsMNERHdSH2jAbvOXEBSRgFST5ahvvHq+Jy4vl6YFhOA+HAVFC5OEldqPxhuTGC4ISKijqi80oAfjhYjKb0Q+89fMm53dpDhDwN9MDnSH+NCfeHBoNOtGG5MYLghIqLOyr9Ui+SMQiRnFuLchavjc5wdZbh1oA8mRfpjXKgfn7jqBgw3JjDcEBHRzRJFEWdKq7HlSNNCntnXDESWO8pw2yBfTIr0x+0hvlz6oYsw3JjAcENERF1JFEWcKqnCliPF2HykqMWMyC5OMtwe4otJEWrcFuIDN2cGnc5iuDGB4YaIiLqLKIo4Uaz9PegUI+/S1aDj6uSA20N9MTnCH7cO8oWrs4OElVofhhsTGG6IiMgcRFHE8SItNh8pxpajRci/dMX4mZuzA8aF+mFShD9uHeQDFycGnRthuDGB4YaIiMxNFEUcLaw09ugUVlwNOu7ODhg/uCno/GEgg05bGG5MYLghIiIpiaKIwwWV2HKkCFuOFKOoss74WQ+5I+64Jug4O3L5h2YMNyYw3BARkaUwGERkFlRgy5Fi/HC0GMXXBJ2ebk6YHKnGtNgAxGg87X75B4YbExhuiIjIEhkMIjLyL+P7w8XYcrQYF6p0xs+Cvd2REB2AhBg1grzcJaxSOgw3JjDcEBGRpWvUG7DnXDmS0guw7XgprjTojZ8NCeqJaTEBmBzpD083ZwmrNK+O/P62mJt5y5YtgyAIWLhwocl2X331FUJCQuDi4oKIiAj88MMP5imQiIjITBwdZBg70Afvz4zBwZfH4737ozBmgDdkAnAo9zJeTj6GYW+m4tHPDmLrsWLoGvU3PqgdsYjZhA4cOICPP/4YkZGRJtvt3bsXs2bNQmJiIiZPnoz169cjISEB6enpCA8PN1O1RERE5uMud8T02EBMjw1EqbYO32UW4duMQpws1mL7iVJsP1EKhYsjJkWqMT02AEODetr9+BzJb0tVV1cjNjYWH330Ed544w1ER0fj/fffb7XtjBkzUFNTg82bNxu3jRw5EtHR0VixYkW7fh5vSxERkS04VaJFUkYhNmUUoUR7dSCyppcrpkUHICEmAH19ekhYYdeyqttSCxYswKRJkzB+/Pgbtk1LS7uuXXx8PNLS0trcR6fTQavVtngRERFZuxCVAosmhmLPi7dj/fwRuHdIINydHZB/6Qr+9VMWbn93F6b+ew8+3Xse5dW6Gx/Qhkh6W+rLL79Eeno6Dhw40K72JSUl8PPza7HNz88PJSUlbe6TmJiIJUuW3FSdRERElspBJmBUf2+M6u+NpVPDkXKyFEnpBfjl7EUczq/A4fwKLN18AmMH+mBabADGh/rZ/ESBkoWb/Px8PPPMM0hJSYGLi0u3/ZxFixbhueeeM77XarXQaDTd9vOIiIik4ursgClRakyJUuNClQ6bjxQhKaMQRwoqseNUGXacKoOH3BHx4SpMDFdhdH9vmww6koWbQ4cOoaysDLGxscZter0ev/zyCz788EPodDo4OLT8D65SqVBaWtpiW2lpKVQqVZs/Ry6XQy6Xd23xREREFs7HQ44HRwfjwdHByCqrRnJGIZIyClFYcQVfHyrA14cK4O7sgNtCfDEhXIVbB/mih9winjO6aZINKK6qqkJubm6LbQ8++CBCQkLwwgsvtPr004wZM1BbW4vvv//euG3UqFGIjIzkgGIiIqIbMBhEHMy9jB+OFmPb8ZIWMyI7O8rwhwHeiA9TYXyoH3q6W9YcOlY7id+tt97a4mmpOXPmICAgAImJiQCaHgUfO3Ysli1bhkmTJuHLL7/EW2+91aFHwRluiIiImta4OlJQia3HS7D1WAlyLtYYP3OQCRjZtxcmhKlwZ5gKforuGz7SXh35/W3R/U95eXmQya4+0DVq1CisX78eL7/8Mv76179iwIABSE5O5hw3REREHSQIAqI0nojSeOIv8YNwtqwaW481BZ0TxVrsySrHnqxyLN50HLG9PTEx3B/xYSr09nKTuvQbsqieG3Ngzw0REZFpueU12PZ7j056XkWLzwb7KzAhXIUJ4SoM8O1htgkDrfa2lDkw3BAREbVfqbYO24+XYOvxEuzLvgS94Wps6OvtjvhwFSaEqRAZqOzWoMNwYwLDDRERUedcrqlH6slSbDtegl/OXkR9o8H4mVrpgjvDmnp0hvXpBQdZ1wYdhhsTGG6IiIhuXrWuETtPl2HrsRL8fKoMNfVXF+/s4+WGn5+/tUt7cmxmQDERERFZph5yR0yOVGNypBp1DXrsybqIrcdKkHKyFNEaT0kX72S4ISIiopvi4uSAcaF+GBfqh0a9Adq6RknrkXzhTCIiIrIdjg4y9JJ4AkCGGyIiIrIpDDdERERkUxhuiIiIyKYw3BAREZFNYbghIiIim8JwQ0RERDaF4YaIiIhsCsMNERER2RSGGyIiIrIpDDdERERkUxhuiIiIyKYw3BAREZFNYbghIiIim+IodQHmJooiAECr1UpcCREREbVX8+/t5t/jpthduKmqqgIAaDQaiSshIiKijqqqqoJSqTTZRhDbE4FsiMFgQFFRETw8PCAIQpceW6vVQqPRID8/HwqFokuPbWl4rrbLns6X52q77Ol87eVcRVFEVVUV1Go1ZDLTo2rsrudGJpMhMDCwW3+GQqGw6f/BrsVztV32dL48V9tlT+drD+d6ox6bZhxQTERERDaF4YaIiIhsCsNNF5LL5Xj11Vchl8ulLqXb8Vxtlz2dL8/VdtnT+drTubaX3Q0oJiIiItvGnhsiIiKyKQw3REREZFMYboiIiMimMNwQERGRTWG46aB///vf6NOnD1xcXDBixAjs37/fZPuvvvoKISEhcHFxQUREBH744QczVdp5iYmJGDZsGDw8PODr64uEhAScPn3a5D5r166FIAgtXi4uLmaq+Oa89tpr19UeEhJich9rvK4A0KdPn+vOVRAELFiwoNX21nRdf/nlF9x9991Qq9UQBAHJycktPhdFEa+88gr8/f3h6uqK8ePH4+zZszc8bke/8+Zi6nwbGhrwwgsvICIiAu7u7lCr1ZgzZw6KiopMHrMz3wVzuNG1nTdv3nV1T5gw4YbHtcRre6Nzbe37KwgC3nnnnTaPaanXtTsx3HTAf//7Xzz33HN49dVXkZ6ejqioKMTHx6OsrKzV9nv37sWsWbPw8MMPIyMjAwkJCUhISMCxY8fMXHnH7Nq1CwsWLMC+ffuQkpKChoYG3HnnnaipqTG5n0KhQHFxsfGVm5trpopvXlhYWIvaf/311zbbWut1BYADBw60OM+UlBQAwH333dfmPtZyXWtqahAVFYV///vfrX7+t7/9Df/617+wYsUK/Pbbb3B3d0d8fDzq6uraPGZHv/PmZOp8a2trkZ6ejsWLFyM9PR3ffvstTp8+jSlTptzwuB35LpjLja4tAEyYMKFF3Rs2bDB5TEu9tjc612vPsbi4GKtXr4YgCLjnnntMHtcSr2u3Eqndhg8fLi5YsMD4Xq/Xi2q1WkxMTGy1/f333y9OmjSpxbYRI0aIjz32WLfW2dXKyspEAOKuXbvabLNmzRpRqVSar6gu9Oqrr4pRUVHtbm8r11UURfGZZ54R+/XrJxoMhlY/t9brCkBMSkoyvjcYDKJKpRLfeecd47aKigpRLpeLGzZsaPM4Hf3OS+V/z7c1+/fvFwGIubm5bbbp6HdBCq2d69y5c8WpU6d26DjWcG3bc12nTp0q3n777SbbWMN17WrsuWmn+vp6HDp0COPHjzduk8lkGD9+PNLS0lrdJy0trUV7AIiPj2+zvaWqrKwEAPTq1ctku+rqagQFBUGj0WDq1Kk4fvy4OcrrEmfPnoVarUbfvn0xe/Zs5OXltdnWVq5rfX091q1bh4ceesjkIrLWfF2b5eTkoKSkpMV1UyqVGDFiRJvXrTPfeUtWWVkJQRDg6elpsl1HvguWZOfOnfD19cWgQYPwxBNPoLy8vM22tnJtS0tLsWXLFjz88MM3bGut17WzGG7a6eLFi9Dr9fDz82ux3c/PDyUlJa3uU1JS0qH2lshgMGDhwoUYPXo0wsPD22w3aNAgrF69Gps2bcK6detgMBgwatQoFBQUmLHazhkxYgTWrl2LrVu3Yvny5cjJycGYMWNQVVXVantbuK4AkJycjIqKCsybN6/NNtZ8Xa/VfG06ct068523VHV1dXjhhRcwa9YskwsrdvS7YCkmTJiAzz77DDt27MDbb7+NXbt2YeLEidDr9a22t5Vr++mnn8LDwwPTp0832c5ar+vNsLtVwaljFixYgGPHjt3w/mxcXBzi4uKM70eNGoXQ0FB8/PHHWLp0aXeXeVMmTpxo/HNkZCRGjBiBoKAgbNy4sV3/IrJWn3zyCSZOnAi1Wt1mG2u+rtSkoaEB999/P0RRxPLly022tdbvwsyZM41/joiIQGRkJPr164edO3di3LhxElbWvVavXo3Zs2ffcJC/tV7Xm8Gem3by9vaGg4MDSktLW2wvLS2FSqVqdR+VStWh9pbmySefxObNm/Hzzz8jMDCwQ/s6OTkhJiYGWVlZ3VRd9/H09MTAgQPbrN3arysA5ObmIjU1FfPnz+/QftZ6XZuvTUeuW2e+85amOdjk5uYiJSXFZK9Na270XbBUffv2hbe3d5t128K13b17N06fPt3h7zBgvde1Ixhu2snZ2RlDhgzBjh07jNsMBgN27NjR4l+214qLi2vRHgBSUlLabG8pRFHEk08+iaSkJPz0008IDg7u8DH0ej2OHj0Kf3//bqiwe1VXV+PcuXNt1m6t1/Vaa9asga+vLyZNmtSh/az1ugYHB0OlUrW4blqtFr/99lub160z33lL0hxszp49i9TUVHh5eXX4GDf6LliqgoIClJeXt1m3tV9boKnndciQIYiKiurwvtZ6XTtE6hHN1uTLL78U5XK5uHbtWvHEiRPio48+Knp6eoolJSWiKIriH//4R/HFF180tt+zZ4/o6Ogo/v3vfxdPnjwpvvrqq6KTk5N49OhRqU6hXZ544glRqVSKO3fuFIuLi42v2tpaY5v/PdclS5aI27ZtE8+dOyceOnRInDlzpuji4iIeP35cilPokD//+c/izp07xZycHHHPnj3i+PHjRW9vb7GsrEwURdu5rs30er3Yu3dv8YUXXrjuM2u+rlVVVWJGRoaYkZEhAhDfe+89MSMjw/h00LJly0RPT09x06ZN4pEjR8SpU6eKwcHB4pUrV4zHuP3228UPPvjA+P5G33kpmTrf+vp6ccqUKWJgYKCYmZnZ4nus0+mMx/jf873Rd0Eqps61qqpKfP7558W0tDQxJydHTE1NFWNjY8UBAwaIdXV1xmNYy7W90f/HoiiKlZWVopubm7h8+fJWj2Et17U7Mdx00AcffCD27t1bdHZ2FocPHy7u27fP+NnYsWPFuXPntmi/ceNGceDAgaKzs7MYFhYmbtmyxcwVdxyAVl9r1qwxtvnfc124cKHxv4ufn5941113ienp6eYvvhNmzJgh+vv7i87OzmJAQIA4Y8YMMSsry/i5rVzXZtu2bRMBiKdPn77uM2u+rj///HOr/982n4/BYBAXL14s+vn5iXK5XBw3btx1/w2CgoLEV199tcU2U995KZk635ycnDa/xz///LPxGP97vjf6LkjF1LnW1taKd955p+jj4yM6OTmJQUFB4iOPPHJdSLGWa3uj/49FURQ//vhj0dXVVayoqGj1GNZyXbuTIIqi2K1dQ0RERERmxDE3REREZFMYboiIiMimMNwQERGRTWG4ISIiIpvCcENEREQ2heGGiIiIbArDDREREdkUhhsiIiKyKQw3RGT3BEFAcnKy1GUQURdhuCEiSc2bNw+CIFz3mjBhgtSlEZGVcpS6ACKiCRMmYM2aNS22yeVyiaohImvHnhsikpxcLodKpWrx6tmzJ4CmW0bLly/HxIkT4erqir59++Lrr79usf/Ro0dx++23w9XVFV5eXnj00UdRXV3dos3q1asRFhYGuVwOf39/PPnkky0+v3jxIqZNmwY3NzcMGDAA3333XfeeNBF1G4YbIrJ4ixcvxj333IPDhw9j9uzZmDlzJk6ePAkAqKmpQXx8PHr27IkDBw7gq6++Qmpqaovwsnz5cixYsACPPvoojh49iu+++w79+/dv8TOWLFmC+++/H0eOHMFdd92F2bNn49KlS2Y9TyLqIlIvS05E9m3u3Lmig4OD6O7u3uL15ptviqIoigDExx9/vMU+I0aMEJ944glRFEVx5cqVYs+ePcXq6mrj51u2bBFlMplYUlIiiqIoqtVq8aWXXmqzBgDiyy+/bHxfXV0tAhB//PHHLjtPIjIfjrkhIsnddtttWL58eYttvXr1Mv45Li6uxWdxcXHIzMwEAJw8eRJRUVFwd3c3fj569GgYDAacPn0agiCgqKgI48aNM1lDZGSk8c/u7u5QKBQoKyvr7CkRkYQYbohIcu7u7tfdJuoqrq6u7Wrn5OTU4r0gCDAYDN1REhF1M465ISKLt2/fvuveh4aGAgBCQ0Nx+PBh1NTUGD/fs2cPZDIZBg0aBA8PD/Tp0wc7duwwa81EJB323BCR5HQ6HUpKSlpsc3R0hLe3NwDgq6++wtChQ3HLLbfgiy++wP79+/HJJ58AAGbPno1XX30Vc+fOxWuvvYYLFy7gqaeewh//+Ef4+fkBAF577TU8/vjj8PX1xcSJE1FVVYU9e/bgqaeeMu+JEpFZMNwQkeS2bt0Kf3//FtsGDRqEU6dOAWh6kunLL7/En/70J/j7+2PDhg0YPHgwAMDNzQ3btm3DM888g2HDhsHNzQ333HMP3nvvPeOx5s6di7q6OvzjH//A888/D29vb9x7773mO0EiMitBFEVR6iKIiNoiCAKSkpKQkJAgdSlEZCU45oaIiIhsCsMNERER2RSOuSEii8Y750TUUey5ISIiIpvCcENEREQ2heGGiIiIbArDDREREdkUhhsiIiKyKQw3REREZFMYboiIiMimMNwQERGRTfn/aaDqQaPvUZcAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.plot(losses)\n", "plt.ylabel('Loss')\n", "plt.xlabel('Epoch')" ] }, { "cell_type": "markdown", "id": "b6a53f16", "metadata": {}, "source": [ "\n", "# 13 - Summarize some Sentences!\n", "\n", "Below you can see an example of summarization of a sentence from the training set and a sentence from the test set. See if you notice anything interesting about them!" ] }, { "cell_type": "code", "execution_count": 36, "id": "2493b755", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training set example:\n", "[SOS] amanda: i baked cookies. do you want some? jerry: sure! amanda: i'll bring you tomorrow :-) [EOS]\n", "\n", "Human written summary:\n", "[SOS] amanda baked cookies and will bring jerry some tomorrow. [EOS]\n", "\n", "Model written summary:\n", "[SOS] amanda will bring some cookies [EOS]\n" ] } ], "source": [ "training_set_example = 0\n", "\n", "# Check a summary of a document from the training set\n", "print('Training set example:')\n", "print(document[training_set_example])\n", "print('\\nHuman written summary:')\n", "print(summary[training_set_example])\n", "print('\\nModel written summary:')\n", "print(summarize(transformer, document[training_set_example]))" ] }, { "cell_type": "code", "execution_count": 37, "id": "15baaa47", "metadata": { "deletable": false, "editable": false, "tags": [ "graded" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test set example:\n", "[SOS] will: hey babe, what do you want for dinner tonight? emma: gah, don't even worry about it tonight will: what do you mean? everything ok? emma: not really, but it's ok, don't worry about cooking though, i'm not hungry will: well what time will you be home? emma: soon, hopefully will: you sure? maybe you want me to pick you up? emma: no no it's alright. i'll be home soon, i'll tell you when i get home. will: alright, love you. emma: love you too. [EOS]\n", "\n", "Human written summary:\n", "[SOS] emma will be home soon and she will let will know. [EOS]\n", "\n", "Model written summary:\n", "[SOS] emma will pick up with emma at home tonight [EOS]\n" ] } ], "source": [ "test_set_example = 3\n", "\n", "# Check a summary of a document from the test set\n", "print('Test set example:')\n", "print(document_test[test_set_example])\n", "print('\\nHuman written summary:')\n", "print(summary_test[test_set_example])\n", "print('\\nModel written summary:')\n", "print(summarize(transformer, document_test[test_set_example]))" ] }, { "cell_type": "markdown", "id": "aebd7ef5", "metadata": {}, "source": [ "If you critically examine the output of the model, you can notice a few things:\n", " - In the training set the model output is (almost) identical to the real output (already after 20 epochs and even more so with more epochs). This might be because the training set is relatively small and the model is relatively big and has thus learned the sentences in the training set by heart (overfitting).\n", " - While the performance on the training set looks amazing, it is not so good on the test set. The model overfits, but fails to generalize. Again an easy candidate to blame is the small training set and a comparatively large model, but there might be a variety of other factors.\n", " - Look at the test set example 3 and its summarization. Would you summarize it the same way as it is written here? Sometimes the data may be ambiguous. And the training of **your model can only be as good as your data**.\n", "\n", "Here you only use a small dataset, to show that something can be learned in a reasonable amount of time in a relatively small environment. Generally, large transformers are trained on more than one task and on very large quantities of data to achieve superb performance. You will learn more about this in the rest of this course." ] }, { "cell_type": "markdown", "id": "41014aac", "metadata": {}, "source": [ "**Congratulations on finishing this week's assignment!** You did a lot of work and now you should have a better understanding of the Transformers and their building blocks (encoder and decoder) and how they can be used for text summarization. And remember: you dont need to change much to use the same model for a translator, just change the dataset and it should work!\n", "\n", "**Keep it up!**" ] } ], "metadata": { "grader_version": "1", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }