{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 实验说明\n", "\n", "本实验的主要内容为：使用一种名为[DistilBERT](https://arxiv.org/abs/1910.01108)的BERT变体来自动识别人们在Twitter上发布文本的情绪状态。\n", "\n", "实验基础代码已经在下面给出。你需要保证自己在熟悉了基础代码后，完成以下任务（使用CTRL+F，搜索‘TODO’，可以快速定位任务位置）：\n", "\n", "1. （2 分）创建一个 `token2idx` 字典，将字符（key）映射为整数（value），用于实现字符级别的标记化操作；\n", "2. （2 分）完善提取隐藏层 [CLS] 特征的函数；（2分）\n", "3. （2 分）使用 Scikit-Learn 库中的不同分类模型对提取的特征进行分类，并比较不同模型的分类效果；\n", "4. （2 分）调整模型参数，以训练出更高分类准确率的模型，探讨不同参数设置对模型训练效果的影响；\n", "5. （1 分）分析分类错误的样本为什么会被分错，分类正确的样本为什么会被分对（选做）；\n", "6. （1 分）生成 5 个对抗样本，尽可能让你的模型无法正确判断其情感类别，并尝试解释其中的原因（选做）。\n", "7. （10分）撰写实验报告\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 实验要求\n", "- 截止日期：$\\text{2024.12.15 23:59}$\n", "- 提交内容：源代码、实验报告\n", "- 报告要求：\n", " - 内容包括摘要、引言、方法、实验以及总结部分；\n", " - 4页左右，PDF格式。\n", " - 如果你在探索中有任何其它发现或想法，也请加入到报告中。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:42.511025Z", "iopub.status.busy": "2025-05-30T08:57:42.510863Z", "iopub.status.idle": "2025-05-30T08:57:44.762003Z", "shell.execute_reply": "2025-05-30T08:57:44.761444Z", "shell.execute_reply.started": "2025-05-30T08:57:42.511005Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GPU was detected! This notebook can be *very* fast with a GPU 🐰\n", "Using transformers v4.45.0.dev0\n", "Using datasets v2.20.0\n" ] } ], "source": [ "# from install import *\n", "# install_requirements()\n", "from utils import *\n", "setup_chapter()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 文本情感分类" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "文本分类是自然语言处理中最常见的任务之一；它可用于各种应用程序，例如将客户反馈标记为不同类别。很可能你的电子邮件程序的垃圾邮件过滤器正在使用文本分类来保护你的收件箱免受大量不必要的垃圾邮件的困扰！" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "另一种常见的文本分类是情感分析，它旨在确定给定文本的情感。例如，像特斯拉这样的公司可能会分析Twitter上的帖子，以确定人们是否喜欢它的新车顶。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"Tesla" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在想象一下，你是一位数据科学家，需要构建一个系统，可以自动识别人们在Twitter上对你公司产品表达的情绪状态，比如“愤怒”或“喜悦”。在这一章中，我们将使用一种名为DistilBERT[V. Sanh et al., [\"DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter\"](https://arxiv.org/abs/1910.01108), (2019).]的BERT变体来解决这个任务。这个模型的主要优势在于它能够达到与BERT相媲美的性能，同时体积更小、效率更高。这使我们能够在几分钟内训练出一个分类器，如果你想训练一个更大的BERT模型，只需要简单地更改预训练模型的检查点。检查点对应于被加载到给定转换器架构中的一组权重。\n", "\n", "这也将是我们首次接触到Hugging Face生态系统中的三个核心库：Datasets、Tokenizers和Transformers。正如所示，这些库将使我们能够快速地从原始文本转变为一个经过微调的模型，可以用于对新推文进行推断。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"Hugging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 数据集" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为了构建我们的情感检测器，我们将使用一篇文章中的精彩数据集[E. Saravia et al., \"CARER: Contextualized Affect Representations for Emotion Recognition,\" EMNLP, (2018), http://dx.doi.org/10.18653/v1/D18-1404.] ，该文章探讨了情绪如何在英语 Twitter 消息中被表达。与大多数情感分析数据集不同，这个数据集包含六种基本情感：愤怒、厌恶、恐惧、喜悦、悲伤和惊讶。给定一条推文，我们的任务将是训练一个模型，可以将其分类为这些情绪中的一种。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "让我们用 `load_dataset()` 函数加载情`emotion`据集:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.763911Z", "iopub.status.busy": "2025-05-30T08:57:44.763409Z", "iopub.status.idle": "2025-05-30T08:57:44.893323Z", "shell.execute_reply": "2025-05-30T08:57:44.892678Z", "shell.execute_reply.started": "2025-05-30T08:57:44.763889Z" } }, "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "emotions = load_dataset(\"./emotions\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "如果我们查看我们的`emotions`对象：" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.894618Z", "iopub.status.busy": "2025-05-30T08:57:44.894224Z", "iopub.status.idle": "2025-05-30T08:57:44.899560Z", "shell.execute_reply": "2025-05-30T08:57:44.899022Z", "shell.execute_reply.started": "2025-05-30T08:57:44.894593Z" } }, "outputs": [ { "data": { "text/plain": [ "DatasetDict({\n", " train: Dataset({\n", " features: ['text', 'label'],\n", " num_rows: 16000\n", " })\n", " validation: Dataset({\n", " features: ['text', 'label'],\n", " num_rows: 2000\n", " })\n", " test: Dataset({\n", " features: ['text', 'label'],\n", " num_rows: 2000\n", " })\n", "})" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "emotions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们看到它类似于一个Python字典，每个键对应不同的分割。我们可以使用通常的字典语法来访问单个分割：" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.900384Z", "iopub.status.busy": "2025-05-30T08:57:44.900167Z", "iopub.status.idle": "2025-05-30T08:57:44.905569Z", "shell.execute_reply": "2025-05-30T08:57:44.904937Z", "shell.execute_reply.started": "2025-05-30T08:57:44.900361Z" } }, "outputs": [ { "data": { "text/plain": [ "Dataset({\n", " features: ['text', 'label'],\n", " num_rows: 16000\n", "})" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_ds = emotions[\"train\"]\n", "train_ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "返回一个 `Dataset` 类的实例。 `Dataset` 对象是图像处理中的核心数据结构之一，我们将在本书的过程中控制许多其特性。首先，它的行为类似于普通的 Python 数组或列表，因此我们可以查询它的长度：" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.906388Z", "iopub.status.busy": "2025-05-30T08:57:44.906168Z", "iopub.status.idle": "2025-05-30T08:57:44.910321Z", "shell.execute_reply": "2025-05-30T08:57:44.909704Z", "shell.execute_reply.started": "2025-05-30T08:57:44.906365Z" } }, "outputs": [ { "data": { "text/plain": [ "16000" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(train_ds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "或者通过它的索引来访问单个示例：" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.911117Z", "iopub.status.busy": "2025-05-30T08:57:44.910905Z", "iopub.status.idle": "2025-05-30T08:57:44.915339Z", "shell.execute_reply": "2025-05-30T08:57:44.914704Z", "shell.execute_reply.started": "2025-05-30T08:57:44.911095Z" } }, "outputs": [ { "data": { "text/plain": [ "{'text': 'i didnt feel humiliated', 'label': 0}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_ds[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在这里，我们可以看到一行数据被表示为一个字典，其中键对应于列名：" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.917752Z", "iopub.status.busy": "2025-05-30T08:57:44.917534Z", "iopub.status.idle": "2025-05-30T08:57:44.921793Z", "shell.execute_reply": "2025-05-30T08:57:44.921171Z", "shell.execute_reply.started": "2025-05-30T08:57:44.917730Z" } }, "outputs": [ { "data": { "text/plain": [ "['text', 'label']" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_ds.column_names" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "值是推文和情感。这反映出数据集基于[_Apache Arrow_](https://arrow.apache.org/)，它定义了一种类型列格式，比原生Python更高效地使用内存。我们可以通过访问`Dataset`对象的`features`属性来看到底层使用的数据类型：" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.922611Z", "iopub.status.busy": "2025-05-30T08:57:44.922406Z", "iopub.status.idle": "2025-05-30T08:57:44.926150Z", "shell.execute_reply": "2025-05-30T08:57:44.925609Z", "shell.execute_reply.started": "2025-05-30T08:57:44.922590Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['sadness',\n", "'joy', 'love', 'anger', 'fear', 'surprise'], id=None)}\n" ] } ], "source": [ "print(train_ds.features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在这种情况下，`text`列的数据类型是`string`，而`label`列是一个特殊的`ClassLabel`对象，其中包含有关类别名称及其与整数的映射信息。我们还可以用切片访问几行：" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.926959Z", "iopub.status.busy": "2025-05-30T08:57:44.926739Z", "iopub.status.idle": "2025-05-30T08:57:44.931190Z", "shell.execute_reply": "2025-05-30T08:57:44.930569Z", "shell.execute_reply.started": "2025-05-30T08:57:44.926937Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'text': ['i didnt feel humiliated', 'i can go from feeling so hopeless to so\n", "damned hopeful just from being around someone who cares and is awake', 'im\n", "grabbing a minute to post i feel greedy wrong', 'i am ever feeling nostalgic\n", "about the fireplace i will know that it is still on the property', 'i am feeling\n", "grouchy'], 'label': [0, 0, 3, 2, 3]}\n" ] } ], "source": [ "print(train_ds[:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "请注意，在这种情况下，字典的值现在是列表而不是单个元素。我们还可以按名称获取整列：" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.931991Z", "iopub.status.busy": "2025-05-30T08:57:44.931779Z", "iopub.status.idle": "2025-05-30T08:57:44.949563Z", "shell.execute_reply": "2025-05-30T08:57:44.949033Z", "shell.execute_reply.started": "2025-05-30T08:57:44.931969Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['i didnt feel humiliated', 'i can go from feeling so hopeless to so damned\n", "hopeful just from being around someone who cares and is awake', 'im grabbing a\n", "minute to post i feel greedy wrong', 'i am ever feeling nostalgic about the\n", "fireplace i will know that it is still on the property', 'i am feeling grouchy']\n" ] } ], "source": [ "print(train_ds[\"text\"][:5])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们已经看到了如何使用hf库加载和检查数据，让我们对推文内容进行一些健全性检查。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 从 Datasets 到 DataFrames" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "虽然`Datasets`库提供了许多低级功能来切割和分析数据，但通常将`Dataset`对象转换为Pandas `DataFrame` 是方便的，这样我们就可以使用高级API来进行数据可视化。为了实现转换，数据集提供了`set_format()`方法，允许我们更改数据集的输出格式。请注意，这不会改变底层数据格式（即Arrow表），如果需要，您可以随后切换到另一种格式。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.950605Z", "iopub.status.busy": "2025-05-30T08:57:44.950186Z", "iopub.status.idle": "2025-05-30T08:57:44.961528Z", "shell.execute_reply": "2025-05-30T08:57:44.961098Z", "shell.execute_reply.started": "2025-05-30T08:57:44.950584Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabel
0i didnt feel humiliated0
1i can go from feeling so hopeless to so damned...0
2im grabbing a minute to post i feel greedy wrong3
3i am ever feeling nostalgic about the fireplac...2
4i am feeling grouchy3
\n", "
" ], "text/plain": [ " text label\n", "0 i didnt feel humiliated 0\n", "1 i can go from feeling so hopeless to so damned... 0\n", "2 im grabbing a minute to post i feel greedy wrong 3\n", "3 i am ever feeling nostalgic about the fireplac... 2\n", "4 i am feeling grouchy 3" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "emotions.set_format(type=\"pandas\")\n", "df = emotions[\"train\"][:]\n", "df.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "正如你所看到的，列标题已经被保留，前几行与我们以前对数据的看法相匹配。然而，标签表示为整数，所以让我们使用标签特征的`int2str()`方法，在我们的DataFrame中创建一个新列，其中包含相应的标签名称：" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:44.962308Z", "iopub.status.busy": "2025-05-30T08:57:44.962137Z", "iopub.status.idle": "2025-05-30T08:57:45.729319Z", "shell.execute_reply": "2025-05-30T08:57:45.728541Z", "shell.execute_reply.started": "2025-05-30T08:57:44.962290Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabellabel_name
0i didnt feel humiliated0sadness
1i can go from feeling so hopeless to so damned...0sadness
2im grabbing a minute to post i feel greedy wrong3anger
3i am ever feeling nostalgic about the fireplac...2love
4i am feeling grouchy3anger
5ive been feeling a little burdened lately wasn...0sadness
6ive been taking or milligrams or times recomme...5surprise
\n", "
" ], "text/plain": [ " text label label_name\n", "0 i didnt feel humiliated 0 sadness\n", "1 i can go from feeling so hopeless to so damned... 0 sadness\n", "2 im grabbing a minute to post i feel greedy wrong 3 anger\n", "3 i am ever feeling nostalgic about the fireplac... 2 love\n", "4 i am feeling grouchy 3 anger\n", "5 ive been feeling a little burdened lately wasn... 0 sadness\n", "6 ive been taking or milligrams or times recomme... 5 surprise" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def label_int2str(row):\n", " return emotions[\"train\"].features[\"label\"].int2str(row)\n", "\n", "df[\"label_name\"] = df[\"label\"].apply(label_int2str)\n", "df.head(7) # 更改这里的数字，可以显示更多的数据" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在开始构建分类器之前，让我们仔细研究一下数据集。正如Andrej Karpathy在他著名的博客文章[\"A Recipe for Training Neural Networks\"](https://karpathy.github.io/2019/04/25/recipe)中所指出的，与数据成为“一体”是训练优秀模型的重要步骤！" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 查看数据的分布" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "当你在处理文本分类问题时，检查一下各类别示例的分布是个好主意。一个类别分布不均的数据集可能需要在训练损失和评估指标方面进行不同处理，与一个平衡数据集相比。\n", "\n", "通过 Pandas 和 Matplotlib，我们可以快速可视化类别分布，如下所示：" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:45.730244Z", "iopub.status.busy": "2025-05-30T08:57:45.730059Z", "iopub.status.idle": "2025-05-30T08:57:46.105279Z", "shell.execute_reply": "2025-05-30T08:57:46.104773Z", "shell.execute_reply.started": "2025-05-30T08:57:45.730225Z" } }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDIwLjg4NzUgMjcxLjQyNTYyNSBdIC9Db250ZW50cyA5IDAgUiAvQW5ub3RzIDEwIDAgUiA+PgplbmRvYmoKOSAwIG9iago8PCAvTGVuZ3RoIDEyIDAgUiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJy9V01vEzEQ9dm/wkc4MPHM+GuPbYFK3EojcUAIhTQNrbZAG0HVf8/YSXa9kN1woYkcrd/afs/jmfFk9nr162a5en9+as4u9azvLTcaza20tbHmVtqjQXMuba2t9O60IwspRS+dtu9QRHDkA3lB7bD7VetrPTuRJTYy6VzHtJvjIckAWRIZQoW0FeIi+JShblaHlHXvzR/LMTtIhgghBvOwMh/MN2G14BwjR/KukU7cPlkyD7JPPfG242UPNmS1iATyDvdI2yOewbtabAcUrRfmv6vtmCOBLcZFH0CGU3R7rK2xxoK3teQOeHbJaBM0qWhuHNjIbHuwHYAUwYVadY88v2znoXFZNtkIYtMoBHuwHYDB5aioZXfI88tO2WmzbBZvkEYudWBbg2QReBiDHfLssgkbSCUOuYkQnQuOO7AdgJyAB9HYI53s2Qlt09JaUpykOxH9mJNURv5OVB1AwOXU9Klkykd9L7/WvLKyRsxJIWKJJIhmeadP53r2FsVDzfy65ND5lf5oXij70nwy83f6zVxf6EKsMQv07JPrOWtsnBaJJThIzG4Tp2PUqGz+HhAg5kvYhJAqARU2IUA8x3NymDAc3zuNCSAv0ZJ89NQLqLFxAeQku1nnLZFHPiaAxwQwBogeE4deQI2NC2AJChetT4HZ4zEBblRADEAYm8oAFTRBHyw0HEK0cgZHPcCP0e9d3Dngkh3EobkCDppebjnfcL4khTbnswnmjfqpHtQPaTfyvBpXkBAo1Ap2wCEFTg4IS7khpop2WkGrvqtfU8yICYhq6j1ykDvKBYWUB0nREqa5r9VKLdTDBLf3gGnAvUMOHzo0oewbvYVA0+QL9U2tRcAUfUOAbkC/Q0YOHj2XvadclR07+YW6EgkredqMSyBqAAeut0cOSfBOXmUDEDmptaYV3MrJP9XM9zqv8SqvJjGeQwutbDdsvXnShxbqi+ykVZ9lRwt1N/QnMu+25XO5T4bl9fit0lW2+vKvmvjuUE0s4/6hmK5H7aaOrWaL7u1diOUmXA+SfJTiMDkuq3OZVtlISpnKSsVMb18a+ScQxH8S549A4ny5KrByr+VPo1+IGe8lKazEkEv1pIwc0rX8nhUjb3KSKO6i98a90L8B6J2IiAplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjg1MwplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxNyAwIG9iago8PCAvTGVuZ3RoMSAxMDc2MCAvTGVuZ3RoIDczMTQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic1XoLeBRVtu7ataqq393Vne48O0nnnfBK7BggEKRFCE8xQECCgmlIQkAggQAKgQmiJKIoUSAIAxgVFAIyARESjAhjRkBkZjyCx7nq9QGKngnIzImvEHbOquoOBOace+d+93zf/W5VdtV+77X+tfZaa3cFGAA46CGCZ+TwEbkwEEYAsD5U6x6Zd9+kpGd7n6LyKAAhfOSkycM2Ne1uBsBCam+/9+78UZZPBvwrDc6nPu77JqV75/xa9gSAVE7tU2bN95cLJaZYKqtzmGctXeyBOdHZAPI5KvOS8tnzF965dC6Ansqwd7a/ohx0dINhFZXNs+ctK4HfN79H5XUAYRNLi/1F+um/rwbwfEnt/UupwvKi7nmAuHAqJ5bOX/zoa5G2D6icQ+W6eWWz/LO2T8+lsjr/+Pn+R8vFDfJCgPhEKnsW+OcXp1we0kLl4UTPufKyisVftfygp6nU+arLFxWXD9b9TZ26nngoBRUrMwQugW4ED9XFQRLl1TYj9IMcEIbnjssH6zz/4gUQTrjS1dUFcCOn9mQPFy9aAHo1pyWRZlDfehBYvtqTzWXbwAAO+KevruHas1Z7Pqw9j6hvtSbQ9n9/dX323zNPjxk/6vqs66PumdX8f/8ahKuBZGMCO+EZDhHghlhNciYI1fDvvlSJ3rxEkLS3rD11JBtGs4TRU5UVBnvIWosBVOnraUZVPySwgBVsoNCKAb3YAfvAqenFcv8i/0yo9S+avwBqZy7yz4HaWf4FFfQsLV5Ez2WL5kHt7OIyys9eVPww1Jb6F1Cf0uKZVPOwf4Efauf5yzzqk/Trifn+xaVQu+BhtaZstn8+1C5asoB6Li5ZMJueper8/4UOasjMmzPbf4seihDQQwb9tbdEHDkJsXhIJT1HQjAUErW3ExIIiVBqESgfR08HjWDaU+X5S/gO8iDHDLrPNXBX0/KFBM75ILyFtwqpu8xKg/nJ/4RgrTTv0v99P5gW6Ku+b+R7zHFL/bQe5UU388JAgGvrKZ9zY2gqYaNoeIUG8WNUIwZbBU0XUsVUUK0GPSlHPUQrq1U1S8qUttCQmMAb/xVKBNrtgklG1IuCEJBDjyuvZEQR+Ehvl8lO7mRbdfPZhVv6YDC5QUUf4AkqMa0sAmFKNCpUI9MMWXAP5MJDMBvmQgUshWWabnggI1jvhzkwD5ao9V0Xus51nez6Q9frXfu69nY1dO3peu1WunpcAdv4ZLBk1OYMJJWuDEpZENDBe4JJ7ZMbTCZKDwWTOpOf0mxKFkpzKJGfID0DogyIapUfIBoBVPnHUFoWTB6WBU1whu4T0ADb2KtUKqH6hVRTLxyENTSqCd5lZ9haoS/VvQpX4SPqWQNnsEEENgYyqRbgL5IA7WSND9Ec2czJsnUybY/x4iFxotgkXhLPwgCxQjwrFooVLBNflqZIr1LKxj+QHE+ThWliXxCdR/F7zMQWcbhohS/wLDbAN7SKisEZWA87oZJocbIyqBIqhYlUc1I6C1vpLqP2s2wH+4ioO8oeh/PwAorCKNjBzhNfZ+AneBzzhSqCNlMoIfpP0lxnafxWqCCzdJ4ZgQu9qY6op7Vmas9o7Cud1+6rUEUr58NOuUl26hJoFRWxV9m7rE3eAPXwET6IC/FTtkZMEHeLo2B9AAEshPU091Z1jFzClhHv6l2pzi48IhayBvheLNTNpLn/oHJEax4SJhJHJdBC6RFZIZ4GszW4lihVW6PhrG6MmE7jaQbdSuIaoAyzSNZl1L4fDkJfrIP1NJPGrzxA+olGbhO/Ip7Xs2eEn+AsDoc0KBGvqDpB6lEHcEQnSyIKDPp4lEYhaXRRo2/CVM+pgri+fW4rehSdpxHyGi3LPE1dXXlTxSipoFFyN2KSvlFMSvjqv2r8qm+fsXlTPY3XRwwPzjqicDjVTZpKWbVE1VQ/YrjWpi7aKCXR3+jCRs+sUs9TylMJg55Sigf1DdgJ8vq0A5FyNV0XxPUkHRP5mQRfiFzvgHrzc4514Qa3LQbdrqhwpbOtve0OUC62tylXMli8YFccmV6HXRFSvGBXICFefQpPb9u+nf62b7/GDPzna9f4z8wg5fGz/ANKZ1km3XeyzHpewat5Da9gz7BlbDl7RrUnX9EWnUZW2gg+n2sY1otCvfSYDuoN+ljZjRDLTMq5sY22/KnN1Nk3sKCttZMISm/ztreda8sgDAri2SEb2kRh+oA4u5SVlGmPc8VxNoZvYcXvszGdOxvEilFNozrON9AEJC9xDHHshh2+lIjIKAx32yUR7JIkDlNesm+01DufE2nfgmIUmNEdpqAcrXSObXTlj20MzX9gbKMz/wGiBLuODyxoPdd2/LjdkR2kpl2jRqdIl3XSZdboVuxh2USbzztZnCJN0S0Xl0tLo2oidLSrI8RIEq97MSyVl0RWRC12r4bqiNWRq6NWu3fD7ij7dJieRExk9YcBd7GsO5MT4mVd1l0s0yu6nLJOBjIlJzrHEYyZ/ntfq37oo0eXn5v6HXOOeCCCtzc0NDzCnhs0f/PoR+qG3fPBHd7vfv/grvJo/lfifhvJu4K4T4VyXz9whRirDbHVnpB6l6XesEF213s2JDwnr3O9khbqDgF0RriTPYobnbEGOU0FITS/m3+Dxj8B0N5GXBICStvF9ottyrdXFO0mVDKYz1AU44/1e4riRJjOYpjLKcbFJ6dkxRAj/Ymr3iwrkLmFPRz63Cv8z/y7GSfn5p+af+xk8679hzfteOWFSccWVZwu+JaZn8Wk2Nbaz/+elPTuHd669U9sevWR8orKxORDHs+HB1fsVTW8iKS8k3RKIGv+mC+aWdACiJZhgCZdvcTwMQMzG8Et60WzVflsbKOJGLNojJlVxs7ltLZ57apcL57LafMSL5pgxdMk3NOqSHuZoBeMggJyFI/AU6ALZb0hmfXG/mw8u898n2UKK2FL2HJcwywkSgOLw0x7pivBnmCPy0KZC4xn8fPnT1+fISV1XsCznZm7eT0rfJcktIMkVESUR8MMX4IYqbNXK9GR9TpnvbLWItTDY5Z1up0xYW5mRDcYFTlG6WQ95aKo5Ad3i6LuFhKR0npF3cDqDibx8NaAdEJIv+wq5uCiaKqnWFRpfI4R1+v7TO3TwRL5Of7DjHdLpx1/+PX33399wkv50vkG/rzNxq/829/4jx7PmTsyDm/bdjgxmaiv6LogpZB+RUB/X6TlJet+4yY7ewn2i5vCnrOvi9RFWCDDqUSSUfEGjYpqVn66knHIFhUbJbDpzNWPJXhUkxLn7T/AZb1RCJVSSi6t7gJ+lSkMVl8qmXv5Cf46X86q2aTqy9LM8w/N4Cf5J/wv/OSMhz4aNYq9yGazUvbiSKLqNGlDAmFqgDSfQ94kCpvgMfF1vcR0BKJoVKk516puYs3GHbSZGJuuwaOmhNM47Xq5kHe98X2V8VEN1weQ1epq49Gik+8jK2p7E/ZQjC4qrTQHWSRG+1Z0XvuE71u/XtXFSpJoX/JZRopnW8jmxJrCDFbYEyY3W+2e6tij7uaEJvu6MDOEYbjFoDfFot45IpmI+uBcm5e0UFXD1ovtnUTce+rOyrZnk/h8CzKiM2IyYjM8GXEZ8UNTfNG+GF+sz+OL88XnRefF5MXmefLi8uLzUspT1kTXxNTE1nhq4tbE16bUp1xNieke2j2oe0BhTGFsoacwrjymPLbcUx63KmZV7CrPqrjw6SSbeNnlDCUdGcIG2BOySDbxyVl39s+M67mDQ4VjX+x7rGxLc1PT0JYn9525fo0Jr20uPJxffGzav18VMksqZ1b85VDauOuPNZT4T7z89nFH1dP9+jWkpHSqWB0lrHbKTvJHbhjoi8Bms83QHO5aZ2uK2hwBDsfIcLOsj8yNVkXmbdfcwMX2VhWYjMOFMati6mOQ6NTICZDKNA0nb0W0ptAGzMRvXnv++dfUdP3ZQQcqP6DTyQeVBwY1NwvpZy5dOkNJmFjk5y38F7pb/EW7iRoGC7su4CWSYQQM9UVBNXtStFZbnjQ228XmsCZVsR0WGOUcQYp9sVuxFd5+RfnxSobPZItSolZF1UbVR0lBBe+mTlPw+KCC46Xx2/PeeO+9N/K2j7931/Tr/GPWl8mTXxaz9vXufeHs2Qu9ezckJhJDVuZgg9SzEFElTiP6lABakc1gdTZL+nXWJrYZw0TQCyPtDtOIaM2He7030Gq9BS17ZlCY5NqBBMg00gIyxZebmgYdWHGmC7rOrDhw/SThtns3YYeHhRm/tu0u8rPhTE/3cD93BeEL0lVFaDkhivxMIriYoVr/pOTaw6RmM3srvNnRZF7njnIJepcexgoO2wi3RmKr5ktV8C5qXqRdM1S+tKHR5dH10X+OvhotDYWhbKgw1DU0SuqjS9enG/oYy6CMlQllrrIow/SFKsBxmiG7aTxIAXQa6DqxqvOg+eyRuSdnzvrzw7ydn2RpnV8zXZOw68mtzVZhxrRjJ++8c3+vPmwgM7IQdg//vHXzof071CglndTzF8I6BAp8bklhZv0emdXAZqvcYhRCdKAzSHqLzTTOqfpJo2p+Tar5Hdto1fKqWc5p7cxpbXVoG/qit5NsDXkWcpOHfa48V70LiXQiMpppVodkkqluLuGXxln3snT+YXNj4/63ZeeWvNJZ6zvT8cP149/aq2LNp4jTCGsT+fQxvoQIc7TBUR0S2mzD5uSEppQWQ7Pt7cjo5AjQm0fKDodnBDny1m51aL0YUAh+XkU6m7Si16pe9b1u20NhinDTPwxhQVWhKDA0LIsOIrs2bdy1a+OmXU2cd/j3TZiwY+Kbh7IPrvhjZ+cfVxzMbhKGnPrss1MnP/vsr/xr/n10zBt9er39zgOzZrJBDJnIBs2c1aDu/BMEMp00yZ7qoLfPKh8TD0CLIDG9CLl6pZO8sIpaZxvtJ8XgM+QZCg3lBtpPIaS9qms90USXWHitXnZ+T/N1fcqnaPOZwAbDfW6ToAPrMbOuRnobWswHFL0iyfdZmN4MuYo2+8VsRzCKaVMCYNBCdp89z15oL7cHFnLKwYglsOArb+beMWectuq6j49v82+RU78nTbnJSfwR2CwwPeSKSiBkzfBZFMkn5UmFUrl0VZID5BPpsvPXNlXLjtKpJJqkGQ/TfMmywxBuAzla5zLXRHuwKaolQtGB3abXy3l2vS3PHU6mMEE1hZ2dnW2BODQn52K7Frw4woiDkIzEvMTyxNrEerrfSfwisSvRQLLV7LSLcAsIuWcm06U1imkjjq/+3bHmRUvWv9q86JFnXm1uHtq4bPleXLti6Y9fX39Q2PHStmM7r9cIO17+7TuvXK8RC/fPnrkiyIFYRByEUAQQgQZAK5NrrPYmc4uRCXoYr3q4XKdKtGa+c1T904g9VOj6k0uNAP4zcoqaVqzYtK+5edgbS068J+xUCXhxh0oALVxc9EPQ5izR9kEY7YMQudkBzeYm9STjsE1Ah2vEbScZX8LQiEqolKt0VfoqQ5WxylRprrJUWatsVUqVvdJRH3E1wt7D55E9vOXAU7Fx395NG/bt23CVOfiVq3/jPzA7fnHp9OlL3506+f02foq38ctkYLLJjjjZQNW30U7dSRSq1vouX1S3tW6yrmNvY0s0WeqRms3u4d0oaOs22D5DwGJ/GSOy6Uk3oAm6tltcXkVz803PJgzs9ne7r++XjQ09fBv7a7fJvsWOaNR1e94m27qotyNaojW/O5I8cA9v0k3de7dR19OBsB6OhayFPYGld/sQoeKmZxnU1HTD/17f38OtFDX8+lNAq3AMUWeHDJ9TNtEuMGGNtcnQojPKetDnOlSjplkI8iLnPlDdxqG8kBdDVH0K+NubyhSGY2JH99n2GqF0dE1IPzcectjPHLt+kFSpZJYk0Wpl5O1P0mopcMmXYzELVtOk2Bi9QdAZJ8XGxgwzmmJiRRdFAWtFZ7VrbbgaBSRRFJAaYzTFRulgYpTeqtM740ekqlSda7uompPs7O6w4Ec1LFA1XjtTWC/T+UmnPQviD0IKhZ6++W6j2+Q29yPn1sfUxzzYMNg42DTYbPKAhyUKqcZUU6+QdGe6q1doakxqbJonLS4xpdpYbao2V1vUX/CZIMhG2YRmtKAVbahgBEZiFLrFaENKetrQtIfSqtJWpdWm1addTQunA8rCm1FJrHZOkxN6HgjSCUPVF1GM8vT43dPWrp25cWjrrp8/mfbuvJL3/KvXFe/17X3hyz+WHBKH7k9Nzc/3jY6z9tqydtvhhIRjWVkFE8bmJdkSN63esS9G1bT9tFOnahbCCYN97ps2Yp2RtTibzGQhnKbxZCtyXaqqZQfketF7w1CUuY6rhiKEApeApt2IYJLZftVQvN7UdM+BJSdOsT+xo8Kr1/0vvnhsp1B5rX5fyayruFu1UkPISlWJhSDDNV8K2kVJFOxMkNQXCjLIzA4gDxMQ3pFkCQUmiaBTf4XQHDsEHLszX/1VQD10gXYYDgv8DpB940cARR9M0mX1zPjsKGGuUClUCdXCKuE5YaegVxcyoIF0ycUiMVJMpjNkGqaJHn0WZLFBOEjM0OdCLhuNo8VcaZTs00+BKawAC8Q8fQmUsDk4R5wtlcqF+iWwmFVipbhEWi6vgTVsLa4V10rVch3Usc3CVnxBfEHaLO+WXpMb9cf1X+i79HeR2EMyDSyTJQx5l81gM97lD3aIhZ35uO9avYoQ2QMVIRt72nePTi8Y7GAz2k1GAJvVbgObxW62gPqyWowmo9luMhmHWUwGBUxSDb5tNbUoVovZaJAR9DbRZlIC6I1t1GuImboh7I6NWlvtYdoOIbsXPOH8JzBqb+lymFfF86oMkl42oCXUGGZRLAmWLMto433G8ZZphmnGucYayyrLBovDCESESTKbrCZbGHMJiqhIYUanyWmOtEbaUiCRdpRH9Ehp+lRDkjHRlGhOsfSy9rJ57ANIBllChpghDTT2N/U3D7RkW7NtGfa7wcd8gg99ok/yyT6dTz/MMMI40jLaOtrms+fDBDZBmIx5Yp40RZ6sm6K/33C/cbJpsrnAWmDLs5ewEqHUOMc6x1Zor9Q/an3UthaeMqwxrTGvtay1rrVtMWwybTJvtW617TTtNO+17rU12v9k/8LeZS8miUlWFghBhjKmCk/YMH7jig3zxuVnxvHBATGWnlq+dVR1vji+cyPOC0auUgPFI4kwyheSrAWq5rhwS4zebo5TnOOSKPxp9aqhqZKjxqd0iPXZDRb7HocQWQPhm+VYR4vJlp7zrdfLc654KWj1ZtwSqN4MVger9Tq1QbUWUkN35Mp1WvC6v3FWSjL79ZYotjuS3ZKaWjorENGmq7/QEb0uOm1N9NmjciFMH2pzino9hhrlcZE36eU55CN9Dj3uAaXGGn4s9IB1swFaJKZSe4VrP0R5KY7roqNXLR3BFO0A9o8kE8WMIjpxTIDS199oVin/tblZjSe7aTzyO5VodvD7IKZBGgf4bGG5FKUazXq9Ijqs40JV+gLkqdTRmWCPQaSDgt3QYhFUwrhGFdOguz3cF8YI0MyP3Iz41cBQqrwt5qfV5U5aPQ36UpwYnp4b1lvfS4ly6SN7GSBW1ifGGOKTx/W7CVSrV312anCFRcUm7Em009ml77FeBxTYHKpLbImIjlNDR69X3X9Km5f+AlIOSnNA/wE3oOqWeY9DikToqQcVVbz3u1PGP06ivlc4qMIZlD0SmiT2gJQnu0NSVDC7we1mTejGVuMuGcb7QlNzLXolNNypVwzqz3VxUYbYhHEpPTjTGNPUINzt2RNnF2rMyZtdurgWW2RMgKX2nH/kp3/mbfDfduIK6GlPWQT5uMHD3p5yuSGbwPcxoWBLS+W37Q/Zcn6EWL32cerD31gvdr9//rhznLXAoH531t/4lkXjdPN5NICV//xxxwRrwT989UoSz2rflUBQD0xP064OgxpKX1Gqo7SNUhGlHVIdVIhb4bQIXW3iJaiUnHBULIGF9F4otsFC4UNIV/NCNpwQsrs+Vd+6M3BUclC/b7R+R9V2HEPl3lBG9fvFFhiiJulxGmsMJF0LySr9FhqHwTL4hc1jp4VUoULYLhzEUbgRvxB7i5vEr6QR0nypQTolp8nF8mGdpMvXPa0fpF9hsBj+bhxj3Gb81DTGtCeIYTLmQ28oBTPpuwJbVKRElxBKb/Vblg6mqV8bRQN1ztC+96l5BqFUCuQF0LPcYB571Is98hKEs/HBvAxOVgL3QBmUEw+LYA7MptUXg4fO1bNop3nACxl0Z1JuJvXwEK9zqL2C0iIoBj/Mhz5UOxoWUP9+lLsb5tHtgYk35qrQSsX0LqYxS+lZRD2N/8Sq/W+smk8rLaW11C9WC6i3SoefxvyfrTiccnNp3BRYQj1mUV+/NluxNsKvceShWRbQs5z6zKR551A/D40vo9X9Wtvt80zSZqkgisrofphq1VUrqG+ZNpOX1s6ErFtGdY8RAgrU9Rvt+/c/XsmaXqj/MaHGjS4IpXNeONWm0WrDYQTkwkgYReiPgXFwL9wHeTCBOJ9EeE2mdaZCATzIBDgOJ9RfHXRLFszJHZaREXxnBt93AjQJq3xd1zh2OPHXJPzFiz/X4U9W/JFjO8d/T8K/W/FvdXg1CX946m7pB45X6vByHbZ14F878N84fj8IvxuGlzh+68VvLk6SvqnDi9Tx4iS88HW6dKEDv07Hrzh+yfELL/5PJ35eh59x/NSB/2Ml/uUt/ITjx9T945V4/txI6fxKPDcSP/qXKOkjjv8ShR9y/DPHP3H8I8ezdfjBmRjpA45nYvB9L57m+N4au/SeG/8Qiq0c3+X4e44nOB7n+A7HYxzf5tjC8S2OR+3YXJ0kNXNsOvKW1MTxyOHp0pG38Mgq8fCbSdLh6b4uPOwT30zCQxzfqMODHA9wbOT4O477i/B1K+7bmyTtK8K9DQ5pbxI2OHAPEb2nA3dzfI3jqxx3OXAnx1detkqvePFlK75UhPXUpb4OX+S4Y7tZ2sFxuxm3/TZC2laEv92qSL+NwK0KbjHiCxw311mkzRzrLLiJBm2qw40brNLGVNxgxec78Lnat6TnONauny7VvoW1q8T1zyZJ66fjep/4bBI+w3Hd0/2kdRyf7odPEZtP3Y1rnzRJa534JB0ZqaKmCKsJqeokXGPHJzg+vtouPc5xtR0f47iKYxVHX9dvVq6UfsNx5UpcUYSV+S6pMgmXc1zG8VErPmLGpUZcwnFxB1Z04KIOXNiB5RzLOC7gOC8OH+Y41z5MmjsJ53AsXYmzqVDCsZhjEcdZHGdy9A/Cwg6cYcbpHB/gOI1jwVSjVNCBU414f2iEdL8Xp3CcTCtPHob5LpzEFGlSOE504oQxIdIEjnkmvI/j+HsVaTzHexUcx3EstYzlOGa0Io0JwdHRFmm0gqMsOJJjbh2OqMPhHO8R+kr3dOCwt/DusejjOJTjXUMc0l1OHJJjk4Y4MGewRcrxddlwsAUHcczmOHCAUxrYgQP6K9IAJ/bPMkn9Fcwy4Z0xmGlB7x0mycvxDhNmpJukDAumm7BfX4PUT8G+Buzjxd69kqTeRdgrzSH1SsI0B6amJEmpd2NKEiYnmaRkGyaZMJFjAsd4G8YRn3EO9BRhbAfGEAsxRRhtQTch6OYY1YGRwzCCChEcw4swjJAK4xhKg0Ij0MXRyTGEo4M6ODjaiVf7MFRWoq0IrRwt5lDJwtFMvc2haOJoVNDAUU/d9Bx1TpSLUKRGkTTAhVSLnI5+iiT0RaYgcGRNrGjNM6z3/w8X/L8m4H95Rf8H1V2r7AplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9MZW5ndGggNzcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicpY0JCoAwEAMHqvU+auttq/9/pUsfoIiBBJLsEvgJ9dInpGgycoqHqzJq9Xm9Fja0dPQYBqx4xyg6MbOwsrFz4CUJnPHnugFNEQIOCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAzMzQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicXVLLboMwELzzFXtMDxGBJEaVEFKVXjj0odKeohyIvURIxViGHPj72h4gUpFgNLM7u2vW8al8LXU7Uvxpe1nxSE2rleWhv1vJdOVbq6MkJdXKcWbhK7vaRLEzV9Mwclfqpo/ynOIvFxxGO9HmRfVXfoqIKP6wim2rb7T5OVWQqrsxv9yxHmkXFQUpbly5t9q81x1THMzbUrl4O05bZ3tkfE+GKQ08wUiyVzyYWrKt9Y2jfOeegvLGPUXEWv2LJzvYrs2an/p8wBl48fIe8v4IeaEJIAXsAYclNTgPUD2cgZAFZDHLAvKxCbKHMzDIAo1EBnmh6CvQQaCvOALE4kCBZ9B6LrBSRGWg2TzjTIUCMACTZTh5hgmydDGGOhl6Z/OhHhRRNPVwBl78Tpaf79fj79K6e3m31q09XLiwb7/pVvN6J01vvMu/f4I9vFkKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvQ0lERm9udFR5cGUyIC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMKL0NJRFN5c3RlbUluZm8gPDwgL1JlZ2lzdHJ5IChBZG9iZSkgL09yZGVyaW5nIChJZGVudGl0eSkgL1N1cHBsZW1lbnQgMCA+PgovRm9udERlc2NyaXB0b3IgMTMgMCBSIC9XIDE4IDAgUiAvQ0lEVG9HSURNYXAgMTYgMCBSID4+CmVuZG9iagoxNSAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvVHlwZTAgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucwovRW5jb2RpbmcgL0lkZW50aXR5LUggL0Rlc2NlbmRhbnRGb250cyBbIDE0IDAgUiBdIC9Ub1VuaWNvZGUgMTkgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9Gb250RmlsZTIgMTcgMCBSIC9NYXhXaWR0aCA5NzQgPj4KZW5kb2JqCjE4IDAgb2JqClsgMzIgWyAzMTggXSA0OCBbIDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IF0gNjcgWyA2OTggXSA3MCBbIDU3NSBdIDk1ClsgNTAwIF0gOTcgWyA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgXSAxMDUgWyAyNzggMjc4IF0gMTA4ClsgMjc4IDk3NCA2MzQgNjEyIDYzNSA2MzUgNDExIDUyMSBdIDExNyBbIDYzNCA1OTIgXSAxMjEgWyA1OTIgXSBdCmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNSAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDAgL2NhIDEgPj4KL0EyIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgPj4KZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjIwIDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjkuMCkKL0NyZWF0aW9uRGF0ZSAoRDoyMDI1MDUzMDE2NTc0NSswOCcwMCcpID4+CmVuZG9iagp4cmVmCjAgMjEKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTAyNjEgMDAwMDAgbiAKMDAwMDAxMDA2NyAwMDAwMCBuIAowMDAwMDEwMDk5IDAwMDAwIG4gCjAwMDAwMTAxOTggMDAwMDAgbiAKMDAwMDAxMDIxOSAwMDAwMCBuIAowMDAwMDEwMjQwIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM0MiAwMDAwMCBuIAowMDAwMDAxMjkwIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMTI3MCAwMDAwMCBuIAowMDAwMDA5NjI5IDAwMDAwIG4gCjAwMDAwMDkyNjkgMDAwMDAgbiAKMDAwMDAwOTQ4MiAwMDAwMCBuIAowMDAwMDA4NzEzIDAwMDAwIG4gCjAwMDAwMDEzMTAgMDAwMDAgbiAKMDAwMDAwOTg1MyAwMDAwMCBuIAowMDAwMDA4ODYyIDAwMDAwIG4gCjAwMDAwMTAzMjEgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSAyMSAvUm9vdCAxIDAgUiAvSW5mbyAyMCAwIFIgPj4Kc3RhcnR4cmVmCjEwNDc4CiUlRU9GCg==", "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2025-05-30T16:57:45.794841\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.0, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "df[\"label_name\"].value_counts(ascending=True).plot.barh()\n", "plt.title(\"Frequency of Classes\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在这种情况下，我们可以看到数据集严重不平衡；`joy` 和 `sadness`类别经常出现，而`love` 和 `surprise` 的数量大约是`joy`和`sadness`的1/5~1/10。 处理不平衡数据的几种方法，包括：\n", "\n", "* 随机oversample少数类别。\n", "* 随机undersample多数类别。\n", "* 从代表少数类别的样本中收集更多数据。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为了简化这一章节，我们将使用原始的、不平衡的类别频率。如果你想了解更多关于这些采样技术的知识，我们建议您查看[Imbalanced-learn library](https://imbalanced-learn.org/stable/)。只是确保在创建训练/测试拆分之前不要应用采样方法，否则它们之间会产生很多泄漏！\n", "\n", "现在我们已经看过类别，让我们来看看这些 tweets 本身。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 我们的推文有多长？" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transformer 模型具有**最大输入序列长度，称为“最大上下文大小”**。对于使用DistilBERT的应用程序，最大上下文大小为512个标记（token），相当于几段文本。正如我们将在下一节中看到的，一个标记是文本的一个原子部分；目前，我们将一个标记视为一个单词。我们可以通过查看每种情感的推文长度的单词分布来粗略估计推文长度：" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.106630Z", "iopub.status.busy": "2025-05-30T08:57:46.106026Z", "iopub.status.idle": "2025-05-30T08:57:46.439152Z", "shell.execute_reply": "2025-05-30T08:57:46.438513Z", "shell.execute_reply.started": "2025-05-30T08:57:46.106607Z" } }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzgyLjI4MTI1IDI2NS42NjU2MjUgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicvVxLb9w2EN6zfoWO8cE035SOSdMGCFCgbgzkUOQQJBs3wTpp4jZB/32H2uVDIqnhaolm4dY7Hs03M5wHKVK6eb7//vHd/vcXz/qfXnU34du7x471n+Dnvqf9J/j50bP+BfzcdxS+PXRi4IQPjCv4doi+ca2I1krDrwdgnX39s+s+dDdPQcgjXPWi6/hI5PEqRQb4H4g1itCYdIhJkp1o4UJPOgnnR+H3oCqoTQZQHKAspVOD09GhBQonYpLSPQObf3Rf4b+0v6YgRYIQToXlGonp3z10z+66m19Yz3h/92Hyxt377o/+ye7t7vPufrfffbvq3/R3L7uf77rbblKhYwwg59gRqQzOqCDjUUUE/AMAv81CG7F0ckRagdaSqJFqHPrT7svu3xSYCzYfyoeYVAbmbCByVDUePwDy990+gz0MRC+wA2kFG4wWTNZY/Qjufg8jvoffHlMNhNTLaItIZQ0E/I0xwwZTocE/u2+7v+DnI/ye84LPkpEoCsYP1g+KjAtidvglGY5+4JKMUgo22M+aPnRFA00JGwYtdKxBIOY0MIT3CqoH10xo+1nDZqvoI3w3VCoVowdiCX3QRFAzKqks1wo6X0VnnBGlODciho+oJXzGNDCPxnBj2VYUEOsKKKjOQtKRzRQI1KIC0hBtGJSB0bKtKCDXFYD2ME5iZgoEalEBMxChrJL2s6aAWlWAMwHFzDpylgGBWlKA05FQYYfJftYU0HMFvvbL5iYkBDNg6v7bvn/df+55/7Kft9xOilNPiPPF8FPhivPFExloyKcgBaq/Pk+dJdyr7rbfpKXvmrGSnijgelsteBuMyJAIZB4621FcOwpKP3QwY5EtLXEYsyTwIJssuXkq3DTHTmyOMWtDNnIQxK0LW1s6+ptfaf/8i+ekvSBT9bcfvo3h+nIRMAly2T3ldwbETgccB8uyQIRQqY6fLAPdaB+unaDEyZiaY85Jgpi8WtBZldM7z3GNs2w0/lqTU1sVWm4dGhQbZcBFMOhSxn0KTmJEGrMW63gM4bGc9+Pm0MLNwpMHlYH7Hx9l3O7brbWxY5RlJogMsmrZYQ4RlVGYSDNmA/cQiSiQ2zS9sFybaeqpbZpFkBcZE8PYpZtxNl6AQ91iKO58jIukH7ZBidSOYTZZU2x+sZdGWL2YWWYwLDMWDGl9ZMvUSYsKw8oAQzN4yYGC5HrEUkia5OcbgwvBrcmU+7M5UKfiIvCxRUFQDNxSDosrr0eWhW5UrsVA4NpVGIA7AR9NlCMzB0uiG08AXJGSkAvanxpPlVALwtXpDg0bjGs1nnqIqLO1XRBRIMcyLijl/pbhTFNPbdQwvLzZsi8iG02Ms/ECHK2T1ar1O02aYhuUSO0YZpM15fYX3dVVx1tbUUfgeNPgaL0/X0haaZcyMo2Ho5VoKSRN3iVHWokSRdKiea4xuMsqnJopZxyrqjzbN853df7CPBUfWnRUKkYWH7fM6m/pdHRkcZgKVVvEaQtVURDcpy3GFq0NFfHYIOMqIgj3KJqSFVlbUQpRY/BYryhjuL0tZKDmohGULzsVWfY/dS686aAceNRUgGjiRRSKCppnFUW0Ip9b9FhcBu4RdOgaFEC8/uGp2mT6dPmspUVXqqgYeBQ2yduSkO2rNPir24DT0779Q8el2785kQ4RiYEzhDxtK4aLC2QvYPtCIxyrCAp6khha3DAM8iIzYphwy+0yHO72KoPeFsbvLraxxqNEascwm6wprswiLw1271nOkkBg6SrQMrvgoFkqehmOnMlygWZfIiStA1X656lp7VrioRwCxnpYNzxTzBOzcMtRTSpkoIOUdxMuGR/bipEreuGC6mtk5riBoG4PId7eCVR7cI65WzhBRIHcZosonDCbnXYa0q2WNiiRMTEM04Yoc2FHGVhyYw8wxra7XQHF6RxjnG9HuQAH54B45rP9GKASj2GJldKljDRNlxyoDJqlZtJ0wZGpVUtsnCUPnqlgZ3OgdmeqTAKSLnmWTkhXiec7oUITPFbwgc4sAs/3Ox4XeEjiHmmiKj42W0O/hZtwGagj8bjBC0hmoXOuojXhi7oD56hJpAZZUpFHlxenfOBJTShf90F0yOt8H+dRNSMuzEolTHFyOpxbHOHLvVZRSvGyUBGLqKZ4SqwMQ8HHWCpvrUS5ufLZtbRJ7lW0qLyRDdIJdwI6ABWDjmdBkylRRR7UxAVecVGn1Qx6C01qnNIi/iraQ0VnR4VU6IrPkVAOVFMco0WMFCbweBA3mfTWhB8+XnjC4bMGHAavYi2M2Tr3qJDcJMG2domKWG0w+WtSzlrUkIqe1yJaGtxgKE0vtt+JE4K5c2lum0FoutzGCKTZgbRwcYHcYB8kPGAZFPSkNjsHQd7shFpEbnNAW4gx2b8BGOXP27WxxqPMDmgHmLYHtGMvpQe0VbbcVFEzWamw3K8SnGbZgiHToxLd0ptV5wvJTIOrDEgLiULL1RK74ITKQjKPOP8o4sDsewHG6WE4/yxjIJ4RbYtHDtwt5RggPDrSAMGfrB0kMe6p2nA6N1C3IoS9YSuUugeH/f5yoG5G8DsfZgjPxYbdk0DdihDqeuyl0BtyXoLyRP1LGxYKJy+KyLwBAuQkr494yL4+AjirXj0x4ztdXZRIJ/2PL55g02sn7ucP1Avw+Xh6pN5M10WPE0sioweKpyeKX1/1yhBFBz1QpZQG0hf7hgdmiNDs9A+I73ePu3732xWMCTHc/eue2FdRwB/urqZ3OYzCCjnJ+QF/2+/+vurfdO6B5dvuP8wL0lkKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyMDgzCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGgxIDExNTAwIC9MZW5ndGggNzg5NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJzVegl4FFW28L116lZXVW/Vne6EJJ2kszRJ2BITAgRQ2ghhFYMEJDg4CSRhEUggIEtkgjokoDigQBBEyCggmxgYBhIICGNGQEDHAeY9n/ocFbcxIuPDUUO4eedWJyzOm/+b+b7/ff/3V/Wpuvs9+z2nuwklhLjxIRP/0MFDcklfcg8htAe2+obm3Tc2+qMeTqwPI0RaPnTsuJx19TsOEAJ/wP6r996dP0wblPlLnNyM9b/dNzYtY/qPZVhnTVgfP2VWUbn0g30/IYoH1+g95ZF5fjI9JpsQC65HeWn51Flzej8ygxAN62T31KKKcmLBm2hXsW6bOnNR6WvhGysI0bEa+da0kqJiddLvqglJFPv3mYYN9i2WZwhJUrGeNG3WvIUvfueYg0WB/7Mzy6YUlSTNxMlJx7E+albRwnL5WQX7Azas+2cXzSpJ/vpOxDWA4+mF8rKKeZfjPh5LSDJ2k1+Wzy0pH2D5KxaTBU3TiOCVmCkuCW8gcdjmJ0lYFn066UUGEmlw7qh84phZNG826YJ8xau9nZAbJTGSPlwydzYRWJs17JPMt0okekqMpB9LBrEST8dupH0rwiGzdB7h/fZDCO+LErlxiTJ/BWGrWTshnnwrP2+2nRZ9/MSNsUsQlneuJ8rtoVmHOlf6v3GF1glh+r914ernb9np/P/mXv/CRVF6VtQWOzGIi4STCBJJYkisqTE2lCs1JY6mFNIL82JEwaelo6YSDTVKrOM1R4uRYoTF7BHaZjW10U4cxGnu4iZhHXq4mewhHlMPFxfNLZpMVhfNnTWbrJ48t2g6WT2laHYFPqeVzMXnorkzyeqpJWVYnjq35GGyelrRbBwzrWQytjxcNLuIrJ5ZVOYXT9TnX84qmjeNrJ79sGgpm1o0i6yeO382jpxXOnsqPqeJ9f+BzptcmTl9atFtei+TkN5T0sd8M5M7PpJAUtCuAKnykkTzHYZtEtbi8RmGVidhG8Xb1UHzn8kXJI8MtBHLB2Ir6XHcvhCZejG0tSjfenXW6bSO8rh/QqgOXPeRf0L6E0NjxftG+ZY1bmufeEt97s2y1I+Qa6uwPPDG1BSkUg8tEnrKDroaOUZYJtuA1djQG/6NlEpuXMGqAKiyJMk3ZnRceaVDikkQebhI8XAP3WiZRT+5bQx0gI+EnO9arFGzLpOF+I5CfQPs8aPU8kg+eYAUkalkBplJyskCssiUvZ+k3+grJtOxbzaZK/raP2m/0H6kvaH9YPsr7bvbd7XvbN/R/nL79ttx/MkV8rt1t9T8HSDwTEfoQ0L6ltcBVoT8DhDjH+gAO0JRBwj+FyNMRcBzDrEkSAVB/SKIbwi8CLMRyhHCEYSMFiBEIizqgDiaRRrIGbxPkF1kE92OtVJsn4MtddJ+sozMx5bX6Rm6QuqJbdvJFXIeR9aQM7BLJnQEycRWQt5lErlK88kBXCObemi2RUETGS0fkO+XG+TP5XOkr1whn5ML5QqaCS+y8Ww7Qjb8HuV9Gn1LA/2QVJDD8CVkQpM8WHaQD+Ec7CKf4i6CN2fIKrKVVCIuHlpGqqRK6X5sOcnOkY14l2H/ObqZnkfsDtMnyEXyHMjSMLKZXkS6zpC/kScgX6pClmdKpYj/SVzrHM7fSCrQNV2kOuFSd2xD7HGvyeYzBnqyi+Z9hVThzvlkq9KgeCyJuIvg2Hb6Om1R1qBkz8PPYA68R5fJifIOeRhZFeIAFJJVuPZGMUcppYuQdnFXitWlBXIh3UW+lAstk3Ht3wuKcM8D0v1IUSlpQligGEjTALoMViCmojeGnLOMkNNwPq5gWYJUE1IGWSj3MuzfS/aTnlBLVuFKJr1KX/Y3nLlJ/ghpXkWflv5GzsFgkkpK5cvIa3FE1xJyyKIwGSRKeviNeikwvLg+OGaC/1RBfM8eP6n6DYu/nuTV2xf5G9rb8ybI0aygnvnqIaDWy4HEj/5R50c9e4zMm+Cvvz5kcMeqQwoHY9vYCVgUNWzG9iGDzT6xaT0L4Gd4Yb1/yjT/k8aTif2fNEr69xT2I4lIA60UsFTT/om8CqVjxTMqMRim1LlJne0Z98oums8ZCz5vdBejreVqyx3EuHS1xbicThMkl+HOzHC7DCk5g7gMkpggntJTm154AT8vvHCNavz7a9f491RjefwcP4twjmbi3Ztm1vEKXs1reAV9mi6ii+nTwu98hKY7ET21ToJBbw7UyVIde8xC6jQ1TvFhmEWtxoWR9c78CY04ONivoKW5DRFKa8m42nKhJR15UJBADzjBKUuT+sa7WFYg0xXvjed0BN9AS96kI9q27pIrhjUMa724CxdAeckjkGIf2RxMjoyKhi4+F5OJizE5x/i1a629zvOMjHZLDF2iui/CACXGaBtZ780fWR+e/+DIek/+g4gJtB/vV9B8oeX4cZc7uwObqyY2FoN9bWFf03qf4YrIRtyCGePk8Wy8ZbG8mD0SXRNpQauOlKNQvL555BFlflRF9Dzf46Q68vGox6Mf9+0gO6Jdk8ikABKR1Yf0vYtm9e6amKBYsu6imRmy16NYFIKu5ETbKGRjZtG9L1f//PzCxRcmfEE9Qx6M5Fd37dq1gD7Tf9b64Qtqc+45e0fGF7/72bbyGP4VUr8J5V2B1KeQ8mAv4g3Tq7W4an9Ynddep61RfHX+NYnPKCu9L6WG+8IIeCJ9Xf2GDzxxmpIqmBCe30m/ZtKPDLjaglQiB4yWS1cvtRifXTbMG7mSToNacWxRXJG/OF4mk2gs9Xrk+ISuyVmxSEgfpKo7zQoVbiMPBj3zEv8D/+KhkzPyT806drJx296D6za/9NzYY3MrThd8Rm2/gkBc8+oPvg0EXr8jo3bVL9dtX1BeUZnU9YDf/87+R3cLDUe/Lm9FnZLQ4z8WjKF2sBMAew4Bq6WOUXhMozad+BRVtjmM90fWW5Ewu0mYTRB2YWBzS4ZLyPXShYEtGUiLKVj5NAr3tBBpNyvpRoaRAjw0FpAniSWcdiddaXfoQ0fT+2z32cfTUjqfLoZl1I6i1Gg8ZLoyvYmuRFd8FihcojyLX7x4+vpDLND2CZxry9zB62jh6yihzSihYsQ8hjwUTJSjLK5qIyaqzuKpM1bYpTrymH2lZWtshI/q4CO6ocQabfRWuRgC/Q5rMYS1oIiM5svCgIUFo3h4c0g6YahfLsFz4vWQ28QipPEBRF6v6zGhRytN4hf4Nw+9Pm3i8YdfefPNV8b8Op9d3MWfdTr55b/8lX/n95+5I/3gpk0Hk7oit1ch9rWmP0kiE4JJYQqxV9tIXbhS5wvfZtTZViQ841sZsCVovsjYMB/Ex0UH0MGgEl0yXcyltks31SfowTOVnpPOwTn5DDujIN37Y6VJdBJNULye8BCu1NuLJiZI0ElIol+4o/iMcGnr8i1bliNQbdTzo06ddw7Y//BHlPErH/Pr/DLNo9GjnocBh1/89ZEjv37xsLSoIakr/5Z/88Ak/s1Xn/G/mA5qMt0WK7QJ8yf5aZSJirFB/2AXus4g67TH3IauYnjFIu2DXMSnyR6kJKNNmAPqjembglanN847yPtz76tehni7OvgbiBeKLnenrkS6hj+9cePTvB89dY1S3n6Nv8nSrr/9bE31s9s/ee+Dj6/vILS9Fff/Eve3kBFBhyKtI4/JNIjyDzLVuIA8M/fLSEdF0PMnHEU0g2Y4raIuuPr2KwiGES2OGNSQ4iyGFtTKtS2aNgmE/FEfFfmb65fPXL+MUm29yLoLjzwYz8btuJ+NVgaHsyiFabomR+kaROlWXYqiktWqKy6LamEumamqRXKBZMPRLgy2cnQmgQLkVatqs+qaGjoarRZiNy6cNVmDBpURkX3DVRpqB7Cb71CxIGE/cVA6KfitIilMEvGv7tZTWJLu1++S7mK99XR9lHQvy9GDeoE0Q3qYTdUL9UqpSnqUVbGleq20jsVYiCapmD4pTCRQ1CIjXywa0WRdtxFHFHhlrxppMxx+OZ75Fb/FryZqSXrA6nf4HQOl/pAlZ7J0tY+WbR1kS3fkklw6QgrKQ1iQ5Sg5lhw1qAa1wfq9tqAj6JggjVcLbHmOUmkqFMmTWaFSaClUi7Vivdi6gDxCK6WFsECexxYpiywL1HJ1oa3KVuWolmpgubyCLdOetK5yrJe3OF51PIheIyxTo+JDEzWaOPgs7U+zPxGPc3wF57/nv+Ps4jW3fFlAa3dmtF5BTa1E++uJ8ZFOAqQJz7c4a4TmIDsjlEaHy18dd9jXmNjgWhlhIxHQxa6p1jhQPUO6ouKevYBScQlHntZ86Wobuoo3hBlmu7KFJc5Oj0mPTY9L96fHpycMSg7GBGODcUF/MD6YkBeTF5sXl+fPi89LyEsuT14WUxNbE1fjr4lflrA6uS75SnJs59TOSZ0TCmML4wr9hfHlseVx5f7y+KWxS+OW+pfGd7nVxu+kfV2JWQ408q5Zvftkxt96WoRLxz7c81jZhsaGhkFNy/ecuX6NSi+vLzyYX3Js4n9dkTJLKydXvHsgddT1x3aVFp148ehxd9VTvXrtSk5uE1Y9B3k1UfGgm/SRfsHIqEbi8DQydaWjga6HCJmo0lCX2zokxoyAMjLEIX/parPwpukHC2OXxtbFgmnTHehhYEQQJdrpmRBLeLGhof++R8+0k/Yzj+67fvLlZ5/dsePZZ1+Gg9JDP7bsKC6ig6mK9+Ai7j3z+ednEDrwqkIZekg0ntJJxEu1anU58+6krNFGj3RpdDfYVvqivZLqVclIye0c4jNRbDYjERGohZzo1ZAXTR0UUx5TF/OHmCsxbBAZRAdJg7yDolkPS5qapvXQy0gZLZPKvGXR2qQ5SI833jwG+nqRJtOXol+1CB+rWOSqtv22c4dmnJw85Q8P86v8JE1t+5haGqRtyzc2OqSHJh472bv33m49aD+q0zB6D/+gef2BvZuFR0lDhv+AvA4jBUEfM6hN3anQGrLeoTTpUpiFWDSm2p3WUR4RZeji8LKKw2tkvcMsi0NtYHPbwOZmt6mil9DPGpfxXMYg42DQm+et8wKijkjG0JBbS8zKFOoi/VA/5V6axt9prK/fe1TxbMibNmVVWxq8s2r0kd2C13y8PBF5bcWIaEQwMdIWo7mrw8IbndDYNbEhuUlrdB6NiukaSVTbUMXt9g/BMKi5Ux2aL4UUgl8UnM5Grei2tFtdN6EVpvhDChxhSDdP1ztph6pgDB0ekYVp3LZ1a7dtW7tuWwPnrUV7xozZfP9vD2Tvf/Sttra3Ht2f3SDdeer990+dfP/9r/jH/MuY2N/06Hb0tQenTEZHAFSm/SdP2SV0+QQyGfN5jOktpDueEcfkfaRJYlSVSa5qtKHLFVxrE0eSOAHytEI8BfBICkPtFYHJiQa85MJrdYrnS1yv/T0+3lzPipnx4KDPKlmI45jNUsOOkibbPkM1mHKfnao2kmuYq1/KdnfEgC1GiBm4kSvoynMVuspdoY08Ske8F9rwpd/m3jF9lLnryj8d31S0QUn5EjXlJiUJh8h6iaokVzZCAX960G6g381jhaycXWFKCH1EXfH82CK07DCek8UozTDSJxgJGgEHVWocrgZbk04llYwWHi/XPKZFSpM2UEjP5caz6ECh922vJKw50RUSnckW08fIxQ2PPrpuT2Njzm/mn3hD2nr9Z9LmLZuPbb1eIxfuLSn+psNi55taFIFaFKY0ukmjrUFkUW7nGHB7h/wkiwomDoqsJJVKlaVKrdKq9Cprpa3KXuWoclYZVa5Kd13klUjX7XHObclWxdo9u9et2bNnzRXq5pev/JV/Q13w4eenT3/+xamTX27ip3gL/xrNMxut0EP7IYaHUc+3IobC190VjO70dQ2OlfQoNMWgnxtqerxc4e0yMkK4Xup0d0Et5O/+HCvTSYEbrEFcJHTLtyo7rWhs7L+v8ixpbz9buU/qhx7vZQE7ru9V9F3FRbyJ/4B3UxH9qtPhheQGIxA7F8GgT8FYwWWFGkeD1mTRFTyuc93C6EwNRi934axwawfywraECYmZrukWcUXAiLjhPTa9jHgcXhbWywcH3K4zx67vR2GVTmEMdytr/wRO4m7J5PPgQLtNcljHxsWqmmTRx8bFxebo1tg42Uuq6QrZU+1d0aXRJTcG8NBMidWtcdEWcn+06rConoQhKQKrCy2XhLpnd3heg3932fjusrszvnF8jeGgxXyKaCZZRDOzfLrP6rP1Qufbw9rDNkAboA+wDrBZ/cRPk6QUPcXaLSzNk+btFp4SmxKX6k+NT0qu1qut1bZqu1t8UydJiq5YwQZ2cIATDIiEKIgGnxyjJaelDkr9eWpV6tLU1al1qVdSu2AgMScUJAsRxZlZmJJ4a7ifhjwUvjIjHJ4avWPiihWT1w5q3vb9v098fWbpG0WPryzZHdz93J/fKj0gD9qbkpKfHxwe7+i2YcWmg4mJx7KyCsaMzAs4k9Y9vnlPrJBlX3Qf37LNaIN4pjqY6oSdxEWb1BrdijxGHTPcDmGDpjvPML15y9VQupWdvv9VLxVWKHy4J3yA8Ohds4Qvd9EFtJIvG1lx9OjFF2tq2Gb+u1XX61aM3rjlj1LhKnqX8IF70QonmNbvIQOCvpv2v1KnTZ4GG1q/xzoa/UCuV5hjdkijLmXccAJl3uPCCYThkR4yuxtne1e6VziBVxoa7tk3/8Qp+jY9LG2/XrRly7GtUuW1uj2lU67ADkH9neiBquRCopBrwWTA6FiWXFRi4gWSgvEnhshKjgTkNaYwjIuZTCzi2w3zyCOhI8+TL75tEMkcMZPsiND3C/9jxCxy0V8Nw9hXRL3V0lLpGWmrpIqNNNBQi700CqLkrpibpkKq7FezSBbtD/3ldFVEssNhuJzLhilBdTwZTwugQM5TS0kpnQ7T5alsmlKozifzaCVUyvPZYmUZWUZXwAqMVKuVWlJL10sb4Tn5ObZe2cFeVurV4+qHart6V2fkShPvfJ0+RB96nf+sVS5sy4c91+qQQ+ORAVnIIRv9CnOLcaHcYhzmFuNEbjHun8otXvsfcgvBw5H1LvENjVs8wsTDGmKj4CsmR7ZQkj+y3rjJ3X85JaHBdiaFS+EsQc/Sh0vDWS5mHw9KD7Jxep4+W5rNSvVFKItFmIHUSBuk59havUlqYm9JJ+FtFsMkDRTZynTVquHL5pUiIVyOYtFqtOaxem0BEqCJUjLEywGWoCRYAmoyZiPx1kRbNvSR+6jZIgeRhkGuHJRzWFAJWoLqYMw/BltF/iGkOF7Kk8ew+5X7LXnqWC1fH2edQoppiTQDSuQZbIYywzJbK7JOtZU55pP5dJG0BBbKS1C6VcpiS5VlobpIq9Iq9UesS2w10nKG+QhZT9dKa2CT/Dx7TnnOskENptXatji2k+10q7QVdsu72U5lp2W3utX2quO30j44Kh9hDdprjmbpdTgrv8kWmRoRTcWHJlpp4viGzz5997NPG/h77/7123dRN2phhoBrdVDbNkNYEZ5Wwoqc9KngPRZV0lzEqbusOiFOh8tJnHaXzU7Ey2FHpbG5UGVy7FbNIFZWA0cd1ibDYbfpGmqK6pSdVqNTO1RT7tZOM+uMLJubzZzdQFfQ0pHx/CNNYF9HZAgduKIQpioa2MP1CLthT7Rn2Yfr9+mj7RO1ifoMvca+1L7G7sZ8VVNQylaH1RlBvZIhGyxC91g9tihHlDOZJKG/98t+lqqmaAE9yZpkS7Z3c3Rz+l190U6zpHQ5nfXT+1j72PrZsx3ZznTX3SRIg1IQgnKwQ/o52hB9qH24Y7gz6MonY+gYaRzkyXlsvDLOMl59QHsANWCcrcBR4MxzldJSaZo+3THdWeiqVBc6FjpXkCe1ZdZlthX2FY4Vzg3aOus620bHRudW61bbbsduZ73rbdeHrnZXCcqQOWgogBtEzbxUWjN67aNrZo7Kz4znA0KmPu3U4o3DqvPl0W1rYWZH3M92YTSXRIYFw7qaYb4tvos9VnXZ4g3PqIA4BzLESWAMFNF98x0k6NLsrp1uKaqGdFmvxLmbrM60gZ9lZPCBlzMw5M9Ivy3Mvxnqm4eFRXSIs4zt6oz7ucUM/ffWT0nuSn+8LQfozAM2pKRMmxLKB9LEt8OIr5dEkvuDruhcEqGGOz2yqkK4royKuokvH4gxUtCt4vFm1Di6HAvf51ivkSZGBbaXufktVgZGwe3RddGro5dGG9EsFF/+BGXEmGI8LI8IYfrKbxoF5j82NopovBPHQ68KpOn+Lzt42oFj36AzIhdjfN2mqobsdowKF/iF0BPYYUa1U5MxzXJpTXZJIMZNrKjJup8mS9IIiTTyQzfzJRFWs8qfZEy4u9KGu6eSnmRisEtabkR3tZsR7VWjumkkTlGTYrWErqN63WRUc4Z4tpnsioiOS9yZ5MLMr+exbvsMsj7cktQUGROfNnDgpYwMYX9GSwZ+QlLukGbfPn1vsKpT5rekeAy5J9I8Id4HfMmjn0BR3yvtF+zskD0gN1HsISmP84UlC2Z2MreTNKmTtyZ1XcnoYHhKrl01wrt4VEMTXxXHR2txiaOSb6HMJMxUgy4+/854l1Rj67rea4lvckbFhki6OvDv6emT+RP2/yRfDenprbLooOMGDbtvlcsN2YR+w5UKNgx9Q9v7c+fA70icav5g+s4vHJc639//qW2Uo0AT/5NQb/y+ivMss3gMIQ7+/Z9axzgK/u7X2CT5nPmbJpFEuvkUWnUEqUH4CKEWYRNCMcJmhFXyeXJa/qS91fIlGSx/TiqZh8yRW8gc6R2SJspSNjkhZbe/J97MTQ7Ln5I52H4YRmC5OymDRNIX2/fKTeRO5SkyXrzZEzhXD4GlCeWUdht+OWQR+YHOpKelFGmNdB6s8IX8Czx7erOl7JJiU4YrU5WXlKuWeyyL1VS1VN2hZesjrenW5db/sEXalts+NykOQD7pTqYRG+q5QTYIDsleKRzf4vdTC5kofgmXNWRYuvnbsyhTEo61UFkiKs3tKMMt7fItZUa60NEdZYV4aCm5h5SRcsR/LplOpuLu84ifpJApaGF+kkHS8c7E0mQc4Uc6p2N/BcJcUkKKyCzSA1uHk9k4vheW7iYz8faT+2+sVWHWSvBdgnMewWcxjtT/iV373Ng1H3d6BPcSv5LOxtECjyKc86/tOBhLM3DeeDIfR0zBsUXmaiXmjCKTIj+uMhuf5ThmMq47Hcf5cX4Z7l5k9v10nbHmKhWIURneD2Or2LUCx5aZK2Xg3pkk67ZZnXNC/8Ih7b8w/3fx91fAtA3x7xuRU3jNf/J0wZMhE7Oc/rjiUDIMeT+CjCL3kvvIGKR6LBmHOzxAJpACXPNnVCLHyQnxTY1l/uzpuTnp6R3vzI537wZpabD9GodWD/wYgB8y4Pta+JsDvuNwlcN/BeBbB/y1Fq4E4Jsn72bfcLhcC1/XQksrfNUKf+HwZX/4Igc+5/BZBnx6aSz7tBYu4cBLY+GTj9PYJ63wcRp8xOHPHD7MgP/0wAe18D6H99zwH0vg3SPw7xz+hMP/tAQuXhjKLi6BC0Ph/B+j2XkOf4yGdzj8gcPbHN7icK4Wzp6JZWc5nImFNzPgNIc3lrnYGz74fTg0c3idw+84nOBwnMNrHI5xOMqhicMRDodd0FgdYI0cGg5hZMjh0MFJ7NAROLRUPvjbADs4KdgOB4PybwNwgMNvamE/h30c6jm8ymFvMbzigD27A2xPMeze5Wa7A7DLDTsR6Z2tsIPDyxy2c9jmhq0cXnrRwV7KgBcd8OtiqMMhdbWwhcPmF2yYL8ILNtj0fCTbVAzPbzTY85Gw0YANOjzHYX2tna3nUGuHdThpXS2sXeNga1NgjQOebYVnVh9hz3BYvWoSW30EVi+VV/0qwFZNglVB+VcBeJrDyqd6sZUcnuoFTyKZT94NK5Zb2QoPLLdCDTbUFEM1cqo6AMtc8EsOTzzuYk9weNwFj3FYyqGKQ7D9F0uWsF9wWLIEHi2GynwvqwzAYg6LOCx0wAIbPKLDfA7zWqGiFea2wpxWKOdQxmE2h5nx8DCHGa4cNmMsTOcwbQlMxUophxIOxRymcJjMoag/FLbCQzaYxOFBDhM5FEzQWUErTNDhgfBI9kAGjOcwDncelwP5XhhLDTa2C9zvgTEjwtgYDnlWuI/D6HsNNprDvQaM4jASe0ZyGDHcYCPCYHiMnQ03YJgdhnLIrYUhtTCYwz1ST3ZPK+QcgbtHQpDDIA533elmd3ngzoFOdqcbBg6ws4HBdicMsEN/Dtkc+vX1sH6t0LePwfp6oE+WlfUxIMsKvWMh0w4Zd1hZBoc7rJCeZmXpdkizQq+eGutlQE8NemRA924B1r0YuqW6WbcApLohJTnAUu6G5AB0DVhZVycErJDEIZFDghPikc54N/iLIa4VYpGE2GKIsYMPOejjEN0KUTkQiZVIDl2KIQI5FcEhHCeFR4KXg4dDGAc3DnBzcCGtrhwwloCzGBwc7LZwZudgw9G2cLBy0A3QOKg4TOVg8YBSDDJ2yqgBXsBW4JjOGkzqCdQAwoE20OJlT9Pu/z9c5P81Av/HK+a/ARw2Hp4KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDc2IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nKWMWQqAQAxDH7jvjjPu2/1vaRj8FAQt5CWlJfBzgpd7SERMQkpG/qG/uL30rB4+as+GFkOnZHFiLw2MTPKZhZWNnUPbeQFIhQHuCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAzNDEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicXVJNa4QwFLz7K3LcHhZ33WooiFC2Fw/9oLYn2YObPBehxhDdg/++SSaxUEGHmTfzfJqXnuuXWg0LSz/MJBpaWD8oaWie7kYQu9JtUMkxY3IQS2D+KcZOJ6kNN+u80FirfkrKkqWftjgvZmW7Zzld6SFhjKXvRpIZ1I3tvs8NpOau9Q+NpBZ2SKqKSeptu9dOv3UjsdSH97W09WFZ9zb25/haNbHM8yNGEpOkWXeCTKdulJQHe1Ws7O1VJaTkv7r9HB+79ps/c35AC7w4+QT5VECO9AjIACfAIyCPCd8gRyIPffPQN4fZQQuEzCHzIHPIBd7noAVCRpMiuCPFAEURi/A+gXbBu1FUBagIVRFk8jIPwwda9FH1Jo6/wMMUkeKncMzEMRMvohVJDOGgBV7cYcVTcefmlmxbCnE3xu6D30S/CG4FBkXbsupJu5S7fwGwbcFnCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL1R5cGUgL0ZvbnQgL1N1YnR5cGUgL0NJREZvbnRUeXBlMiAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zCi9DSURTeXN0ZW1JbmZvIDw8IC9SZWdpc3RyeSAoQWRvYmUpIC9PcmRlcmluZyAoSWRlbnRpdHkpIC9TdXBwbGVtZW50IDAgPj4KL0ZvbnREZXNjcmlwdG9yIDE5IDAgUiAvVyAyNCAwIFIgL0NJRFRvR0lETWFwIDIyIDAgUiA+PgplbmRvYmoKMjEgMCBvYmoKPDwgL1R5cGUgL0ZvbnQgL1N1YnR5cGUgL1R5cGUwIC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMKL0VuY29kaW5nIC9JZGVudGl0eS1IIC9EZXNjZW5kYW50Rm9udHMgWyAyMCAwIFIgXSAvVG9Vbmljb2RlIDI1IDAgUiA+PgplbmRvYmoKMTkgMCBvYmoKPDwgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9Gb250TmFtZSAvQk1RUURWK0RlamFWdVNhbnMgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0FzY2VudCA5MjkgL0Rlc2NlbnQgLTIzNiAvQ2FwSGVpZ2h0IDAKL1hIZWlnaHQgMCAvSXRhbGljQW5nbGUgMCAvU3RlbVYgMCAvRm9udEZpbGUyIDIzIDAgUiAvTWF4V2lkdGggOTg5ID4+CmVuZG9iagoyNCAwIG9iagpbIDMyIFsgMzE4IF0gNDggWyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgXSA4MCBbIDYwMyBdIDg0IFsgNjExIF0gODcKWyA5ODkgXSA5NyBbIDYxMyBdIDEwMCBbIDYzNSA2MTUgMzUyIDYzNSBdIDEwNSBbIDI3OCAyNzggXSAxMDggWyAyNzggXSAxMTAKWyA2MzQgNjEyIDYzNSBdIDExNCBbIDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IF0gMTIxIFsgNTkyIF0gXQplbmRvYmoKMyAwIG9iago8PCAvRjEgMjEgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAwIC9jYSAxID4+Ci9BMiA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+Ci9BMyA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAwID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9NMCAxMyAwIFIgL00xIDE0IDAgUiAvTTIgMTUgMCBSIC9NMyAxNiAwIFIgL000IDE3IDAgUiAvTTUgMTggMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC04IC04IDggOCBdIC9MZW5ndGggMTMyCi9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0XlCm49gpvJJN5/OxAHxNRNA+3L4xflLwlvVAqMT5Kw5DRLrjcNKa3LxIhhXi2Lcgwa1WLtXIeD+gwNN0a3GEthSm2Njje6mW7jPfmjD5sVYvTD4fBKuGcxZMKVEmNw+L3wsj38H+HpRQ38IdrpB2N8RA4KZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC04IC04IDggOCBdIC9MZW5ndGggMTMyCi9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0XlCm49gpvJJN5/OxAHxNRNA+3L4xflLwlvVAqMT5Kw5DRLrjcNKa3LxIhhXi2Lcgwa1WLtXIeD+gwNN0a3GEthSm2Njje6mW7jPfmjD5sVYvTD4fBKuGcxZMKVEmNw+L3wsj38H+HpRQ38IdrpB2N8RA4KZW5kc3RyZWFtCmVuZG9iagoxNSAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC04IC04IDggOCBdIC9MZW5ndGggMTMyCi9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0XlCm49gpvJJN5/OxAHxNRNA+3L4xflLwlvVAqMT5Kw5DRLrjcNKa3LxIhhXi2Lcgwa1WLtXIeD+gwNN0a3GEthSm2Njje6mW7jPfmjD5sVYvTD4fBKuGcxZMKVEmNw+L3wsj38H+HpRQ38IdrpB2N8RA4KZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC04IC04IDggOCBdIC9MZW5ndGggMTMyCi9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0XlCm49gpvJJN5/OxAHxNRNA+3L4xflLwlvVAqMT5Kw5DRLrjcNKa3LxIhhXi2Lcgwa1WLtXIeD+gwNN0a3GEthSm2Njje6mW7jPfmjD5sVYvTD4fBKuGcxZMKVEmNw+L3wsj38H+HpRQ38IdrpB2N8RA4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC04IC04IDggOCBdIC9MZW5ndGggMTMyCi9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0XlCm49gpvJJN5/OxAHxNRNA+3L4xflLwlvVAqMT5Kw5DRLrjcNKa3LxIhhXi2Lcgwa1WLtXIeD+gwNN0a3GEthSm2Njje6mW7jPfmjD5sVYvTD4fBKuGcxZMKVEmNw+L3wsj38H+HpRQ38IdrpB2N8RA4KZW5kc3RyZWFtCmVuZG9iagoxOCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC04IC04IDggOCBdIC9MZW5ndGggMTMyCi9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0XlCm49gpvJJN5/OxAHxNRNA+3L4xflLwlvVAqMT5Kw5DRLrjcNKa3LxIhhXi2Lcgwa1WLtXIeD+gwNN0a3GEthSm2Njje6mW7jPfmjD5sVYvTD4fBKuGcxZMKVEmNw+L3wsj38H+HpRQ38IdrpB2N8RA4KZW5kc3RyZWFtCmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagoyNiAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My45LjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My45LjApCi9DcmVhdGlvbkRhdGUgKEQ6MjAyNTA1MzAxNjU3NDYrMDgnMDAnKSA+PgplbmRvYmoKeHJlZgowIDI3CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDEzNzI3IDAwMDAwIG4gCjAwMDAwMTE4OTggMDAwMDAgbiAKMDAwMDAxMTkzMCAwMDAwMCBuIAowMDAwMDEyMDY4IDAwMDAwIG4gCjAwMDAwMTIwODkgMDAwMDAgbiAKMDAwMDAxMjExMCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDMgMDAwMDAgbiAKMDAwMDAwMjUyMiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDI1MDEgMDAwMDAgbiAKMDAwMDAxMjE5NyAwMDAwMCBuIAowMDAwMDEyNDUyIDAwMDAwIG4gCjAwMDAwMTI3MDcgMDAwMDAgbiAKMDAwMDAxMjk2MiAwMDAwMCBuIAowMDAwMDEzMjE3IDAwMDAwIG4gCjAwMDAwMTM0NzIgMDAwMDAgbiAKMDAwMDAxMTQ0OCAwMDAwMCBuIAowMDAwMDExMDg4IDAwMDAwIG4gCjAwMDAwMTEzMDEgMDAwMDAgbiAKMDAwMDAxMDUyNiAwMDAwMCBuIAowMDAwMDAyNTQyIDAwMDAwIG4gCjAwMDAwMTE2NzIgMDAwMDAgbiAKMDAwMDAxMDY3NCAwMDAwMCBuIAowMDAwMDEzNzg3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgMjcgL1Jvb3QgMSAwIFIgL0luZm8gMjYgMCBSID4+CnN0YXJ0eHJlZgoxMzk0NAolJUVPRgo=", "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2025-05-30T16:57:46.225324\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.0, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "df[\"Words Per Tweet\"] = df[\"text\"].str.split().apply(len)\n", "df.boxplot(\"Words Per Tweet\", by=\"label_name\", grid=False, showfliers=True,\n", " color=\"black\")\n", "plt.suptitle(\"\")\n", "plt.xlabel(\"\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从情感上看，我们可以看到对于每种情绪，大多数推文大约在15个字左右，最长的推文远远在DistilBERT的最大上下文大小以下。如果文本超过模型的上下文大小，需要对其进行截断，这可能会导致性能下降，如果被截断的文本包含关键信息的话；但在这种情况下，看起来这不是一个问题。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "让我们现在想一想如何将这些原始文本转换为适合 Transformers 的格式！顺便说一下，我们也要重置数据集的输出格式，因为我们不再需要DataFrame格式了。" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.440672Z", "iopub.status.busy": "2025-05-30T08:57:46.440096Z", "iopub.status.idle": "2025-05-30T08:57:46.444362Z", "shell.execute_reply": "2025-05-30T08:57:46.443783Z", "shell.execute_reply.started": "2025-05-30T08:57:46.440640Z" } }, "outputs": [], "source": [ "emotions.reset_format()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 从 Text 到 Tokens" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transformer模型如DistilBERT不能接收原始字符串作为输入；相反，它们假定文本已经被标记化（tokenized）并编码（encoded）为数值向量。 标记化（Tokenization）是将字符串分解为模型中使用的原子单位的步骤。 人们可以采用几种标记化策略，通常从语料库中学习单词分割的最佳方法。 在查看用于DistilBERT的标记器之前，让我们考虑两种极端情况：字符标记化和单词标记化。" ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### 字符标记化（Character Tokenization）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "最简单的标记化方案是将每个字符单独提供给模型。在Python中，`str`对象在底层实际上是数组，这使我们能够仅用一行代码快速实现基于字符级别的标记化：" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.445756Z", "iopub.status.busy": "2025-05-30T08:57:46.445267Z", "iopub.status.idle": "2025-05-30T08:57:46.449073Z", "shell.execute_reply": "2025-05-30T08:57:46.448455Z", "shell.execute_reply.started": "2025-05-30T08:57:46.445725Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['T', 'o', 'k', 'e', 'n', 'i', 'z', 'i', 'n', 'g', ' ', 't', 'e', 'x', 't', ' ',\n", "'i', 's', ' ', 'a', ' ', 'c', 'o', 'r', 'e', ' ', 't', 'a', 's', 'k', ' ', 'o',\n", "'f', ' ', 'N', 'L', 'P', '.']\n" ] } ], "source": [ "text = \"Tokenizing text is a core task of NLP.\"\n", "tokenized_text = list(text)\n", "print(tokenized_text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这是一个不错的开始，但还没有完成。我们的模型期望将每个字符转换为一个整数，有时候这个过程被称为数字化。一个简单的方法是使用一个独一无二的整数对每个唯一的标记(在这种情况下是字符)进行编码：" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.450335Z", "iopub.status.busy": "2025-05-30T08:57:46.449937Z", "iopub.status.idle": "2025-05-30T08:57:46.454607Z", "shell.execute_reply": "2025-05-30T08:57:46.454016Z", "shell.execute_reply.started": "2025-05-30T08:57:46.450307Z" }, "tags": [] }, "outputs": [], "source": [ "# TODO: write a dictionary that maps each token to an integer\n", "token2idx = {\n", " ' ': 0,\n", " 'a': 1,\n", " 'b': 2,\n", " 'c': 3,\n", " 'd': 4,\n", " 'e': 5,\n", " 'f': 6,\n", " 'g': 7,\n", " 'h': 8,\n", " 'i': 9,\n", " 'j': 10,\n", " 'k': 11,\n", " 'l': 12,\n", " 'm': 13,\n", " 'n': 14,\n", " 'o': 15,\n", " 'p': 16,\n", " 'q': 17,\n", " 'r': 18,\n", " 's': 19,\n", " 't': 20,\n", " 'u': 21,\n", " 'v': 22,\n", " 'w': 23,\n", " 'x': 24,\n", " 'y': 25,\n", " 'z': 26,\n", " 'T': 27,\n", " 'N': 28,\n", " 'L': 29,\n", " 'P': 30,\n", " '.': 31\n", "}" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.455865Z", "iopub.status.busy": "2025-05-30T08:57:46.455419Z", "iopub.status.idle": "2025-05-30T08:57:46.458901Z", "shell.execute_reply": "2025-05-30T08:57:46.458292Z", "shell.execute_reply.started": "2025-05-30T08:57:46.455836Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{' ': 0, 'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9,\n", "'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18,\n", "'s': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, 'T': 27,\n", "'N': 28, 'L': 29, 'P': 30, '.': 31}\n" ] } ], "source": [ "print(token2idx)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这为我们提供了一个从词汇表中的每个字符到唯一整数的映射。现在我们可以使用 `token2idx` 将标记化文本转换为整数列表：" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.460148Z", "iopub.status.busy": "2025-05-30T08:57:46.459708Z", "iopub.status.idle": "2025-05-30T08:57:46.463364Z", "shell.execute_reply": "2025-05-30T08:57:46.462757Z", "shell.execute_reply.started": "2025-05-30T08:57:46.460118Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[27, 15, 11, 5, 14, 9, 26, 9, 14, 7, 0, 20, 5, 24, 20, 0, 9, 19, 0, 1, 0, 3, 15,\n", "18, 5, 0, 20, 1, 19, 11, 0, 15, 6, 0, 28, 29, 30, 31]\n" ] } ], "source": [ "input_ids = [token2idx[token] for token in tokenized_text]\n", "print(input_ids)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "每个标记现在都映射到了一个唯一的数字标识符（因此称为`input_ids`）。最后一步是将`input_ids`转换为一个独热向量（One-hot vectors）的二维张量。One-hot vectors经常用于机器学习中对分类数据进行编码，这些数据可以是有序的或无序的。例如，假设我们想要对《变形金刚》电影中角色的名字进行编码。一种方法是将每个名字映射到一个唯一的ID，如下所示：" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.464648Z", "iopub.status.busy": "2025-05-30T08:57:46.464185Z", "iopub.status.idle": "2025-05-30T08:57:46.471827Z", "shell.execute_reply": "2025-05-30T08:57:46.471270Z", "shell.execute_reply.started": "2025-05-30T08:57:46.464618Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameLabel ID
0Bumblebee0
1Optimus Prime1
2Megatron2
\n", "
" ], "text/plain": [ " Name Label ID\n", "0 Bumblebee 0\n", "1 Optimus Prime 1\n", "2 Megatron 2" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "categorical_df = pd.DataFrame(\n", " {\"Name\": [\"Bumblebee\", \"Optimus Prime\", \"Megatron\"], \"Label ID\": [0,1,2]})\n", "categorical_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这种方法的问题在于它在名称之间创造了一个虚构的顺序，而神经网络非常擅长学习这类关系。因此，我们可以为每个类别创建一个新列，并在该类别为真时分配一个1，在其他情况下分配一个0。在 Pandas 中，可以使用 `get_dummies()` 函数来实现这一点，方法如下：" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.472992Z", "iopub.status.busy": "2025-05-30T08:57:46.472670Z", "iopub.status.idle": "2025-05-30T08:57:46.479705Z", "shell.execute_reply": "2025-05-30T08:57:46.479154Z", "shell.execute_reply.started": "2025-05-30T08:57:46.472964Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
BumblebeeMegatronOptimus Prime
0100
1001
2010
\n", "
" ], "text/plain": [ " Bumblebee Megatron Optimus Prime\n", "0 1 0 0\n", "1 0 0 1\n", "2 0 1 0" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.get_dummies(categorical_df[\"Name\"],dtype=int)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这个`DataFrame`的行是one-hot向量，其中有一个\"hot\"条目为1，其他地方都是0。对两个one-hot编码的加和结果很容易解释：两个\"hot\"条目表示相应的标记出现。\n", "\n", "下面我们可以通过将`input_ids`转换为tensor，应用`one_hot()`函数来在PyTorch中创建one-hot编码，如下所示：" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.485725Z", "iopub.status.busy": "2025-05-30T08:57:46.485387Z", "iopub.status.idle": "2025-05-30T08:57:46.490577Z", "shell.execute_reply": "2025-05-30T08:57:46.490029Z", "shell.execute_reply.started": "2025-05-30T08:57:46.485704Z" } }, "outputs": [ { "data": { "text/plain": [ "torch.Size([38, 32])" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import torch\n", "import torch.nn.functional as F\n", "\n", "input_ids = torch.tensor(input_ids)\n", "one_hot_encodings = F.one_hot(input_ids, num_classes=len(token2idx))\n", "one_hot_encodings.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "对于这38个输入标识符，我们现在有一个n维的独热向量，因为我们的词汇表包含n个字符。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 警告：在`one_hot()`函数中始终设置`num_classes`非常重要，否则one-hot向量可能会比词汇表的长度短（需要手动用零填充）。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "通过检查第一个向量，我们可以验证在由`input_ids[0]`指示的位置上出现了1:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.491836Z", "iopub.status.busy": "2025-05-30T08:57:46.491361Z", "iopub.status.idle": "2025-05-30T08:57:46.496101Z", "shell.execute_reply": "2025-05-30T08:57:46.495532Z", "shell.execute_reply.started": "2025-05-30T08:57:46.491793Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Token: T\n", "Tensor index: 27\n", "One-hot: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", "0, 0, 0,\n", " 0, 0, 0, 1, 0, 0, 0, 0])\n" ] } ], "source": [ "print(f\"Token: {tokenized_text[0]}\")\n", "print(f\"Tensor index: {input_ids[0]}\")\n", "print(f\"One-hot: {one_hot_encodings[0]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从我们简单的例子中，我们可以看到字符级的标记化忽略了文本中的任何结构，将整个字符串视为字符流。虽然这有助于处理拼写错误和罕见单词，但主要缺点是语言结构如单词需要从数据中学习。这需要大量的计算、内存和数据。因此，字符标记化在实践中很少被使用。相反，在标记化步骤中会保留文本的某些结构。单词标记化是实现这一点的一种直接方法，让我们看看它是如何工作的。" ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### 单词表计划（Word Tokenization）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "相较于将文本分割成字符，我们可以将其分割成单词，并将每个单词映射到一个整数。从一开始就使用单词使模型能够跳过从字符中学习单词的步骤，从而减少训练过程的复杂性。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "一种简单的词汇标记器类使用空格来对文本进行标记(针对英文文本)。我们可以通过直接在原始文本上应用Python的`split()`函数来实现这一点（就像我们用来测量推文长度的方式一样）。" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.497324Z", "iopub.status.busy": "2025-05-30T08:57:46.496968Z", "iopub.status.idle": "2025-05-30T08:57:46.500428Z", "shell.execute_reply": "2025-05-30T08:57:46.499812Z", "shell.execute_reply.started": "2025-05-30T08:57:46.497283Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Tokenizing', 'text', 'is', 'a', 'core', 'task', 'of', 'NLP.']\n" ] } ], "source": [ "tokenized_text = text.split()\n", "print(tokenized_text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从这里开始，我们可以采取与字符标记器相同的步骤，将每个单词映射到一个ID。但是，我们已经可以看到这种标记方案可能存在一个问题：标点符号没有被考虑在内，因此`“NLP.”`被视为一个标记。考虑到单词可能包括词形变化、时态变化或拼写错误，词汇量很容易增长到数百万以上！\n", "\n", "> 请注意：有些词汇分词器会有针对标点符号的额外规则。为了将单词归一化为其词干（例如“great”、“greater”和“greatest”都变成“great”），也可以应用词干提取或词形还原，尽管会损失文本中的一些信息。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "拥有庞大的词汇量是一个问题，因为需要神经网络具有巨大数量的参数。为了说明这一点，假设我们有100万个独特的单词，并希望将这100万维的输入向量压缩到神经网络的第一层中的1千维向量。这是大多数自然语言处理架构中的标准步骤，这个第一层的权重矩阵将包含100万×1千 = 10亿个权重。这已经可以与最大的GPT-2模型相媲美，其总参数约为15亿！" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "自然地，我们希望避免在模型参数上浪费，因为模型训练是昂贵的，而较大的模型更难维护。一种常见的方法是通过考虑语料库中最常见的10万个单词来限制词汇量，并且舍弃稀有单词。不在词汇表中的单词被视为`UNK`(Unknown)，并映射到共享的`UNK`标记。这意味着我们在词汇标记化过程中丢失了一些潜在重要的信息，因为模型对与`UNK`相关的单词没有信息。\n", "\n", "是否有一种在字级和词级标记之间取得平衡，可以保留所有输入信息和部分输入结构？有：子词标记化（subword tokenization）。" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "### 子词标记化（Subword Tokenization）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "基于子词标记化的基本思想是结合字符和单词标记化的最佳方面。一方面，我们希望将罕见单词拆分成较小的单元，以使模型能够处理复杂单词和拼写错误。另一方面，我们希望将频繁出现的单词保持为独特实体，这样我们可以保持输入长度在可管理的大小范围内。子词标记化（以及单词标记化）的主要区别特征在于其是通过混合统计规则和算法从预训练语料库中学习的。\n", "\n", "在自然语言处理中常用的有几种子词标记化算法，但让我们从WordPiece算法[M. Schuster and K. Nakajima, \"Japanese and Korean Voice Search,\" _2012 IEEE International Conference on Acoustics, Speech and Signal Processing_ (2012): 5149–5152, https://doi.org/10.1109/ICASSP.2012.6289079.] 开始，这是BERT和DistilBERT标记器使用的算法。理解WordPiece工作原理最简单的方法是看它的实际运行过程。Transformers提供了一个方便的 `AutoTokenizer`类，允许您快速加载与预训练模型相关联的标记器 - 我们只需调用其`from_pretrained()`方法，提供Hub上模型的ID或本地文件路径即可。让我们开始加载DistilBERT的标记器：" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.501645Z", "iopub.status.busy": "2025-05-30T08:57:46.501242Z", "iopub.status.idle": "2025-05-30T08:57:46.555537Z", "shell.execute_reply": "2025-05-30T08:57:46.555029Z", "shell.execute_reply.started": "2025-05-30T08:57:46.501616Z" } }, "outputs": [], "source": [ "from transformers import AutoTokenizer\n", "\n", "model_ckpt = \"./distilbert-base-uncased/tokenizer\"\n", "tokenizer = AutoTokenizer.from_pretrained(model_ckpt)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "AutoTokenizer 类属于一个更大的[\"auto\" classes](https://huggingface.co/docs/transformers/model_doc/auto)，其工作是根据检查点名称自动检索模型的配置、预训练权重或词汇表。这使您能够快速在模型之间切换，但如果您希望手动加载特定类，也是可以的。例如，我们可以这样加载 DistilBERT tokenizer：\n", "\n", "```python\n", "from transformers import DistilBertTokenizer\n", "\n", "distilbert_tokenizer = DistilBertTokenizer.from_pretrained(model_ckpt)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 请注意：当您第一次运行AutoTokenizer.from_pretrained()方法时，您会看到一个进度条，显示从Hugging Face Hub加载的预训练的令牌化器的哪些参数。当您第二次运行代码时，它将从缓存中加载令牌化器，通常位于~/.cache/huggingface/。\n", ">\n", "> **在我们提供的作业中，已经完成了这个下载过程，因此只需要调用本地的文件就可以了**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "让我们看看这个分词器是如何工作的，让它吃掉我们这个简单的“将文本标记化是自然语言处理的核心任务。”示例文本：" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.556868Z", "iopub.status.busy": "2025-05-30T08:57:46.556377Z", "iopub.status.idle": "2025-05-30T08:57:46.560858Z", "shell.execute_reply": "2025-05-30T08:57:46.560382Z", "shell.execute_reply.started": "2025-05-30T08:57:46.556837Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'input_ids': [101, 19204, 6026, 3793, 2003, 1037, 4563, 4708, 1997, 17953,\n", "2361, 1012, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n" ] } ], "source": [ "encoded_text = tokenizer(text)\n", "print(encoded_text)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.561883Z", "iopub.status.busy": "2025-05-30T08:57:46.561492Z", "iopub.status.idle": "2025-05-30T08:57:46.564558Z", "shell.execute_reply": "2025-05-30T08:57:46.564029Z", "shell.execute_reply.started": "2025-05-30T08:57:46.561863Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['[CLS]', 'token', '##izing', 'text', 'is', 'a', 'core', 'task', 'of', 'nl',\n", "'##p', '.', '[SEP]']\n" ] } ], "source": [ "tokens = tokenizer.convert_ids_to_tokens(encoded_text.input_ids)\n", "print(tokens)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们可以在这里观察到三件事情。首先，一些特殊的`[CLS]` 和 `[SEP]`标记已经被添加到序列的开始和结尾。这些标记因模型而异，但它们的主要作用是表示序列的开始和结束。其次，这些token已经被转换为小写，这是这个特定检查点的一个特征。最后，我们可以看到\"tokenizing\"和\"NLP\"已经被拆分成两个标记，这是有道理的，因为它们不是常见词。在`##izing`和`##p`中的`##`前缀表示前面的字符串不是空格；当您将token转换回字符串时，带有这个前缀的任何标记应与前一个标记合并。`AutoTokenizer`类有一个 `convert_tokens_to_string()`方法来执行这个操作，所以让我们把它应用到我们的标记上：" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.565475Z", "iopub.status.busy": "2025-05-30T08:57:46.565160Z", "iopub.status.idle": "2025-05-30T08:57:46.568035Z", "shell.execute_reply": "2025-05-30T08:57:46.567613Z", "shell.execute_reply.started": "2025-05-30T08:57:46.565456Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[CLS] tokenizing text is a core task of nlp. [SEP]\n" ] } ], "source": [ "print(tokenizer.convert_tokens_to_string(tokens))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`AutoTokenizer`类还具有几个属性，可以提供有关分词器的信息。例如，我们可以检查词汇表大小：" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.569009Z", "iopub.status.busy": "2025-05-30T08:57:46.568782Z", "iopub.status.idle": "2025-05-30T08:57:46.571793Z", "shell.execute_reply": "2025-05-30T08:57:46.571406Z", "shell.execute_reply.started": "2025-05-30T08:57:46.568991Z" } }, "outputs": [ { "data": { "text/plain": [ "30522" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.vocab_size" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "和相应模型的最大上下文大小：" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.572583Z", "iopub.status.busy": "2025-05-30T08:57:46.572418Z", "iopub.status.idle": "2025-05-30T08:57:46.575510Z", "shell.execute_reply": "2025-05-30T08:57:46.575114Z", "shell.execute_reply.started": "2025-05-30T08:57:46.572565Z" } }, "outputs": [ { "data": { "text/plain": [ "512" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.model_max_length" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "另一个有趣的属性是了解模型在前向传递中期望的字段名称：" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.576342Z", "iopub.status.busy": "2025-05-30T08:57:46.576060Z", "iopub.status.idle": "2025-05-30T08:57:46.579268Z", "shell.execute_reply": "2025-05-30T08:57:46.578719Z", "shell.execute_reply.started": "2025-05-30T08:57:46.576324Z" } }, "outputs": [ { "data": { "text/plain": [ "['input_ids', 'attention_mask']" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.model_input_names" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们对单个字符串的标记化过程有了基本了解，让我们看看如何对整个数据集进行标记化！" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 警告：在使用预训练模型时，确保要使用与模型训练时相同的tokenizer是非常重要的。从模型的角度看，更换标记器就像是洗牌词汇一样。如果你周围的人开始将“house”这样的随机词语互换成“猫”，你也会很难理解发生了什么！" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "### Tokenizing 整个 Datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为了对整个语料库进行标记化，我们将使用`DatasetDict`对象的`map()`方法。在本书中我们会经常遇到这个方法，因为它提供了一种方便的方式来对数据集中的每个元素应用处理函数。正如我们将很快看到的那样，map()方法也可以用来创建新的行和列。\n", "\n", "要开始，我们需要的第一件事是一个处理函数来对我们的示例进行标记化：" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.580153Z", "iopub.status.busy": "2025-05-30T08:57:46.579913Z", "iopub.status.idle": "2025-05-30T08:57:46.582586Z", "shell.execute_reply": "2025-05-30T08:57:46.582157Z", "shell.execute_reply.started": "2025-05-30T08:57:46.580134Z" } }, "outputs": [], "source": [ "def tokenize(batch):\n", " return tokenizer(batch[\"text\"], padding=True, truncation=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这个函数将分词器应用到一批示例上；`padding=True`将用零填充示例，使它们的大小与批中最长的示例相同，`truncation=True`将截断示例以符合模型的最大上下文大小。要查看`tokenize()`的运行情况，让我们传递训练集中两个示例组成的一批。" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.583340Z", "iopub.status.busy": "2025-05-30T08:57:46.583161Z", "iopub.status.idle": "2025-05-30T08:57:46.587053Z", "shell.execute_reply": "2025-05-30T08:57:46.586489Z", "shell.execute_reply.started": "2025-05-30T08:57:46.583322Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'input_ids': [[101, 1045, 2134, 2102, 2514, 26608, 102, 0, 0, 0, 0, 0, 0, 0, 0,\n", "0, 0, 0, 0, 0, 0, 0, 0], [101, 1045, 2064, 2175, 2013, 3110, 2061, 20625, 2000,\n", "2061, 9636, 17772, 2074, 2013, 2108, 2105, 2619, 2040, 14977, 1998, 2003, 8300,\n", "102]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", "0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", "1, 1]]}\n" ] } ], "source": [ "print(tokenize(emotions[\"train\"][:2]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这里我们可以看到填充的结果：`input_ids`的第一个元素比第二个元素短，因此在该元素中添加了零，以使它们具有相同的长度。这些零在词汇表中有一个相应的 `[PAD]`标记，并且特殊标记集还包括我们之前遇到的`[CLS]` 和 `[SEP]`标记。" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.588252Z", "iopub.status.busy": "2025-05-30T08:57:46.587871Z", "iopub.status.idle": "2025-05-30T08:57:46.596997Z", "shell.execute_reply": "2025-05-30T08:57:46.596437Z", "shell.execute_reply.started": "2025-05-30T08:57:46.588224Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01234
Special Token[PAD][UNK][CLS][SEP][MASK]
Special Token ID0100101102103
\n", "
" ], "text/plain": [ " 0 1 2 3 4\n", "Special Token [PAD] [UNK] [CLS] [SEP] [MASK]\n", "Special Token ID 0 100 101 102 103" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokens2ids = list(zip(tokenizer.all_special_tokens, tokenizer.all_special_ids))\n", "data = sorted(tokens2ids, key=lambda x : x[-1])\n", "df = pd.DataFrame(data, columns=[\"Special Token\", \"Special Token ID\"])\n", "df.T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "另外，请注意，除了将编码后的推文返回为`input_ids`之外，分词器还会返回一个`attention_mask`数组列表。这是因为我们不希望模型被额外的填充令牌混淆：注意力掩码使模型能忽略输入中的填充部分。`attention_mask`提供了一个关于输入ID和注意力掩码如何填充的视觉解释。\n", "\n", "\"attention-mask\" " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "一旦我们定义了一个处理函数，我们就可以用一行代码在语料库中的所有分割中应用它：" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.598168Z", "iopub.status.busy": "2025-05-30T08:57:46.597833Z", "iopub.status.idle": "2025-05-30T08:57:46.619447Z", "shell.execute_reply": "2025-05-30T08:57:46.619001Z", "shell.execute_reply.started": "2025-05-30T08:57:46.598139Z" } }, "outputs": [], "source": [ "emotions_encoded = emotions.map(tokenize, batched=True, batch_size=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "默认情况下，`map()`方法会对语料库中的每个示例进行单独操作，因此设置 `batched=True` 将对推文进行分批编码。由于我们设置了 `batch_size=None`，我们的 `tokenize()` 函数将在整个数据集上作为单个批次应用。这确保了输入张量和注意力掩码在全局上具有相同的形状，我们可以看到这一操作已将新的 `input_ids` 和 `attention_mask` 列添加到数据集中：" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.620669Z", "iopub.status.busy": "2025-05-30T08:57:46.620232Z", "iopub.status.idle": "2025-05-30T08:57:46.623733Z", "shell.execute_reply": "2025-05-30T08:57:46.623122Z", "shell.execute_reply.started": "2025-05-30T08:57:46.620640Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['text', 'label', 'input_ids', 'attention_mask']\n" ] } ], "source": [ "print(emotions_encoded[\"train\"].column_names)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note: 在后面的章节中，我们将看到如何使用数据整理器来动态填充每个批次中的张量。在下一节中，全局填充将会派上用场，我们将从整个语料库中提取特征矩阵。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 训练文本分类器" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DistilBERT等模型经过预训练，可以预测文本序列中的屏蔽单词。然而，我们不能直接将这些语言模型用于文本分类；我们需要稍微修改它们。为了理解需要哪些修改，让我们看看基于编码器的模型（如DistilBERT）的架构。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"encoder-classifier\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "首先，文本被Tokenize并表示为称为token encodings的one-hot向量。Tokenizer词汇表的大小确定了Token encodings的维度，通常包含20k至200k个唯一token。接下来，这些Token Encodings被转换为Token Embeds，Embeds是存在于较低维度空间中的向量。然后，将Token Embeds传递通过Encoder Stack层以产生每个输入token的隐藏状态(Hidden States)。对于语言建模的预训练目标，每个隐藏状态被送入一个层，该层预测被mask的输入token。对于分类任务，我们用分类层替换语言建模层。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> note: 实际上，PyTorch在实践中跳过了为令牌编码创建独热向量的步骤，因为将矩阵与独热向量相乘与从矩阵中选择一列相同。通过从矩阵中获取令牌ID对应的列，可以直接完成这一步骤。当我们使用nn.Embedding类时，我们将会看到这一点。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们有两种选择来训练我们的 Twitter 数据集上的这种模型：\n", "\n", "* Feature Extractors（特征提取）：我们将隐藏状态用作特征，只是在其上训练一个分类器，而不修改预训练模型。\n", "* Fine-tune(微调)：我们端对端地训练整个模型，这也会更新预训练模型的参数。\n", "\n", "在接下来的章节中，我们将探讨 DistilBERT 的这两种选择，并检查它们的权衡。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Transformers作为特征抽取器" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "使用transformer作为特征提取器是相当简单的。我们在训练过程中冻结主体的权重，并使用隐藏状态作为分类器的特征。这种方法的优势在于我们可以很快地训练一个小型或浅层模型。这样的模型可以是一个神经分类层，也可以是一个不依赖梯度的方法，比如随机森林。如果GPU不可用，这种方法尤其方便，因为隐藏状态只需要预先计算一次。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"encoder-features\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 使用预训练模型" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们将使用另一个方便的自动类: Transformers的`AutoModel`。与`AutoTokenizer`类似，`AutoModel`具有`from_pretrained()`方法来加载预训练模型的权重。让我们使用这个方法加载DistilBERT 的 checkpoint(检查点)：" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:46.624972Z", "iopub.status.busy": "2025-05-30T08:57:46.624552Z", "iopub.status.idle": "2025-05-30T08:57:47.364530Z", "shell.execute_reply": "2025-05-30T08:57:47.363935Z", "shell.execute_reply.started": "2025-05-30T08:57:46.624942Z" } }, "outputs": [], "source": [ "from transformers import AutoModel\n", "import torch\n", "\n", "model_ckpt = \"./distilbert-base-uncased/model\"\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "model = AutoModel.from_pretrained(model_ckpt).to(device)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这里我们使用PyTorch来检查GPU是否可用，然后将PyTorch `nn.Module.to()` 方法链接到模型加载器。这确保了如果有GPU，模型将在GPU上运行。如果没有，模型将在CPU上运行，这可能会慢得多。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`AutoModel`类将token encodings转换为embeddings（嵌入），并通过编码器将其馈送以返回隐藏状态。让我们看看如何从我们的语料库中提取这些状态。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 抽取最后一个的隐藏层状态" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为了热身，让我们获取单个字符串的最后隐藏状态。我们需要做的第一件事是对字符串进行编码，并将标记转换为PyTorch张量。这可以通过向分词器提供`return_tensors=\"pt\"`参数来完成。" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.365868Z", "iopub.status.busy": "2025-05-30T08:57:47.365326Z", "iopub.status.idle": "2025-05-30T08:57:47.369760Z", "shell.execute_reply": "2025-05-30T08:57:47.369313Z", "shell.execute_reply.started": "2025-05-30T08:57:47.365844Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input tensor: tensor([[ 101, 2023, 2003, 1037, 3231, 102]])\n", "Input tensor to token: ['[CLS]', 'this', 'is', 'a', 'test', '[SEP]']\n", "Input tensor shape: torch.Size([1, 6])\n" ] } ], "source": [ "text = \"this is a test\"\n", "inputs = tokenizer(text, return_tensors=\"pt\")\n", "print(f\"Input tensor: {inputs['input_ids']}\")\n", "print(f\"Input tensor to token: {tokenizer.convert_ids_to_tokens(inputs['input_ids'].squeeze().tolist())}\")\n", "print(f\"Input tensor shape: {inputs['input_ids'].size()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "正如我们所看到的，生成的张量形状为`[batch_size, n_tokens]`。现在我们已经将编码作为张量，最后一步是将它们放置在与模型相同的设备上，并按照以下方式传递输入：" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.370611Z", "iopub.status.busy": "2025-05-30T08:57:47.370393Z", "iopub.status.idle": "2025-05-30T08:57:47.560517Z", "shell.execute_reply": "2025-05-30T08:57:47.559992Z", "shell.execute_reply.started": "2025-05-30T08:57:47.370593Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BaseModelOutput(last_hidden_state=tensor([[[-0.1565, -0.1862, 0.0528, ...,\n", "-0.1188, 0.0662, 0.5470],\n", " [-0.3575, -0.6484, -0.0618, ..., -0.3040, 0.3508, 0.5221],\n", " [-0.2772, -0.4459, 0.1818, ..., -0.0948, -0.0076, 0.9958],\n", " [-0.2841, -0.3917, 0.3753, ..., -0.2151, -0.1173, 1.0526],\n", " [ 0.2661, -0.5094, -0.3180, ..., -0.4203, 0.0144, -0.2149],\n", " [ 0.9441, 0.0112, -0.4714, ..., 0.1439, -0.7288, -0.1619]]],\n", " device='cuda:0'), hidden_states=None, attentions=None)\n" ] } ], "source": [ "inputs = {k:v.to(device) for k,v in inputs.items()}\n", "with torch.no_grad():\n", " outputs = model(**inputs)\n", "print(outputs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这里我们使用了`torch.no_grad()`上下文管理器来禁用梯度的自动计算。这在推理过程中非常有用，因为它减少了计算的内存占用。根据模型的配置，输出可以包含多个对象，例如隐藏状态、损失或注意力，这些对象以类似于Python中的`namedtuple`的方式排列。在我们的示例中，模型输出是`BaseModelOutput`的一个实例，我们可以通过名称简单地访问它的属性。当前模型仅返回一个属性，即最后的隐藏状态，所以让我们来检查其形状：" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.561632Z", "iopub.status.busy": "2025-05-30T08:57:47.561227Z", "iopub.status.idle": "2025-05-30T08:57:47.565027Z", "shell.execute_reply": "2025-05-30T08:57:47.564493Z", "shell.execute_reply.started": "2025-05-30T08:57:47.561611Z" } }, "outputs": [ { "data": { "text/plain": [ "torch.Size([1, 6, 768])" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "outputs.last_hidden_state.size()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "查看隐藏状态张量，我们发现它的形状为`[batch_size, n_tokens, hidden_dim]`。换句话说，对于每个输入令牌，都返回一个768维向量。对于分类任务，通常的做法是只使用与`[CLS]`令牌关联的隐藏状态作为输入特征。由于这个令牌出现在每个序列的开头，我们可以通过简单地索引到`outputs.last_hidden_state`来提取它，如下所示：" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.565784Z", "iopub.status.busy": "2025-05-30T08:57:47.565619Z", "iopub.status.idle": "2025-05-30T08:57:47.569253Z", "shell.execute_reply": "2025-05-30T08:57:47.568784Z", "shell.execute_reply.started": "2025-05-30T08:57:47.565766Z" } }, "outputs": [ { "data": { "text/plain": [ "torch.Size([1, 768])" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "outputs.last_hidden_state[:,0].size()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们知道如何获取单个字符串的最后一个隐藏状态，让我们为整个数据集做同样的事情，创建一个新的`hidden_state`列，存储所有这些向量。就像我们对分词器所做的那样，我们将使用`DatasetDict`的`map()` 方法一次性提取所有隐藏状态。我们需要做的第一件事是将先前的步骤封装在一个处理函数中：" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.570298Z", "iopub.status.busy": "2025-05-30T08:57:47.569897Z", "iopub.status.idle": "2025-05-30T08:57:47.573585Z", "shell.execute_reply": "2025-05-30T08:57:47.573095Z", "shell.execute_reply.started": "2025-05-30T08:57:47.570277Z" }, "tags": [] }, "outputs": [], "source": [ "# TODO: write a function to extract hidden states\n", "def extract_hidden_states(batch):\n", " # tokenize\n", " batch = tokenizer(batch[\"text\"], padding=True, truncation=True, return_tensors=\"pt\")\n", " \n", " # Place model inputs on the GPU\n", " inputs = batch.to(device)\n", " \n", " # Extract last hidden states\n", " with torch.no_grad():\n", " outputs = model(**inputs)\n", " last_hidden_state = outputs.last_hidden_state\n", " \n", " # Return vector for [CLS] token, turn it into numpy array\n", " return {\"hidden_state\": last_hidden_state[:,0].cpu().numpy()}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这个函数和我们先前的逻辑唯一的区别就是最后一步，我们将最终隐藏状态放回 CPU 上作为 NumPy 数组。当我们使用批处理输入时，`map()` 方法要求处理函数返回 Python 或 NumPy 对象。\n", "\n", "由于我们的模型期望张量作为输入，下一步要做的是将 `input_ids` 和 `attention_mask` 列转换为 \"torch\" 格式，如下所示：" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.574395Z", "iopub.status.busy": "2025-05-30T08:57:47.574212Z", "iopub.status.idle": "2025-05-30T08:57:47.580421Z", "shell.execute_reply": "2025-05-30T08:57:47.579899Z", "shell.execute_reply.started": "2025-05-30T08:57:47.574376Z" } }, "outputs": [ { "data": { "text/plain": [ "{'label': tensor(4),\n", " 'input_ids': tensor([ 101, 1045, 2514, 2004, 5457, 2055, 2166, 2004, 1037, 10563,\n", " 2030, 2004, 12323, 2094, 2004, 1037, 2095, 2214, 2158, 102,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0]),\n", " 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])}" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "emotions_encoded.set_format(\"torch\", \n", " columns=[\"input_ids\", \"attention_mask\", \"label\"])\n", "# show emotions encoded\n", "emotions_encoded[\"train\"][7]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们接着可以一次性提取所有分割中的隐藏状态：" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:47.581543Z", "iopub.status.busy": "2025-05-30T08:57:47.581174Z", "iopub.status.idle": "2025-05-30T08:57:50.699026Z", "shell.execute_reply": "2025-05-30T08:57:50.698491Z", "shell.execute_reply.started": "2025-05-30T08:57:47.581515Z" } }, "outputs": [], "source": [ "#hide_output\n", "emotions_hidden = emotions_encoded.map(extract_hidden_states, batched=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "请注意，这种情况下我们并没有设置`batch_size=None`，所以默认的`batch_size=1000`被使用了。正如预期的那样，应用`extract_hidden_states()`函数已经在我们的数据集中添加了一个新的`hidden_state`列。" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:50.700086Z", "iopub.status.busy": "2025-05-30T08:57:50.699772Z", "iopub.status.idle": "2025-05-30T08:57:50.703886Z", "shell.execute_reply": "2025-05-30T08:57:50.703177Z", "shell.execute_reply.started": "2025-05-30T08:57:50.700064Z" } }, "outputs": [ { "data": { "text/plain": [ "['text', 'label', 'input_ids', 'attention_mask', 'hidden_state']" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "emotions_hidden[\"train\"].column_names" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在我们有与每条推文关联的隐藏层状态，下一步是对其进行分类器训练。为了做到这一点，我们需要一个特征矩阵 - 让我们来看一下。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 创建特征矩阵" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "预处理后的数据集现在包含了我们训练分类器所需的所有信息。我们将使用隐藏状态作为输入特征，标签作为目标。我们可以按照以下方式轻松地创建对应的数组，使用众所周知的Scikit-Learn格式：" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:50.705215Z", "iopub.status.busy": "2025-05-30T08:57:50.704749Z", "iopub.status.idle": "2025-05-30T08:57:50.753305Z", "shell.execute_reply": "2025-05-30T08:57:50.752821Z", "shell.execute_reply.started": "2025-05-30T08:57:50.705174Z" } }, "outputs": [ { "data": { "text/plain": [ "((16000, 768), (2000, 768))" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "\n", "X_train = np.array(emotions_hidden[\"train\"][\"hidden_state\"])\n", "X_valid = np.array(emotions_hidden[\"validation\"][\"hidden_state\"])\n", "y_train = np.array(emotions_hidden[\"train\"][\"label\"])\n", "y_valid = np.array(emotions_hidden[\"validation\"][\"label\"])\n", "X_train.shape, X_valid.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在我们对隐藏层状态进行模型训练之前，最好先进行一次健全性检查，以确保它们提供了我们想要分类的情绪的有用表示。在下一个部分中，我们将看到可视化特征为实现这一目标提供了一种快速方法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 可视化训练集" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "由于在768维空间中可视化隐藏状态有点棘手，我们将使用强大的UMAP算法将向量投影到2D上。由于UMAP在特征被缩放到`[0,1]`区间时效果最好，我们将首先应用`MinMaxScaler`，然后使用`umap-learn`库中的UMAP实现来减少隐藏状态：" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:57:50.754429Z", "iopub.status.busy": "2025-05-30T08:57:50.754011Z", "iopub.status.idle": "2025-05-30T08:58:20.748955Z", "shell.execute_reply": "2025-05-30T08:58:20.748389Z", "shell.execute_reply.started": "2025-05-30T08:57:50.754408Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2025-05-30 16:57:55.675947: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", "2025-05-30 16:57:55.702242: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
XYlabel
04.2393176.3400990
1-3.1587995.5053840
25.3850682.8883323
3-2.4232523.7789362
4-3.3830343.6373443
\n", "
" ], "text/plain": [ " X Y label\n", "0 4.239317 6.340099 0\n", "1 -3.158799 5.505384 0\n", "2 5.385068 2.888332 3\n", "3 -2.423252 3.778936 2\n", "4 -3.383034 3.637344 3" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from umap import UMAP\n", "from sklearn.preprocessing import MinMaxScaler\n", "\n", "# Scale features to [0,1] range\n", "X_scaled = MinMaxScaler().fit_transform(X_train)\n", "# Initialize and fit UMAP\n", "mapper = UMAP(n_components=2, metric=\"cosine\").fit(X_scaled)\n", "# Create a DataFrame of 2D embeddings\n", "df_emb = pd.DataFrame(mapper.embedding_, columns=[\"X\", \"Y\"])\n", "df_emb[\"label\"] = y_train\n", "df_emb.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "结果是一个具有相同数量的训练样本的数组，但只有2个特征，而不是我们最初的768个！让我们进一步研究压缩数据，并分别绘制每个类别的点密度：" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:20.750353Z", "iopub.status.busy": "2025-05-30T08:58:20.749680Z", "iopub.status.idle": "2025-05-30T08:58:21.376660Z", "shell.execute_reply": "2025-05-30T08:58:21.376156Z", "shell.execute_reply.started": "2025-05-30T08:58:20.750329Z" } }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDkyLjQ4IDM0OC4zMjA2MjUgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic7X1Lkxy11m2P9StyCANX6/0Y4gtUhEefoSMYEAxOmLbBYXOucVyI+++/rUdWlezMtmyrq1ZmpU2DRa3aVkp7r/3QI2+/v//nzxf3P+2fDv/nZ3Z7bL14z8Twmn5eDXx4TT//DmLY088rxqn1lukgd9rTH9+Mf1Ta75TkVhr6f7xu/sHYS3b7HX39PX1lz9xODsLznZMkShi3C4f2m7GtpI5i3yTwoZEkvWPPh3fDiRBh8n+02jk7/H0//DL8Ndx+J+PfR88R+07PEnsuhifG7LhTgURaYwfBd5Ibp402Qg0v3g63/8OH7/97gNOP2HnPQxBOGdMZwXfBOW+0dfSX01ifNvd181WbOC1dEFJpkcUdm/u62Siuc++8D9IpReOexB2b+7rZIi4+4aiU55mtdoTdecND/j08EYrvbDi0u6uQtNbpw6AeWvuq1TZDNggVjBUu689Jc18328RprpxXwvKsPyfNfd1sE8e9CMo5YbO4k+a+bjb2jp7FGDM+7ElzXzc3dTwqGw82WOm4y9p2bO7rZtsUeH2qu8fWvmo16q5SXjhnpc66e2zu62ajOG649F6ZYgrH5r5utokTgXNPjsfLJO6kua+bbeIM98Ypcq8miTtp7uvmBYj+cqbQUdG9k0FwLUKe/pPmvm42DrAxjgelZWHpY3NfNxvtxhiKYJRS2QxPmvu62SbOkTobRSaSlfOkua+bjcRKxCmUtCUKOWnu62abOCmVDN6popwnzX3dbJwKohUvObchT8Wxua+bq9D1GFcpRcrKlSlR16G5r5uXiQk7Ri+dbZZi5WKX2SqOzX3dbHQo2kshpbTFoRyb+7rZaGSegmWnuc8UcNLc183LZA49Q336gvDCG7LxonTH5r5uPp7NttsbBQVElqFMzElzXzdb7U0qkYy02Nuhua+bjTbChSfN9TxT/UlzXzcbe9c3HuzsiKxznAthigWfNPd1s1GcIDtI3iaLOzb3dXOjvnPaW9Daqui28zgem/u6eRH/1nlivNVaCuVsnpiT5r5uNooTxjrug8ljd9Lc181GtaGkiJ6FIrysNsfmvm5eJKPrnbv2ZVKiIc2D1+VhT5r7utnqgDtGfVMWfN4aV2ej9EIRMVhR0qdjc183LxCd0N9PsVH+Tu7dsbmvm41GyY3L5YVslMfmvm5eZOy6RsWX90UdXTq2X6PYydNklzhtbOxPGxepjHS2xc4+o/fDdk41emrc51naF+U5zWbpfCIRbgsFHpv7unmZ5LZzgcYKqSlw00Upjs193bxI/tl9hbJj765lDaez+vasp25rzqtQsrPnnmufmYXN7obYJgkhrERDLGx2+0d/j12VuvSILVQV+lcbOpfE+u536LzNrWvBDn5HQSd965uEeMrfuVFeZmI5ae7r5kWW/rsVy4xTwmphXR61k+a+bmJrGpouaq+kM7bsKz1p7uvmRXSx+4q9kNxJWXj+0NpXrcVsy1ihOnZeYfEqFYFtKN7p2NzXzcv4zp69Q1YhMCXTmispg5Fl+9ixua+bF9m403vZpWfReVOyiyUVPV3XVhduR5CeqJ2hEcm/qS113M6jBbGEEv31pq/5d94GqISgT7guWxZOmvu6eYHeIWviB9Qj/O6A1pdVkM676jofWJAU73FHdpd576S5r5tN4qKM+MXWvzwGAYLrkHnypLmvm9ja2ah73RfjO+/l6pwa9KtLWC41BXkhizpp7utm49mGvudGux7l++x9LZdf/fgM5YfeGdC3ptz7rEHf4z3Wxh1J1ody1uDY3NfNRgPVTlGWY3jWwpPmvm5eZstSt8D+DNbZbk2dtzp13uzXMZnqffqkr2Uab7mXcqyZH1r7qnWRYlrnvXq9K4fdTuc3HCB4wI6gT+R0PcXV8zqEvnXaritP3ZdcOocG5L1sLoDmDP/Y3NfNs4bZX2FETWPYeeNl5+Cvo7vqfEytp3/pXfnqHEX2fNKeR2U+P29p9z6da9idU6zOS9edyxV9XW1X9et8SqLzOfieB1Y+NydyUgcvbcjEfWztq9aXmFLns1mdY4i++VXnLUSde9f7BGxPazp/lfiLVgAuVVLuHadhG2XnsbuWVd+OeUHvw5Od46llM89itQL2pPva5/CMs7whrmQK+vvZa7FoqInuHxt2Dua2w2SXVaH+5aeeix99j01I562V9JWsbCfNfd28wPrHlShb1wvWux+E6bwK0DtN6km9K9C37iFK5z2fn1s0PaeiX/4yz8UpW+9NAp23rnU+ntqXurYN7V9Abn2rYp1vTo3vS4pvVinX1p0093XzIpHeBRfLzoN4zsrLrOTwLA7Q8O8Qt/DXL+yqXqpVvTSL/fzBS7befvSSLYI8+FKu8nnBT36fx76lwYsv3Io9fMXepYd7Ep/OqJ2KO13MoFS8np4ejz29Y7c/ikGQkOHuZXq12N3v7Nfhm5v3N/+5+f3mr5t7+tP7b4ffhrtn7Ie7NBAiv9WLZrq8mYyPs+XiwtXxGZQUO3t8hNw89Ligp94sVgt66OViscDsrXTaiBSCaOJwF9+6ItjfeSTmEcNP+4c+/fuofk+M3XEdhOJSmRC1RHARrLCGFy0Rn9SjbgjqpqFB8y7+5TFp19oJZWiEIk0Z7miSjafuD7NIViHzMLTJbOsf8ZozwXGfxpRTEOecz7KoJYykqDn3bxrJKmSZpiaZTf3zgZ7HaOldWrgNnOxCCZnrM+mlPTLk/s0gWYX8aT+P/FBm4/jNqmS7OrN5dX5c/eyJ+PRZ2Y5mRaNFOYczadK4IoelyAsm2whWW+nDqBWTQFYBs1I0SWzrHHGio9yKGDByo/GBcy2zrGKeuvRuGskqZO5em8xew+u9NV5ykhktJAbbWnBZ6KZcrZ6fYBrJKmR+gjaZK2HV62CFbjYfE3WpuPU8jYp3lmvyGklVtPHW0oTIonGTQFYBs8I1SWzqHGUTVlM44aP1ERsIL6XKpijyIoEs+jaDZBUyda9RZlv/lM2n/KIoeXhjWBwDS2AfXXHu3iSQVcDcuyaJTZ0jXSZTEj7ZkqdG3H2V5oFE+uAF/SWpc9NAVgFT59okriO+IGqUlP9norTeWBNIRGIlKaQV0UllVplGsgqZGaZN5sZ0V8h03AStPMXr8evKkAekwEPJVAcJXnGvdNG3GSSrkKl7jTKb+ifocWx6J+IQE2Nug+RaRJcdq4yGxNpCdTNIViFT/xpltjFxrhUpihbp8ciyhHVSR8Feam0kRT0jFU8jWYXMXNwmc+mBMzbdbEHnGWkUPwPheYh0NEnhvVTk01N8pEWwNHE0hIUmJ4GsAmaWbJLYRpKSk2JpF+KDaiWV4FaHWI1xXChtrBqLOjNIViEzSbbJ7GTnFP36QCGU9ike1soF7pONcG7JgwRdTG0ayCrgT/tZ4IcS2zpHibeliC7xcVxgVFGETpEfzWIQY7Q9A2SnwNy5JoltBG9kuaEjPVswQrhYb0tlORPoafloONNIViGz4bTJbGMp4pX0rgKb6M86Q4SgkxESBUkKY3Tp3wySVcjMXm0yV8HylHmNWy3ig8c7xYz1YyTP40L8OL/TSFYh8/y2yVyIF2r2MdB2TOCUcssoijSHgvsoIvKtFjYEpW2Z5xkkq5CZw9tkriLQhS6JUIodV0KCjDbmPOVnXKZiLuE1GaAYC8TTQFYBs+Y1SWyr1+T8UQWXqZ9LS+4yreXQwxvHCZ/rNdNAdgrM9ZomiW30R/5Geu2TgdHn5NKtytlGeoO39WP+MglkFTCTX5PEVRSTgJM/8JAPfZn0sb1us0+FJg/JlSENJUWI/k+MW+iT/3PBKW9sKb3NIFmFTN1rlNnWv8NG6yhLCxPIKxuTB8FI4oAx9pxBsgqZ+9cms6l/FDnEwhhFh8m6yLyMJXqK80JMZGU49G8GySpk6l+jzFXEJJQpa3pQmdLmuMdVesF1Slqi+/acEoFiHdNIViGzebTJXEVu4a1QFOik0mwQpMxOaJfT0Lw3no/+axLIKmDuXZPETkUNFYg08w1G8ToZenxBgVmKIFVc2jdmdHEzSFYhs3q2yWxzIqm8o3hSdfJOlghCZNceAyNitbF/M0hWIbNzaZO5LCeHPlDYPN1kKtwKmgCVVnUpAPDEE4FHF0uDFHI1qpjKNJJVyGwqbTKb+mdKRTZ5fRcX8aNjTRHKuJZU4sEZJKuQqX+NMtuCafL5xscMIXmFQ7gbY99UAie3VqLpaSSrkDmcbpO5Dk8DvYUr1h9OVnkpZc2/imBhhLfalfmdRrIKmee3TeYqIp2LriMC5JLt63Dgywno64TQm5+gC+DoFGLHaUhPx6MgkwthsejkrTKjC5tBsgqZh69NZhvNcCcEqUSKJ7Wmf+jBZaGHRAjjOtskkFXATDJNEpdfxkV3rpdfYkNf4G0vVwor0l2YKaZVKlcXE5vSJyau2xZFnEGyCplJpk3m0ise8dl0qS0mm4tHVb3X2easDRTphtH3TiNZhczda5O5giRoDWkIMhOdF9FeOwKI/hedmYOTDvoacBCHt24ki/c6WKeyK1XSxiWr0r8ZJKuQWf/aZG6UgnooYfGhCPauvU3vF6v3G2KbpBUjoCaJL+C0I3LwvrmZBas+dHgFn3QhD54LSmtPCZHNqXBagkiitKSkmMy7VJOngawC5gpjk8QVMNpa+Oq8iRr6kqjXJnCjpUh2llcMZa5shFwgL/YwCWQVMNtDk8RVbOwhVhwPg6YXIpazoYkH8vFRXkZvBskqZO5fm8wV1NzXwidojCOEsJqi6rykIAJF2yF5Th1b1o/rpdNAVgHLIkWLxDaNRD5rZeTJqW+ngggyXgSUAxCjKdQYA6sZJKuQ2ZzbZK7Cl0Bvcti45hG4xvDDYMT7WeICplApTPfGaW9MvAo3W8w0klXIbDFtMlexeQR9WxV4tnQdFg1m80oeXkwXvS+l397l26Q8p+iP63gRST4NMI1kFTKfBmiT2Wbz4FsB4Xc55ZMvFNnFr0snpQs+5IhA5O2uY/1mGskqZNlz0CRzM3tcs4evTWEHn+CDt9lUM4IMTe1MJLH0m9pSx3cu6biTSonrW4YD2GL8oD8DXy4BJ67rIIZPvw9nHWuHKmaoPJj0dQpbvXZEXemYhsiXUYkxeJ9GsgqZg/c2mW3BO/IhKPjIeNtNuiQy4fhXF8hQ1p+iIXnKeMl+dPJ5+pCw56sLppGsQuarC9pkNvUP/Qoh9IsTiKclBYJ57cSbTI2pMkfRoJck2JfQcgbJKmQuRbTJXAehQW90XgvhNtMpejUdna5OL72WQRGf2JCSGuPob6AczJfj8dNAVgE/ukZ7XmKbL4K+Rgd/qQk9DUY/6nbJS0rQ5+7aXwTR7KHQ18tjnSi+cNamO6mtCSa+j1mksE7xfINr8VDTSFYhs4dqk9nUP5Eup5EmiJRJcB1L3TmTyFt9xgtcZoDsFJhv4W6SuI5on5JCCnZVvoGX8kByeDIXVY/UmMOjaSSrkDk8apO5DpYDz9bRFwrQ5xd+e92SPB30jf/obg59iw0xQona4rPGNyMJZ3XeUB9veI87hfPwTQPZKTCPXpPEdeziBadB4PXIpigH+UALepHgzJc6P1DKA49V0Ut50hpHkZpUudhGaWF0SUlDHMFJdiiJ5gySVcjcvzaZbR4O/PZ19JNT0O9ngb+mFnwdX9vyK8U/3NPExDA8rpBS8kUsMNbhp4GsAuZCd5PEVQweQhbc7ObQU07oYEr7eBhW0LeiXEEMEV9/lbTaKAox7eH1RjNIViGzobTJbBs86NumoR0I+FoQuoOD3kW3nSmbi9k9J7cc39QX42vvKJ0ndU1ERArqU224xOzTSFYhc8zeJvMLPBe0CTS5D/DNbZ5Lxa3nSdv8SY4ftKFMgpx58b3TQFYB8/g2SVw+w0DfZ48e8iGsMjzM0NtRKljXuprEBz0CRa+PAi8icPCFfGySOS+i47Es6JuH0M9QIhTVNnvAsynkItIWKGLo28K0ekNcyRQsIdXdGOLsqrWAYAz6VOTyE7DrMbyzmhX6vkf0y5yh7xVBv9QOfa/AOhinY2qJXnEgaymmHSdfByPJXBIL2TTxakwup4GsAmauaJLY1jnwe/Oh6wZlXVSF9Gwms45KCb8S1rgDU8wA2SkwE1mTxBXEfhuPLW4hCXivaROTgB+ogL9hZxX2elaLprHnOkXO0Ssa5ShOzbuUnLXlnEbZpjWNZBUyb9Nqk9nm/MHvGNi2Dn3x1iHkDHBjks/PcZA3SWBbgo6pdr6Bn0zccqt43pZk4l5X6cPYuUkgq4C5c00SG0M+8BMyWw5xUSZ5zm6/k8Or94McnsWpHf4l8H64/f7+nz9f3P+0fzq8eM+Ei+HmIHz0t8Pbsa2k3lEq+4axn5mSYmePiNw8BXwoo/rCKaJ8aUoGj30cXtEDvaaf2NNX7F16yCfxKaUyO4pAnRmUihUxekz29I7d/igGQUKGu5csDsfd7+zX4Zub1zf/vfn/3w6/DXfP2A93aSBEHAj6F4Fe008eCmozpfTOnDweRRk76Y+9L+1DXwv+0P6DsZfU0+fDu6EWJUz+j1YUXA1/3w+/DH8NZUaSUp0WOsJ4MYFhf4+DMAuJKvjQxyc6+MTYnYsXWphAwYngO+FciFsUhc+aIj+pS90QsfyliYop0kxpmrVaxfpO2takyHaV0WNkMYNkFTJbYpvMtid4YEg/OSGsaUIeHp5024NU+Q5IEy8HFDZt8U83P3Aux1rkNJJVyDI8TTIb++fLJRQpB1SCS2tkHnfPDXc6uLF/k0hWIUv/mmTiT995EZ9+JUtXs0XXy3GuotVz4z25CxkViiIW6zhFqgdamUSyCjn2r0XmCsfvJO4hQcIGY8YFyWkgq4ATozcncfMZV0o66K4kWhQpbNx2Has79OdyIyI1+PFyxJKWTiJZhSx5aZPMtVDeUZaJ18sIztN5S1k2Uigx0b8jklXIj/s3L3MNlHciyibFsCatEihrVHBBqqnJPSJZhfy4e/MyN8p7RMrbVO4LVQ5+8JBDlG3wvnLwkCOVFZFxI9Wiz0ikOhG88ia9xFeOd57GgjUXIgSjvB2PAE4jWYUsS49NMts02gvuo7woy4iQLwCMDfqzttKIw77zaSSrkHn82mSuL7aF85UrcAjI5g1QJEb2lyD+qNmXwHO18UF5b9OWTm2U9C6Sa+RVKVwIih9Ojk0C2SmwHBxrkfgYRINeBoCj8s0Vfv34WcWDILOLguN1kBGWXxmbdf/g6CaB7BRYetcicQWOLpJv3ETDdZoIoowgnVJpfypPVsaNLRv8ZpCsQpbziU0yV+hpgJk8nh/j5JtMDtg9TYX0KpkcF8rr5MLKVs5JIKuA5URai8TNiZzbiUCOHzRJL2ntFjxCgEs1kTkZm1bO4n4/oySITCHIWub9mK1GJZCeSzKpdNwy35DLD5vDZ5CsQpYLEptkrmDw8HcJgDsv5FrWEiIneOcPrH8ApdzLFwA+w8XC2yq2rq3BmoFDeejpR4/m0SNoaO4BqUQ+NIA6XyAsTPwLxPje0xh08OMLUfP1C9NIViHz/QttMj/f08CnJEuiGjieht98gkw16MoHT4XnRayhfrUOo0Ee3s1iQPfsr6NEdPkaw1dO6mYfwNoPrl2b7qxY/zbENklbjAcyA6tVPmjVQq/2QFe9r8ouoayOY+/rwFZbfEICrlBuOcsaGWcBJgt9mGJJfIwV41yVRaPZ/GZTa41xNrNCNrwF7NBYzk46ONuDzi82s79kjA2rFkuw+S3tBDGaHgiyNbUz8Q0J6Te1pd45w3V8S5YSj5HfgldV0LcAoe/TQKavq6KHT79555pyWNzSC7rRLCHO32ISJEpBD7HhuQo8BgEfvwUcZwenU1hfeZ1cCjsd6FywWeOFFw3RY9ur5BPk5Tx03w1/0S40my2fMBYQnUMnw0sYP9jU9bOpnuKm8isyix6pJTGfN8oI5wvXzyBZhUxP0yizaawR7nNYcrUdfgMD/C0JsLH1IohouZnhAiKls99LusiUYfFmDs/h0D5w4UTKF7AhAth24HUTicPBrQA86UcOFhbGInCGuhUl+hkvpHEgT+6FlyAwikRfthi6Ecn69m7gZ0PLcbSQXIy+OgHraOHXqjHY+TNcCTJVXzwsWE7iD5cvAswdMsvCE1lPRLdTGOixx1XNKgqiq3Zt7hBkVlEQZzxBtiG2KQBFLGwKNiYHmISFahd6YrWE+hZyjM7Bd/KshRjOe/QeepPlKoxmI0X0rYxrtGuAYvqirRZ8qRN7/FZktWe069blvmEWeOXb7JAzV3STRXcZG6Wscd1nEXt2kFeykSlvM+qEeM5uv5PDq/eDHJ7FIR3+JfB+uP3+/p8/X9z/tH86vHjPlNI7CgWFp2GWw9uxraTeaT+8Yexnpr3ZSX+ElPYp5EMp9VdOIeVbk1J47OcQh/81/cTevmLv0oM+iU+qgt6RgXkaZaV4/NaLt+zpHbv9UQyCpAx3L1kck7vf2a/DNzdvbv5788/N/bfDb8PdM/bDXRoPEceD/kWw1/STR4TazO3kEH/eMmEcTWhqvBkbwsQZp3b8v4fGH4y9pA4+H94N49fjZ/TEQiuysOHv++GX4a+hTMPHWpUO+WrHbVa61xN6eYRExXtQwkElnhizM5xr7Snoin2KFk7hv41VraQf6pMa1A0RDVuRAxbU1aiFynqyZS5NJiFDhMn5wb9PIlmFLGTRJLOxf5F0JOdxRH0oW0TiHmDSNiOlDuLgqSaRrEKO4W+LzMb+OUO6z10OpWNQ7TNHEtQ77g+OYALHKlzpW4O8xp4Vh6yTLtITJp8cGyJwJWh+Dm5gEskqZOldk8w23XzY3h62tY9IftLWHtd42hGfvou8q0Wb/MsmtSFG9hRA5OSOO+EDpXvjvE8iWYUs894ks61/XjutlQxxjtN/OWlNbCjuKBOVh9BzGshOgaW03CKxqXNejPvQ6NtCWCe1UimIFVJpQXYpSoVnBskqZA4822Q29U/niygonongHU+XENH/KjM6/3G+yOLwMfvo2yvQrY1TvoRTcu5BQy1tmgHSipQPxeGi/6ckt9qc1HAmkKxCHvKZBpmNjJELQDp+3XJD7psiimhAFKdS+iHsoew6jWQVsnBGk8xVxFD0ZRc5yKdk156WmCnSIQo6ZPszSFYh8w1jbTLbxs8QgTvDkyx6Zk3JpkxpupSG5obmY3QJ00hWIcc15BaZbeNniI+UtklXuD2wAjW8Lr/K+E0jWYXM49cmc2O9L2I97Lh8y2i+dOS4Ii8iZKo7OSXIeqyziY20kIKe8nAGYRLJKmTpXZPMRj91YBZyI8fHIwLSwkfalqMfmEayCln8VJPMtv65IIhxvEyhMcVV2gWuo8pI5+ifwwmOaRw7weW+tchr65n2Qisu0xpUvE9TRC+X4mVSH/LMIowefhrJKmTuXZtMfIbFNsmtyNDTaXnNKRhIcTJFVGWXVOJ0Ulcd5GgF00B2ChyPWDVIxB9K9NTzg4SCnisxs8jZheDkR8Khf9NIViE/TlLmZa6g7OIpZ3BB2zj2wjnOvdfpJKCIKhqTx8N63ySSVciy3tcks6l/zhjhBE8kyf24dS2K4kYI8sSusNw0kFXAfJ1Fk8TFe4ct1P2SkVue3wKPAlrY21rjg0vL7xT4k2vgPIbZ8Sp1YjLl7IG9p5CsQo7s3SLzC0pgThgiCJFC7ZgeKBobPVkCOyBZhZwogc3KbOyfU9LR80V1p2xFGpowG6dOkye1UY0O7mUSySpk6V+TzLb+WXIH8eb7tN3Y6OAEZXopM4gbzwXZ7Wjr00hWIXP/2mSuIAFaAJOjR1/o47cIBm9f80Bwn4vWV6I2Uqm0IGIcd1wZl6hD0zymuProbiaArAKOvWuQuI7BQ3c2leMnx6qkFPlclszFAu2ngokDklXIiWBiVuZK1iOtJurVicsMF55bcqiReCmiU8J6Lg76N4lkFbKMX5PMtmyVKDcZVRx+ipSOXoKHuEHX2XE9cgbJKmTOV9tkrsAZnsV7nHlHEGxZEV0ZFlF3hPbU2Gk1+M4N7MmFXo69cBIAwLqr2Cuxea6vqImAb0dAHz9ofsM2W3C3CjpySyhcARhFu2ODL3QgMyBABAVeB4JODTcFfHQ67Inoep4DeVb5QhabYPVyG8DrNOwr0YxNdxarfxtim6QVI6AmCbeKAV223WI35OG5cnuGtRr0nLLpCcBPmQNP/4pYYbuKZ7Y+vp0q+OxTBeiUAr0RdUWsAsc7l1+kWzTvwJ/Gwcz8Npt/vCUd2OCQt56d+NBIz3j4Cj4MQ97libwIf1V8ghZlQEeX6Jr7YcSCeK3hVqkEsWs0y0d3qNDR0mZWKEbTA0G2pnaGVCL/prbUO2e4FkqSijyC7YHvraYsMt9rHZ02+UdtohuKea7lNCTByLH2MI1kFTL3r01mo9fFPryLfWTuqujh0y9cWo/VQi/v4bMK/JF78PHDjihXxHqNnIZbY15AuRE9r9+ugf7SdRd0FoU+8XthFkUvCII4mc9wEMh1DPgiEHg+AL0kBb3tZuHDe3mWhi7QXH54LhqXdy2JgJcc0Dcfom/yQl8ehuaZBYQI0PEV+s0f0DWMszi59SQa8I4CmUg+35q3/cKfv18Ym6qBFzw5eFENPY5BciX46dVyaZyDV1cXkC9tYcRXliyxeRp+iwD0suXy/Ah40oJuzejVQXRvAu+N0ecX3D62AhJ6xeAzClzIm9+W4MvAuQ5AGVfDdXCRKXp5Ddp4L+wolpJXticm17w54eGeoftYzJGDN1A0RNcaArLObmSDOTDLQ1zB8s2mE0vWqw2xTcFiEWecAvRsE9c/LKCIBV+why3CrMj5b2SyqLokOKXAb9K+fOVvY5U11UyiScc9WUKatElHBy9t8Om4N3eOHtd5W3a7ziBZhUz9a5S5EpvE5lznHQ19sDJvwpLChaDS9jsh6GEVd6bsC5lBsgqZd3m1yVyBT904bYmchr2+3fYE4Htvt90+qyuncPwTskvaFwJnFej512bXj2bX4HqJH6Oi760Gj8egefsqWOU5u/1ODq/eD3J4Fgds+JfA++H2+/t//nxx/9P+6fDiPXM7OcSft+lPwsQnG94w9jMTxu3C+GFpnH5+/OYJ8vBJQU5+k8f+DHGYX9NP7NUr9i490JPEgmJnKSGnYJW+ZeO3XrxlT+/Y7Y9iEHqnh7uXLD773e/s1+Gbm//c/HXz6ub+5u9vh9+Gu2fsh7v05CI+Of2LgK/pJz87tZlwNBa+dF1J+stKz/OfDx0tuEP7D8ZeUjefD++GExHxY0f/0WoXj/PeD78Mfw1l3D9gzJD1Raaj1tGHGuttvqwvjsQseqjQHzAxm0eeqOITYyj8N1pzMm4bNYZrL7WWFJBljdGf1KluCOqkc1yRBaXT0MHF262oNyo9mJHWSa7HoGQaySpkuQGySWZT/4hYyKEpzxMRUov4j6ejx0GFmAcGnbs3A2SnwNS7Noltg2fyRZfJi1Qn9EmQJhdI0svgTSNZhSw3jzXJXMXkNlnZx5HRnPV+HBk12OPjGlg74tNv7Oho9d7F5cqYLEQbcCoe6U/eU1E2obQUpag7iWMVLttUi7ymnqVStQgqXdVAAaThQukcBdE8UphmTAmmZpCsQuaCeJvMpv4pClzTlQcxhdbFOHPcLUm/glEllpoGsgqYetcmsalzLkU+xvsc1QbudbCJ2g5GUQZvBskqZI7k22QuXO02JnoEJorhmHbBk0+NxharFFqZ4JPlUQREIVAIZcpnkKxClkPuTTLbpp30JBin8lIFN06roJNg8rxKOorQRgc+jWQVMk97m8ym/o01OpEq3Twf4ecxGCBo3hBTSukzSFYhU/8aZa5iftHHz+Z1Wpm+bkkxDNlYqpl4EZfRlODj8u80klXI7O3aZG60eDlaRM8bvDz5upeBfClZYJoiPt4rnB31NJCdArOnbpK4/IwQPXzFjhCbbMca4ozAbdJzywunJT2n8XBGu7GaOY1kFbKskjXJXIVtr4XSG+kYXWHWEKVCMzJ6CEjD7nhwsrwgJ8qRqWElF9ryeGllLihMAlkFzAWFJolLDwRWUf/cfEUHX9HuCaB5dNkFNM+l4tbztPfE85SmO5uMPL6ySpBVlEx2BskqZO5dm8y2/hlyHOSj85MaTaQYQnbf4w6a0UlNI1mFzP1rk9mWOGgn6al4VKO4J1pYT8LzIDjrhbUlSJkGsgqY04YmiVcRoqxBgdfGxZtr/jrXDG1wG99+Bd/icxF0woduGeg1f1mKeZTCEninAxd5h13u1PzHqSfHj9lH326bPhkETb7X2a4l9TxXFoOxpAtW+fH1D5NAVgHz5DVJXIwTPutGEejZQPcx6DSOPbnggwftA0lvpVFOp2VzejKKOa3gaXt2jFJojnThqRkkq5C56Nkms80HCuesVGHc5yTie2RCdK7OK22kEb746Bkkq5DZB7bJXAzPX02y1TQh0Is4yIU59EQWmkehPST6zAZ9oN049JZ7K2TW4ngskdhQlKmdQbIKmfvXJnMdq0PQTghD/5pdIHo4iTCYmwN+FAeMTjPoNI1uGdA0fV5Ezz2y6GYDrZab4iIizlu6BffYm/rhIc6ooBtim6QVI7ZJWgACapLQQ0r8iMIZbpVW9FxDbGhBKJNOTXkbM1Kv5HgadRrJKmQ+3NQms61/4Kdl0ef3eiJGMGICLq6hb/ZC3maIvkvgeuwdihHwAxH0U9boi03Ya+7ohd+NlS7DSsCHUfFNHvviB/DJ3UwelhQ0xemcQkyd3qSgiMuDUyn4tCEoCjn4GArMIFmFTFPTKHMVCQr8HvqNFzZe+JIUBjyKBC6pgCd/m9H1RZAlqp0J0uXf1JY6lr21UDJWaq+tuADtcbCXEDbz/SqPOPGSmfZhBd/FiR5naqNlehVrelFWSC/JlKluTdG/oehflXBhBshOgTmLaJK4ikI9dnFhY6Uv4Bz8CBr6bBZ8ORA6joj3RuR3cKlUgiEeIEoy6SV3ToT89s9ya+MMklXITMhtMlfAePhbYbD5dgHkBz1+n+9r0HceoeesyAu6JojyQvAYaTonrLQy+ERyQpFvFePkziBZhczRa5vMVXAl+vhBKx82UeIHOtjjt7KkpmlCwF0l9o5HcCpFT8uAV63AzQLbSYIXjxEmdyUuBr80hb5SsK0Pfk1dD3mZBZ0GoX0IdNxHT6lSTSVaWHxEmgRt0tt7lHVcSOvHt+xMI1mFTN1rlLmYLGslDq5puKE3ZS47EEcPL+BLueDjp4Wyxsrk/yOlpHxXJYp2ynGp5bjfehLIKmD24E0SV2DYG8+fmefRzwluWxi+gojQ36MBPbnYeeJSqBS6TNDO8+AREXpajr5+jJsroM8seikSmwPPgwBwtCuJqDnEOsui7RHdk0LzxQLsA3r8zovoeXZpM5vNbK7EbKBMb0NsU7BYxMKmYKPpddE0lPrhp25LV67NfHHM94ymh29YF19neLBz0DvWwBcQoWd2Y5svKXSAc4mxlvtAEuOqkzFSeGclj7KcCUp5sq/i4maQrELm4zFtMlexnwO7Pr0FOddKO+D1VfQFdPhTg+CXkaAv818PqZyVduDja+TkBN6mwXdaQ0/uRjiPE+dceiPQkoMI9O3e8IQDnfttnBMRz9ntd3J49X6Qw7M4osO/BN4Pt9/f//Pni/uf9k+HF++ZcJHeB7eTw9uxIUyMK4c3jP3MlBQ7Wz7Ofz79tPr2EXr6WYFPfZvHfg2v6CFe00/s3Sv2Lj3Yk/hkFN7QzDtD37HxOy/esqd37PZHMQi908PdSxZH4O539uvwzc3Lm/ub/9z8/e3w23D3jP1wl55exKenfxHsNf3k56c2U0rvzPhY2pud9KXjpXHoZkEe2n8w9pI6+Xx4N5wIiR87+o9WO0c2fj/8Mvw1lNGPX4yqoY2olCiF+dxlPYq6GIdhDjxU4FIKK8CP1P1U6kFPnhizo0yVFNVQXDQItePKOUmEQFqblMZ8Uq26ISIZKWkpAgu5kGcpUhMiXeE0FqjE6L6mkaxCFoJrktnrCRomqlQ3Pz399Yw+KHEFwws9eOdFfPo1IR2NzscB4oFikxTHS4riuQgpwufZm431zhkkq5BllaVJZlP/aMbSl3mMAGz+I80dNdTR3ab+zSBZhUz9a5TZ0r8ozZZfMQkXKbuMipY69cDHsSfsoW83DY8ZVTnZnMrDrkKKhkeNH4PqSSSrkCWobpK5dKPeGBGEEbvx3QLMAVzjwMfv0myLPn3AhLEN3vrYFn1GwdkMfvxIc/OCSXw8R6J8MD4tPOm01ijHVekZIDsFlrvqWiSuITHY8eOKUryQOiqJFOl2NFnnVTNAdgpMvWuTuIqsb+PitXGx8FLbtP0ykagi/bDp9cKKW2fiYmJRuGkgq4Cpd20SmzrnyIDyvrwhrWxaIax3mQSccJ5zX7hkBskqZLkMpknm0s0Bf/DgowDkyUWnFeDB6+4Gttn4KlWG5wHoyV3DCMNHvdAagO/p0ccPmB4fISOBrh6gUwE6lUJPLvrgofMUOs9jz+/izefibuqsG2CWr0zovgy69Ic9eIvnEnRXcnGu61gdWr4qY3Mx+JoeelSJrZ7QVABO8tvMPuIiAcbwti9RLHusseOVNbiBFYzwklW8J+Jqtq8sXCvQhxd68JaHOGP57Dpc1qad16ud14nYJmkBiNVN0uYJVuQJVqidCIWYh/oHXSTcTGeZhoF/0grbaWy2gWkb6HqDvxqKPn7o7hp9/DZW2lhpaes3+OMHz0qb1W+88AV6jZ4lbNHUFg1svLBlMUsbv83uOiLIGBVpj3T5N7Wljm/K0TRh8Z1h11YxRnfam3Guxjg//ZqL1bg86HUYdJNHry+gxzPoLudKKLMb3bWMKfQJS+hDU5s1r/ueio1tPje42vzv115RTmzknOQiRUPCeGFDUDpHpA98XK4on//2CoZn+dH96kkHYwIaCQ3fQSIM5nL5At8doc8vdC0CfnJxiRxf87bB6+XjNjNZ8bW/6AEMvPIBWPI2fte7Xw46wLo4M6/Hx4GbCfr4oZsx+vyihzHo+oe+7A29SoY+uUtxc2swdOx4B5rF1zD92CO8qedWsoMo2UGPJHosjh5uQHMM+uRC71VD1zxgWukT/iBo7xpIfikO9Yo32qAh2tV+4XqzaRYcomP9BMGDbJq1IMRZT2pgc+emnevVvWvQ303DN93buPPimgWle/B6Q9+RtvyKq7OCmzLVQzkxNPdxOTE0/+1VBMzoK2JbzRCClrYD10tYnhgufDwS3VjRyXgjk7VdVoOucchsBk4nm0FfxuTRI1Zsk18DqWLzwmb1j2H18HMObjPYrLSA+d1s+lOI5+z2Ozm8ej/I4Vkc8OFfAu+H2+/v//nzxf1P+6fDi/dMKb0zdnA7ObwdG8LEhx7eMPYz097sKELMn5fG6efV90/Apx8W/OT3eezb8Ioe5DX9xB6+Yu/Swz2JT6c8QQNprzT0NRu/9uIte3rHbn8Ug9A7Pdy9jHn2cPc7+3X45ub9zf+7+fvm/9LPn/Tn+2+H34a7Z+yHOxoM9r8X9VGZCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTE2MzkKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMjMgMCBvYmoKPDwgL0xlbmd0aDEgODM5MiAvTGVuZ3RoIDU3MDAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic1Vl7fBXVnf+d+c3Mnfueubk3D/K6SciTR+INAQJBLgjhKQYISFBsLnkQXkkgYMWAoVoSo1hSgSCUYqqovEoDIiQQKNRUdBGtVdx2tesrlrINSHfxFcPJ/mbuDYLd7qfb/WM/eydnzmPO+T2+v9/5nd9MgAGAi24ieCdNmJgPg+gCNphGYyYV3DU7+cCQudSfTOXhSbPnjN86b896AOF1en7tznGFk81js38IgKep/8VdszN9S0ZVUV9so/7ckuWBaiHJshdAUqnfVXL/Ki8sjs2l/jWix8urFy1fMez+JQAm6sP+RYGaajDRBUoL9W2Llq0pT9satpH6BwCcxRVlgVJlwa/qASJt9Hx4BQ3YnzZtoP4E6g+sWL7qgVJZOUz9CuovXFZVEkgZlPEE9XV5hy0PPFAt1ssrAKKoC97KwPKy1MtjOqjvJXneqa6qWXXZ9tFSgAG6PMuqV5ZVjzb9hZrRxSRzBehY6Zz1n0AXQhiNuSGS2vozCwyFPBAm5E8vBMeywKpKeiLqk/v6AG609JlsadnKSlD0llFEoqDXCtUfGzOvMTch4YR/+Nf3/j++tp9C39v9dPT2/56i8WOkl0zFBnbSz2VgaKI2uzFDRzaEF0g0V8dHpbs+Lhoj5DKElRl0zK0GJQdAyAa74AC4DRs8GFgZWAhNgZXLK6Fp4crAYmgqCVTW0L2ibCXd16xcBk2LyqqovWhl2VJoqghU0pyKsoU0sjRQGYCmZYEqr34nW/5weWBVBTRVLtVHqhYFlkPTytWVNHNVeeUiulfo9P+GvQ29ly1eFLjF5iIEbc5guFFLpIkbYiAR0shzkHS0QoRRmyGctLeCh+5mmiPQGDNwC+r8IfwJCiDPBqY/GAA+TOyLCbwLQdZ6++Zff59VhNpz/g6zEcLC/X+HfecH5+r1jfZNNG4Zn39Tf+W3bWEkwDebqJ13Y2kaaWkJEgneRQdrIsRAypa2UzcuWOM/Q7ngIgpWGVERBUG8sSL0KyifWAp+CMAa2c3dbIdpOfvkljkYKjEABsdV1GNGX4R5VLvIF5EsEIBFsASWQTXUwP2wxrB5AEphMY1VwkpYrY/1Pdv3s76Wvqf7dvX9tG9n345bZQn9gjGlOtRTDDrBoktfSmUR6DsBiDYQTzA8f1moWKlUhtbrlHQca6josWM1Fd1mGpU1oRLGcqANztF1BvbBTvY89cppfAWNtAiHYQOtaoOX2TnWKAyhsefhKrxNMxvgHO4TgU2FbBoF+L0kUJQqhCNEI5e5Wa5JJpeeIR4RZ4lt4kXxPIwQa8TzYrFYw7LxGWmu9DyVXPw12ec1iIc29gHJeRwvYTZ2iBNEB3yA53EffEpcdL3PwSbYDbUki5tVQZ1QK8yikbPSedhBVxU9P892sbdJuuPsEbgAT6EoTIZd7ALpdQ6+gEewUKgjU2YL5ST/WaJ1ntbvgBoKIReYBbgwiMZIeuK10LjH4hDpgnFdhTriXAi75TbZbUoiLjpiz7OXWbe8GVrgbbwXV+B7bIOYJO4RJ8OmIAJYDJuI9g59jVzO1pDu+lWrUxe+LxazfXBJLDYtJNq/1jUinkeEWaRROXRQ+b6skk6j2QZsJEn1p7Fw3jRVzKT1RMG0jrQGqMIc8oEqen4QDsMQbIZNRMnQVx4hfUErd4ofkc6b2BPCF3AeJ0A6lItXdJ9wAzQDHDPJkogCg8FetVVInlLa6p85z/tqUcKQwd/pelWTtxUKWu1rvG19fQXzxGipqFWKacVkpVVMTvrobz38aMjgaQXzvK3XJ04IUZ1YPIHGZs+jpt6jYRqfOMF4pjNtlZLpb0pxq7ekwvuY+ljSqMfUslFD9P0g6Kci7S59z9X2fSIOITwtkAwd/tSoeGuE2QF7I+R2h+atjz8e057Upm2MsEEERtrNijUeFffEFLW3+/V3un0+LTf3Nsjs7LrWe61bfeWKekXL1XJduVn+yqzYrLis+CxvVkJW4thUf6w/zh/v9/oT/IkFsQVxBfEF3oKEgsSC1OrUDbENcQ3xDd6GhA2JTaktqVdT4/qX9i/qX1AcVxxf7C1OqI6rjq/2Viesj1sfv967PiFyAVvAEmWPOzzbN3wMG6El5ThYUmJKzrDh2Qk5w1KSEmVTzu0s2xcunPrgwA+qtre3tY3tePTAuevfMOGFbcVHC8tOzf+Pq0J2ee3Cmt8fSZ9+/Qf7ygNnnjl52lX3+NCh+1JTe3XUVhBW82U3RasYGOmPGtAODne7pGx0tLFtGCGCIkzSXNaJsYTONQKHkOnuutbZrXZeyTpaHLc+riUOSU4tOySeoKlAIjFD6qCU+Exb26hDa8/1Qd+5tYeun33hySf37HnyyRfwqHDf1917SgNsAlPomhDgnnMXL56jEpKrjmzohmio9g8EDzPXK49Knr1MarexE5Htrjbbxphoj6B4FJgmuJwTYwwROzUXmU/tutbdpZLt1GtXNN126WNjq2NbYn8TezVWGgtj2VhhrGdstDTYlKlkmgdbqqCKVQlVnqpo84IVpI8nIY6QHT7CQzp5wdAJTEOZjrlY13vYdv7YkrMLS36zlF/jZ1l678fM1CY89+iOdodw3/xTZ4cNO5gxmI1kFhbG7uB/6Nx25OAu/ezIJMC/IqzDoMgfI6nMpuyVWQNsc8gdFiGMclqzpNid1ulutXdaq6VwXjtFbf/IommtDqPdd3pkUV5nb15np8tw0S5fb7d6xUedLHbU7ynwtHiQRCchY1mCJ0FLIptk6+4ifNVacifL5G+1t7YePCm7txdUlGzqzcS3Ns04sV/Hms8V5xPWVjo/p/qTomyxZld9WHi7E9tTktpSO8ztzpMDYlOiQLFNkl0u78R0tbez3x06u4IOwS/oSOeSV2Ssz2jJ0L3CMH/QgSNUISExJTXHgHUMC7mKi1wlIofC/nNbtzz33Jatz7Vx3hM4MHPmrlkvHck9vPaN3t431h7ObRPGvPr++6+eff/9P/OP+aXYuBcHZ5z85T0lC9kohkxkoxaW7NN9+QyBTOe1kQsN8jvkU+Ih6BAkpoiQr6i9ed26vF293Vl+q2r2mwvMxeZqs8QWhJH3atmepDNt9BOLv2mR3ZeIXt97fK5Bz0qn5AR/jFUwgeOUzdQgnYQO2yFVUSX5LjtTbJCvGtS7cg3fIzzILgYYxEjzawVasVatBRm55UEsR98bQYbPvpR/2+LpBteN757eGdgup10iT/lWk8RjsE1gCuSL5N767svy21XJLxVIxVK1dFWSg+KT6LL7627dy45TQlBK1gyD4f4oNAM6mNzg0NpsHRYmKDBDj3j55GLdvms6wTzdeporIjfrSLHnTY+g7+YkLWg6AxYjxoilbWvXbj3Q3j7+xdVnXhF2X79X2PX0rlO7rzeIxQfLSj8L7djVhhdFkBeFye0uaLe1uTZGml3OmejyTIw0Nmhwc6pXsvxJY6NqoVauM9UpdeY6S5211lZnr3PUOevUOq3W1RJ1NUq7KQZSNEn16XsxKVG/CzVbDuzfuvnAgc1XmYtfufoX/hnT8IOLr7128U+vnr20k7/Ku/ll2p65tAvdbCRJeJz8fDdJqMe62/3R/bGuzbGRncSOWIpzk4yIl69HO58vKGtXf7jzm4Px7sM4kS1IvgENySJQWL7Z2VlNe/uoQ7WvU17/eu0hYSRFvBf0suf6QdmyrzTAO/hXdHUE2J/7A17QbjiVpNMgy++WrSbQrNjgaDN3mCyyAkq+S990hgdTlHvndT2sHSkIezpMt5gRmm4yVwROjZ8yeOcLJMfxDWFDY/CISzt36vphMlZ5iSQRt6q+T/AscUuFi/48u01wWGfHxylmwWSZHR8fN95ijYsXPVDPGkV3vacxsl0T25Pp0EyLs1jjo00wK1pxmBR34sQ0Xap3urt0d88NRV6Vf35F/fyK7lN0mJtUx2UtItdk3IsSD0MqYwv8y2MsMdYY21AKvoOtg22jzaMto62jbVYveNlAIc2SZs0Iy3RnejLC0+LS4tO96QkDU+st9dZ6W73dpWfigiBbZCva0I4OdKKKUTgAozFGjDWnZqaPTf9eel36+vSm9Jb0q+mRC4DCuieIkjs8nsUxj1tOuhGQyLUyCUM9VvrC8fEZe+Y3Ni7cMrbzuS9/N//lZeWvBB7eWLbfv/+pD98oPyKOPZiWVljon5LgyNjeuPNoUtKpnJyimdMKkp0Dtz6860CcHokO0l6YZ+xBN4z2x3y7CzdaWIe7zUZ70G2dQbsx36NvitygXbt8N7Zilee0vhXD6GANOv+NEzaFHdS34s/b2u44tPrMq+xNdlx4/nrg6adP7RZqv2k5UF5yFffo/jSG4kAdvbfJ8I0/FTVREgWNCZJeoSCDzCjdl8cLCL+UZIkSPUkEk/pO6OCB4MHjLpzW6im8Rx/QT5/OCP3c6faF7CpdNqlKqEiXixKZ/0eThSWUgtcJ9cJ64cfCbkHRGZnRTL7kYQNwgJgCKSwd00WvkgM5bBSOErOUfMhnU3CKmC9Nlv3KXJjLirBILFDKoZwtxsXiIqlCLlZWwypWi7XiaulBeQNsYI3YKDZK9XIzNLNtwg58SnxK2ibvkV6QW5XTygdKn3I7mT0s28yyWdKYl9l97L6X+b09YnFvIR74pkVHiOKBjpCTPe6/w6QIZg2cFs1Kr3NOh+YEp12z2UGvHHaL1WLTrFbLeLvVrIJVasCTDmuH6rDbLGYZQXGKTqsaRG9aq2IgZu2HsP/s7uzUIowdQpEllFP+FzAatXQ5wqfjeVUGSZHNaA+3RNhVe5I9xz7Fcpdlhn2+eb5liaXBvt6+2e6id16zbJVsVofVGcE8giqqUoTFbXXbBjgGOFNhIO0or+iV0pU0c7JloHWgLdWe4chwerURZIMcIUvMkkZahluH20bacx25zixtHPiZX/CjX/RLftlv8ivjzRMtk+xTHFOcfq0QZrKZwhwsEAukufIc01zlbvPdljnWObYiR5GzQCtn5UKFZbFjsbNYq1UecDzgbITHzBusG2yN9kZHo3O7eat1q22HY4dzt3W3bb9jv7NVe1P7QOvTyshikoMFj8ixjOnGEzbP2LJ287LphdkJfHTQjBWvPrhjcn2hOKN3Cy4LZVbSPjovB8Jkf1iKkUjZEiLtcYpmS1Dd05PpeO706amTmqfnT523gV8z27W9LmFAA0Ruk+NdHVZnZt4ffT6ed8VHSZUv65ZE6ttkarQ+btIf6NFC2tefWXGTkVwdbC1JTWFf35Jl9Wda29PSKkqCGRfJK04leT0QBbP8WnQ+RCjhTreoKBhukacP+FZenkenkN+l4F5QGxyRp8IPObaZoUNiurRXuJ7iXvFRntEX3RLdFL0+Wo2Wgif4d0QmiRllHOLUoKQ/f7Fdl/zr9nY93+mX8dgvdKHZ4UshTEMyjvA7I/Ipi7LYFEUVXY7p4bp8QfF06Shn3WsWKZHVzB12QReMG1IxA7rvpqPCVAHa+bFvM1I9cZFqv5OTEne5l7inwxCY74/MzI8YpGSo0R5lQIYZ4mVlYJw5MWX60G+B6vTp914Drojo+KS9AzXKrYecyjikwrZw08COqNiEzLy8Lp9P339qt4/+glYOWXPE8BE3oOq3+U1JtETo6Ym0bt67Y1JnPEKmvlM4rMMZsj0SmmT2oJXnxISl6mD2g9uvmtCPraFdCszwh6fl2xU1PNKtqGaIkZWEaHN80vTUmzQzFDPcIDLGuzdBExpsKds8poQO54C4oErX8v5aH3pNvBX+77wRBP30ZluE9Lihw/6b7XLDNvq3KaFoe8CXIH7Pmfc5xCvGB6m3HnJ09ddfvts73VFkfj/0nQr6v8iZlvNYAAf/8t2emY6iv/rKFSmeN74xgaCn84+T/12EWskNK8RuWCG8BZl6W8iFM0Ju33t6LbnguPgprKDx4ziV2oOgisYOih0wRi/SI7TGEiymDkI98xZ+zfAxm86eYr3CQCFAh9YJyibuwSfFceIjklv6Wp4pPyt/bJppajUkjcJCGAQVYCPvVGG7rpnoEcKp1r9DmWC+/gVQNJOiWca3Qb3NIJx6wbYACssPtfGmcfGmtgSRbEaoLYOblcMdUAXVsAZWwmJYRNxXgZfe0kpoX3jBB1l0ZVNrIc3wwniaswpqqKyEMgjAchhMo1OgkuYPpdY4WEaXF2bdoFVj9MqoLqM199O9lGZa/g6uw29wLSRO9xMv/WtTJc3W5QjQmv8ZxwnUWkLr5sJqmlFCcwMGtTJjRcDQyEtUKuleTXMWEt3FNM9L66uIe8B49l06sw0qNSQRveHDUhrVudbQ3CqDko94Z0POLav61whBJ+l7yPje/Ne/KMOn9f82TIBJMJlwngrT4U64C2aShrNhDlGbB0VwLxPgNJzR31RNqysX54/PygrV2aF6WJuw3t/3DcceN36djF/58Mtm/MKBn3O8xvE/kvHfHfiXZryajJ89Nk76jOOVZrzcjN09+Oce/DeOl0bhn8bjRY5/9OGnXbOlT5uxiyZ2zcZPPs6UPunBjzPxI44fcvzAh//qxj804/sc33Phv6zD35/A33F8l6a/uw4vvDNJurAO35mEb/82Wnqb42+j8S2Ov+H4Jsc3OJ5vxtfPxUmvczwXh//kw9c4vrJBk16JwV+HYyfHlzn+iuMZjqc5/pLjKY4nOXZwPMHxuIbt9clSO8e2YyekNo7Hji6Qjp3AY+vFoy8lS0cX+PvwqF98KRmPcHyxGQ9zPMSxleMvOB4sxZ878MD+ZOlAKe7f55L2J+M+F+4loff24B6OL3B8nuNzLtzN8dlnHNKzPnzGgT8rxRaa0tKMT3Pc9VObtIvjT2248ydR0s5S/MkOVfpJFO5QcbsFn+K4rdkubePYbMettGhrM27Z7JC2pOFmBz7Zgz9uOiH9mGPTpgVS0wlsWi9u+lGytGkBbvKLP0rGJzhufHyotJHj40PxMVLzsXHY+KhVanTjo/QSRwMNpVhPSNUn4wYNf8jxkYc16RGOD2v4A47rOdZx9Pc9tG6d9BDHdetwbSnWFnqk2mR8kOMajg848Ps2vN+Cqzmu6sGaHlzZgyt6sJpjFcdKjssScCnHJdp4aclsXMyxYh0uok45xzKOpRxLOC7kGBiFxT14nw0XcLyH43yORfMsUlEPzrPg3eFR0t0+nMtxDnGeMx4LPTibqdLsSJzlxplTw6SZHAuseBfHGXeq0gyOd6o4neM0ejKN49QpqjQ1DKfE2qUpKk624ySO+c04sRkncLxDGCLd0YPjT+C4aejnOJbj7WNc0u1uHJPnlMa4MG+0Xcrz9zlxtB1HcczlOHKEWxrZgyOGq9IINw7PsUrDVcyx4rA4zLaj7zar5ON4mxWzMq1Slh0zrTh0iFkaquIQMw724aCMZGlQKWaku6SMZEx3YVpqspQ2DlOTMSXZKqU4MdmKAzkmcUx0YgLpmeBCbynG92AcqRBXirF2jCEEYzhG9+CA8RhFnSiOkaUYQUhFcAynReFR6OHo5hjG0UUTXBw10lUbj+o6dJaig6PdFi7ZOdpoti0crRwtKpo5KjRN4Whyo1yKIj0UyQM8SKPI6WVMlYQhyFQEjqyNlW54gg36//CD/2sB/ttf7H8CcILv1gplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9MZW5ndGggNDggL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicY2AY8oAFTLIysDGwM3AAWZwMXECSG4h5GHgZ+IA0P4MAkBRkEAKrFAYAD1AAuQplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9MZW5ndGggMjkxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nF1SS2uEMBC+51fMcXtYXG2rLYhQthcPfVDb07IHTUYJ1BhiPPjvm2SiCw2Y4XtMGL8kOdevtZIWkk8z8QYt9FIJg/O0GI7Q4SAVSzMQktuIws7HVrPENTfrbHGsVT+xsoTky4mzNSscXsTU4R0DgOTDCDRSDXD4OTdENYvWvziisnBiVQUCe3fcW6vf2xEhCc3HWjhd2vXo2m6O71UjZAGnNBKfBM665WhaNSArT25VUPZuVQyV+Kc/UVfX7/Y8dXYqF6rXQD8QXUQ6wkcq+SaS95lgG707JJUT5FHlkcZAF6dIE8z7jQ2mIiN4T6YbJJWmKfKo7pBUGsOXC9WrD2T7dZ+Nv8g9eL4Y4zIPtx3C9jFLhfuD0JP2Xf77A92iocQKZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvQ0lERm9udFR5cGUyIC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMKL0NJRFN5c3RlbUluZm8gPDwgL1JlZ2lzdHJ5IChBZG9iZSkgL09yZGVyaW5nIChJZGVudGl0eSkgL1N1cHBsZW1lbnQgMCA+PgovRm9udERlc2NyaXB0b3IgMTkgMCBSIC9XIDI0IDAgUiAvQ0lEVG9HSURNYXAgMjIgMCBSID4+CmVuZG9iagoyMSAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvVHlwZTAgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucwovRW5jb2RpbmcgL0lkZW50aXR5LUggL0Rlc2NlbmRhbnRGb250cyBbIDIwIDAgUiBdIC9Ub1VuaWNvZGUgMjUgMCBSID4+CmVuZG9iagoxOSAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9Gb250RmlsZTIgMjMgMCBSIC9NYXhXaWR0aCA2MzUgPj4KZW5kb2JqCjI0IDAgb2JqClsgOTcgWyA2MTMgXSAxMDAgWyA2MzUgNjE1IDM1MiA2MzUgXSAxMDUgWyAyNzggMjc4IF0gMTA4IFsgMjc4IF0gMTEwClsgNjM0IDYxMiA2MzUgXSAxMTQgWyA0MTEgNTIxIF0gMTE3IFsgNjM0IDU5MiBdIDEyMSBbIDU5MiBdIF0KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDIxIDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMCAvY2EgMSA+PgovQTIgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvUDAgMTMgMCBSIC9QMSAxNCAwIFIgL1AyIDE1IDAgUiAvUDMgMTYgMCBSIC9QNCAxNyAwIFIgL1A1IDE4IDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0Zvcm0KL0JCb3ggWyA2NS42OTkzODcwMTExIDE3My4wODg0Njk0MjQ3IDcyLjU1MDI5NjEwMiAxODEuMDA5MTMwNTgxNyBdCi9MZW5ndGggOTEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicXcy5DYBADADB3FW4AsvP+WuBQhBC0H8KJIgjHa1WcAfGBSCV3Fk7UNKJo8Icz4mbWKvD8YBoEh01FKWEmFtMHnaKbqv81R9+39MkjbhqRN+8AaxwAdgdHd0KZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybQovQkJveCBbIDIzMC4zNTQ1ODQ3ODE0IDE3Mi4xOTIyNTMyMjU1IDIzNy4yMDU0OTM4NzIzIDE4MC4xMTI5MTQzODI2IF0KL0xlbmd0aCA5MiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxdzMsJgEAMRdF9qkgFIS8fMmnBQkRE+986uBB1e7hc8E7KC5F5iWlGB6NCUBZoPr8+BG4Vzcd0lxqq3oyhAlgjblfxjBz571/+/L+fMkGbpU/fiFa6AHojHrgKZW5kc3RyZWFtCmVuZG9iagoxNSAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybQovQkJveCBbIDM5NC43MjE4MzA0NTM0IDE3My4xMTYzMDUwMzQyIDQwMS41NzI3Mzk1NDQzIDE4MS4wMzY5NjYxOTEyIF0KL0xlbmd0aCA5MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxdzMsJw0AMRdG9qlAFQm/0byGFBGOS/rc2gRAm28Plgk9SfhC5QqJWOaNCdNKL35uOaGQr+EU2LfBaHYyGqOVkftylFtr+8x9/3/ukTIA0jdsPoiddzTIdrAplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9Gb3JtCi9CQm94IFsgNjYuMTI1OTAyNzU0NyAtNS43ODQ4Mjc3NTQ4IDcyLjk3NjgxMTg0NTYgMi4xMzU4MzM0MDIzIF0gL0xlbmd0aCA5MQovRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxVjLkNgEAMBHNX4Qaw/Jyfa4FCEELQfwqXwBGtNJpZwQMYV4BU6hkliotRcYtQvCbKJO4RhSdEJ3cxT1R6pswGDBL1zvYzX/idzv3ilNVKh7sDbHADxVkb1QplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9Gb3JtCi9CQm94IFsgMjI5Ljg0MDg5NTEwMjEgLTEuMjc0MDI5MzQ5MSAyMzYuNjkxODA0MTkzIDYuNjQ2NjMxODA3OSBdCi9MZW5ndGggOTEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicVczBDYAwDAPAf6bIAkRJmrrNCgyCEIL9v5QPlJ91tmx8kPJK5AWCtK7BKk1hBXzNGgIg0PgcWsSBUnm0MYI/6Ck9tGf9TT99b+eDxcRbqOfQnWijG/9EHDgKZW5kc3RyZWFtCmVuZG9iagoxOCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybQovQkJveCBbIDM5My40NzI2NzAxMTAxIC0zLjI2MzUxNDY5NjcgNDAwLjMyMzU3OTIwMSA0LjY1NzE0NjQ2MDQgXQovTGVuZ3RoIDkxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nFXMywmAQAxF0X2qSAOGyX/SgoWIiPa/dRBEZ/c4PC7jAQ1XAGuNVNSzcGGSrmqF15+FIqM64wlaQX1McTQKT7Z4VMlSIufri192CixKEursg3eADW45+xyyCmVuZHN0cmVhbQplbmRvYmoKMiAwIG9iago8PCAvVHlwZSAvUGFnZXMgL0tpZHMgWyAxMSAwIFIgXSAvQ291bnQgMSA+PgplbmRvYmoKMjYgMCBvYmoKPDwgL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuOS4wLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuOS4wKQovQ3JlYXRpb25EYXRlIChEOjIwMjUwNTMwMTY1ODIxKzA4JzAwJykgPj4KZW5kb2JqCnhyZWYKMCAyNwowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAyMDkyMiAwMDAwMCBuIAowMDAwMDE5MDk5IDAwMDAwIG4gCjAwMDAwMTkxMzEgMDAwMDAgbiAKMDAwMDAxOTIzMCAwMDAwMCBuIAowMDAwMDE5MjUxIDAwMDAwIG4gCjAwMDAwMTkyNzIgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQwIDAwMDAwIG4gCjAwMDAwMTIwNzYgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDEyMDU0IDAwMDAwIG4gCjAwMDAwMTkzNTkgMDAwMDAgbiAKMDAwMDAxOTYxOSAwMDAwMCBuIAowMDAwMDE5ODgzIDAwMDAwIG4gCjAwMDAwMjAxNDYgMDAwMDAgbiAKMDAwMDAyMDQwNCAwMDAwMCBuIAowMDAwMDIwNjYzIDAwMDAwIG4gCjAwMDAwMTg3MjggMDAwMDAgbiAKMDAwMDAxODM2OCAwMDAwMCBuIAowMDAwMDE4NTgxIDAwMDAwIG4gCjAwMDAwMTc4ODQgMDAwMDAgbiAKMDAwMDAxMjA5NiAwMDAwMCBuIAowMDAwMDE4OTUyIDAwMDAwIG4gCjAwMDAwMTgwMDQgMDAwMDAgbiAKMDAwMDAyMDk4MiAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDI3IC9Sb290IDEgMCBSIC9JbmZvIDI2IDAgUiA+PgpzdGFydHhyZWYKMjExMzkKJSVFT0YK", "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2025-05-30T16:58:20.883363\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.0, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, axes = plt.subplots(2, 3, figsize=(7,5))\n", "axes = axes.flatten()\n", "cmaps = [\"Greys\", \"Blues\", \"Oranges\", \"Reds\", \"Purples\", \"Greens\"]\n", "labels = emotions[\"train\"].features[\"label\"].names\n", "\n", "for i, (label, cmap) in enumerate(zip(labels, cmaps)):\n", " df_emb_sub = df_emb.query(f\"label == {i}\")\n", " axes[i].hexbin(df_emb_sub[\"X\"], df_emb_sub[\"Y\"], cmap=cmap,\n", " gridsize=20, linewidths=(0,))\n", " axes[i].set_title(label)\n", " axes[i].set_xticks([]), axes[i].set_yticks([])\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", ">请注意：这些只是在一个低维空间中的投射。 一些类别重叠并不意味着它们在原始空间中就无法分离。 相反，如果它们在投射空间中是可分离的，那么它们在原始空间中也将是可分离的。\n", "\n", "从这个图中，我们可以看到一些明显的模式：如`sadness`, `anger`, 和 `fear`等负面情绪都占据着相似的区域，只是分布略有不同。另一方面，`joy` 和 `love`则与负面情绪明显分开，同时也共享着一个相似的空间。最后，`surprise`则散落在各个地方。尽管我们可能希望能够有一些分离，但这绝对不是一定的，因为模型并没有经过训练来区分这些情绪。它只是通过猜测文本中的掩盖单词而隐式学习这些情绪。\n", "\n", "现在我们已经对数据集的特征有了一些了解，让我们最终在上面训练一个模型吧！" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 训练一个简单的分类器\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们发现，尽管有些情绪之间的隐状态有所不同，但对于其中几种情绪来说并没有明显的界限。让我们使用这些隐藏状态来训练一个分类模型，使用[Scikit-Learn](https://scikit-learn.org/stable/)。下面我们以逻辑回归分类模型进行举例，训练这样一个简单模型很快速，而且不需要GPU。" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:21.377645Z", "iopub.status.busy": "2025-05-30T08:58:21.377385Z", "iopub.status.idle": "2025-05-30T08:58:21.728137Z", "shell.execute_reply": "2025-05-30T08:58:21.727507Z", "shell.execute_reply.started": "2025-05-30T08:58:21.377613Z" } }, "outputs": [ { "data": { "text/plain": [ "0.2335405945777893" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#TODO：try a classification models in sklearn\n", "from sklearn.linear_model import LinearRegression\n", "\n", "lr_clf = LinearRegression()\n", "lr_clf.fit(X_train, y_train)\n", "y_preds = lr_clf.predict(X_valid)\n", "\n", "lr_clf.score(X_valid, y_valid)\n" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:21.730677Z", "iopub.status.busy": "2025-05-30T08:58:21.729004Z", "iopub.status.idle": "2025-05-30T08:58:37.916272Z", "shell.execute_reply": "2025-05-30T08:58:37.915602Z", "shell.execute_reply.started": "2025-05-30T08:58:21.730652Z" }, "scrolled": true }, "outputs": [], "source": [ "# We increase `max_iter` to guarantee convergence \n", "from sklearn.linear_model import LogisticRegression\n", "\n", "lr_clf = LogisticRegression(max_iter=3000)\n", "lr_clf.fit(X_train, y_train)\n", "y_preds = lr_clf.predict(X_valid)" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:37.917269Z", "iopub.status.busy": "2025-05-30T08:58:37.917027Z", "iopub.status.idle": "2025-05-30T08:58:37.925188Z", "shell.execute_reply": "2025-05-30T08:58:37.924601Z", "shell.execute_reply.started": "2025-05-30T08:58:37.917244Z" } }, "outputs": [ { "data": { "text/plain": [ "0.635" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lr_clf.score(X_valid, y_valid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "看准确率可能会觉得我们的模型比随机好一点点，但因为我们在处理一个不平衡的多类别数据集，实际上效果要好得多。我们可以通过将其与简单基准进行比较来检查我们的模型是否有效。在Scikit-Learn中，有一个`DummyClassifier`，可以用来使用简单的启发式构建分类器，比如总是选择最常见的类别或总是随机选择一个类别。在这种情况下，效果最好的启发式是总是选择最频繁的类别，这样可以获得约35%的准确率：" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:37.929549Z", "iopub.status.busy": "2025-05-30T08:58:37.926039Z", "iopub.status.idle": "2025-05-30T08:58:37.939726Z", "shell.execute_reply": "2025-05-30T08:58:37.939140Z", "shell.execute_reply.started": "2025-05-30T08:58:37.929518Z" } }, "outputs": [ { "data": { "text/plain": [ "0.352" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.dummy import DummyClassifier\n", "\n", "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n", "dummy_clf.fit(X_train, y_train)\n", "dummy_clf.score(X_valid, y_valid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "所以，我们带有DistilBERT embedding 的简单分类器明显比我们的基线更好。我们可以通过查看分类器的混淆矩阵进一步调查模型的性能，混淆矩阵告诉我们真实标签和预测标签之间的关系：" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:37.940791Z", "iopub.status.busy": "2025-05-30T08:58:37.940382Z", "iopub.status.idle": "2025-05-30T08:58:38.278531Z", "shell.execute_reply": "2025-05-30T08:58:38.278010Z", "shell.execute_reply.started": "2025-05-30T08:58:37.940761Z" } }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDE4LjM5OTM3NSAzOTcuOTMwNjI1IF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nM1ZTXPbNhDFGb8Cx+SQFRafxDFuWk1z6NSOZnro9KAqsmuPbNdWErf99V1QUgRYBL1KOqk1Q4tcge89YN8CID15s/x0uVieTU/Ud+/kZH+1WEtUV3RcKK2u6HhQqKZ0XEhNV9fSYQc2JRs9Xa7KS5siJKuD8RTX9eUfUp7LyWuCWdNtUyljB973tzkE53tgBExFaFWGbHSgE8X2N34O9eB36jGktQaC233dL9Uv6kZNXpusgfpIx0PWoure3+1uyN211oINB8iLazX5EdWbW3UqT9XdDlTToGVgDd0WmiISdQATq64WIUuDlLsqT2jMH+Qd/dXqlSacDiGhwY4aGQ/WZFp5MpOTH1ChUbPzPiGz9/JX9UKsxVy8FzdiSWfrl+o3NXsrv5/JU9lrkRgQItYa9qG2BvQGOsOTcCVuxd+H1AYjYJ3pItSmNtqBj9ZZFvmKyD+J5QB9NOBTTb8PjdB7D2g7Jv2chv6CBv/+UIA1HWhbCShCbQEWA+guV8/T9OdEPR8k7yy42ntFaITcJ7DoLbLo1+KjuBd/0nFJ51USqoRqMGmT0ARxDPDnl4p6n0zUnc0fClHnDPXJ7j4UWpLjL8VCfOjPFHlgLn6n89XhOBSThgtgI41D9KCr0KALsv1zC4ROx6+pwL0CkzTNi5WCbWhIgbcQY9fnoSOZYVxEowYLcuvBpZp8ExoizwPTF7/VORtfVoF7cuw0kVbk29CgBwOEvgm1dt04+Uj9FfTGQcCafhMa7HuEsCkAJONpNy6gWYF7+pAAa+NtIoO+s2B67kD6njTeSPVlsFcZluopVzFmPUn32COYMypA5+tyuyeeZavMjHq72S/0S1+9oh4u9QNruHx3uAG4HtwAUEvW5qFqt727iaizfkiBJtzoPOZzmvvRRloHaAjvL/Y7hnars6kcxSgznMzn1Z2mRHLEeJK1ABEF1v7SG3AqXg3YdVTIJrnM6zCRazJ5Vt5qKKuGZ9Nmw8eI1Q7B0f7PH9cRrDtSrRE03aXj0LQwLTRaxYw9VptuoFmkWozHavMttEgjfKQBdHPcCjfRbo2mNA5Yqt00Zv5R05e/ylHTl2bh6uzqhDwP15c+5Q+4Y/iUj2YZPv0P0Aqf8tE4PqWnLIycgnT/r0+ZOp0Iz9ynzI7Y5gxY+ZSJpkXk+JSNxvIpG63V08KnGANE+zSYqefThpu4aNieLIqUctEeLZHfpIZKs3B1use1/hxqqPQpP33NXUThU376GPMpfdHjNUdaqyArn7LROFs5Llq7vCs3sbXpb+76yixMnf45rhyVT9npa01ahU8DPVay9uEtrNKmTDDb9HzpUiZYuxpLk7LBmg8IhZPYYxYYsw4TzNRjpvt/CGzeaFfJoKdt7F/y2/wqJ5WwDlwB3CP/JG7zWxKMYANuP/KFuBZzsRKX4p/tq7wFNbsR5+KjWFM0n6u+zYf+LcdfJEzuhJ3KfwF2JNi6CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTEwMgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxOCAwIG9iago8PCAvTGVuZ3RoMSAxMjQyMCAvTGVuZ3RoIDg1NjIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic1Xp5fBRVtv+5daqqq/fqTnfWTtLZE7bEjgHCIi1CWMUgYQmKk0AIOwkEUIhOAIdEBCdBIAgiRARkEwMykEBA0IyKyKgD+Hs+dRgEt2dEZh6OY0huv1PVCYvO/GbeX7/Pryu36t6qu5zzPds9lQIGAE46ieAdMmhwDgyAMQCsG931DMl9YEzUjZRmag8FEJ4dMmbswPVf7zoPgJ/S8+v335s31Dgg8zc0+ANq/+2BMem+GT+VrACQ3qH2uClzCkuF4ZbxALKX5vBOWbTACzOiswEMfmrz4tJpc+bdvWgmgJHasHdaYVkpGOgAYwu1LdNmLy5ece4/76L23wGiM6ZPLSxSJr1RCZCSSs97Tqcb1q2GjdQuoHbi9DkLHntrWcjL1Caa4ODskimFq4etJXpTe1P74TmFj5WKm+R51F5Hbe/cwjlTU77r30TtQ0TP+dKSsgVXv78cDdDlKXq+rnT+1NK+hr9QtauHeJoOGlYWCP4EOhBS6F4qdKO69swEPaAfCINyRuaBbXbhgrkQTrjSLxAAuFnTerJZU+fPBUWr6UWkGbSrAgI7pPVk7zFO87kB+Cl93Hb9fEQ/n6PyaeAIlU+1WgdFELBqLf4Kp7482F8fy7fzc/wVup7WW6fg3/oF5w3M+td9bqfh35/5X/QJcj2oo3WOeD13k6ZPtfbNnjpXgcL/DQWdVARuwyLwC1z4v8sVI0mp4IBQCINoiAEvJEISJNN97R672U/TmFs/ESQ6yx0tA8neqGtCwm0j7vxp4zUtkmhUsL+JWma6WoBkDzaw63Q4IQRcAB06uQX2gUvXySWF8wsnQ03h/DlzoWby/MIZUDOlcG4ZnadPnU/nxfNnQ820qSVUnzZ/6iyomV44l/pMnzqZ7swqnFsINbMLS7zamXT7N3MKF0yHmrmztDsl0wrnQM38hXOp54LiudPoPF2b/5/ov47a7BnTCu+wARGCNsCgp36ViC8XeCBetzEkdKOgq36NgC6ERhSk0TmCngp0j9ER1sHzn+FryIV+FjB8pgO3nJYvIOAudEBfcCewnW02vaM+9t8Quo3mXfSv+8HEYF/terN+2xx33J94W3v+rbpA/utGNdX73RyaSlyagpMEz6KN1WgaJWVK5BFZTPCK/weKBSfNYJYRFVEQxJsjOn65xYOLwE86u0R2cRfbZJjDLt/RBzsK+UB9xdXUYnpbhFK6hpPOIUnLS94vG0YR7nkwDWbSs0WwRJe89iSj40khzIDZMB8eCwQClwMfBS4ETgVOBo4HDgTqA68G9gde+afa3+l713W0zPrMwaLR16OjaBxmUKE4o1vYqI5i15jtKCqVvI7ioKJ5j2lUCCmiD4h6IL0CohR0HqOoaPJY1LHeY1SWUEliWdAAZ+g4BXtgM9tJrWK6P4/u1AkHYQUspDtvsjNspdCd7u2Ea3COelbBGdwjAhsOmXQX4GNJgOssDw7RHNnMxbINMpnDKPGQ+KDYIH4lnoVeYpl4ViwQy1gmbpPGSTupZOPvSbanIRYa2EUog6P4DWZikzhItMFFPIt74AtaRcPjDFTDdignWlysBCqEcuFBuvO2dBY20VFCz8+yLewcUXeUPQkX4DkUhaGwhV0gvs7A3+BJzBMqCOZMoZjof5vmOkvjN0EZObELzARc6Er3iHpaa7J+jsbu0gX9uAYVtHIebJcbZJchgVbRENvJ3mQt8lqog3P4MM7DT9gKMUHcJQ6F6iACWADVNPcmbYxczBYT79pRrs0uPCoWsD3wjVhgmExz/17jiNY8JDxIHBVDE5VHZZV46stW4EqiVHsaDWcNw8V0Gk8zGJ4grgFKMItkXULP98NB6I61UE0z6fzKvaS/0cjN4iXiuZo9I/wNzuIg8jjF4lXCWnOwtQBHDLIkosCgm1etF5KGFdX7R0/wvpMf173bz5pe1eCth9x662JvQyCQO0GMkvLrJU89Jin1YlLCpX/28FL3biNyJ3jr2wcP6ph1cMEgujdmAlW1Ft2m+4MH6c+0ReulJPobVlDvnTLd+7T6dEKfp9WpfbprNiNoOwyySc2CPyEj3w+c6o4j8KTAIiBcVNta7oL0loxeme6ET86d45xGVAUui9UkRTN51wR/iFznhDrLGufqcKPHHoMed1Q4jbpO49Qr11vUqxksXnCozkyf06EKKT5wqJAQr52FVZtfeIH+XnjhBjPyH2/c4D8yo5TLz/L3qJxlmXTczTLreBmv5FW8jD3DFrMl7BnNF10is55I3tsEfr97INaJQp20zAB1RiVW9iDEMrN6fkS9PW9CI3X2985vaQ4y4rvecr4lg7DKj2eH7GgXhUm94hxSVlKmI84dx9lwvpFNfZcNb9u+Rywb2jC09cIemoDkKg4njj2wxZ8SERmF4R6HJIJDksSB6ouOddY61xqR7BtUk8BMnjAV5Wi1bUS9O29EfWjeQyPqXXkPESUYONk7v/l8y8mTDmd2BzXXdWoMqvSdQfqO1XtUR1g20eb3jRXHSeMMS8Ql0qKoqggDWX+EGElq4FkAi+SFkWVRCzzLoTJieeTyqOWeXbAryjEJJiURE1k9odc9LOvu5IR42ZB1D8v0iW6XbJCBXM6ptpEEY2bh/S9X/urcY0vOT/iauQY/FMGv79mz51G2ps+cDcMerR1433t3+b5+4+EdpdH8W+J+M8m7jLhPhVJ/D3CHmCqNsZXekDq3tc64VvbUedcmrJFXu19KC/WEALoiPMle1YOuWKOcpoEQmtfJv1HnnwC43kJcEgJqy5XrV1rUL6+q+kGoZDC/sSimMLbQWxQnwiQWw9wuMS4+OSUrhhjpSVx1ZVnByh3s4YA1L/EP+NePvD0z7505J95u3LH/8PotLz035sT8stP5XzLLbzEptrnms78mJb15l6+2+jfrdz5aWlaemHzI6/3w4ON7NUsoIilvJ50SaO+0zB/NrGgFROtAQLOhTmK4zMgsJvDIimixqZ+OqDcTY1adMYvG2Pl+zS0+hybXK+f7tfiIF12w4mkS7mlNpF3MtEkZCvkUUB6Fp8EQyrpCMuuKPdko9oDlAes4VswWsiW4gllJlEYWh5kOMjtHgiMuC2UuMJ7FL1w43f6IlNR2Gc+2Ze7idazgTZLQFpJQEVEeDY/4E8RIg6NSjY6sM7jq1JVWoQ6WWVcbtseEeZgJPWBS5Ri1jd0uF1Ujv8NaVM1aSERq81XNgDULJvHw5qB0Qki/HBrm4HbBHWLRpPEZRrTXdZvQrZUl8vP8+0fenD7x5KxX3n33ldEv5kkX9vBn7XZ+9b/+wn/wes/clXF48+bDicmEdjVRX6v7k0SY4E8MkcFaaYG6ULnOE7pDrbOsjF/jWZ1kiTd6ImJCPBgXG5VEDoaU6IruYq60XbmlPn4XxV52VjiLZ8Uz0hmZ+D4YI0xik1i87HaFBmll7h4sIV7ATkYSvJo7ivOFCtuf2rr1KSrMOPL5ke+cs/c9OOsSk/i1z3k7v8pyWdTI57Hv0W0vHjv24rajwuKGxGT+V/79+En8+2+/5P+lO6jJbEeM5qF2kTZNJ5nIMMUfLjkEFNAhkr+QSB4oIRMZyAa17b1mh2YJ6bf5ASqkLpqAJhynbb2fJqMkmGTk6NU73++cIDAZI6Vsaag0DeuhXjaQtpBgWAKL24Un2z8/x3h7pnRhXOsyqau2E1pF+K7S8U2AdLjPnxRO6KbIdTHd65xrYlanvJQRbkns4nEneuxG8t7kwu1xURlqW3PL9eYWHdhOW9Vb2WSkt4GZ1IN8TWKmL1RzMrq5JsQnZt3dM6SzA2mGsKpmx46amp07+I7layDwp4t8zbJnX+I//vgj/3H70DVPLl+7dvmTa4Tfb6qq2vR8ZdWmcd6DS1/74IPXlh70xr9V/fHXX39c/RYrXLB8+QIqpDHLiKMq4ihc15gEQ2wEq4SIOtMOsQ5WhsbWqWtCVycZPJ64kBiIj/dYdYUh8jtj0pf8h059CW2OeCPyZNRJz8noN2KaYw17nE3Ob5xIGtNL121niI10BbLuhsyglsQns062CINLIzePID3pc3D2n/kNpn7OkDn4Af7FyM3sng5diiUtYVbmHPcws3/7JQvVw9lW/lCMsKFTkzR9Oa3v1C9Q7rbYH2NwMIEJDvLaAw0CwquKJDOD4BF7GmiTbSJm2lqCakOOJrvT0Xynlfz4g7Fmxib5M3oKvQ1DhSGGGUKxYalgkJlRdrNIOYcNk8ezCfJUNkNeLK9gT8vraU+11azqWsQcFAiZI+E0U4XaZn6tfWazdOFGrHiptat46UYsYU90is8QnQrllH384Wy9CuuNy5yqSaFUQoqwDnCAxyi6iEZfm+bmdd2mmOs3292x7gHuX7lfdUtkj44O7UiK0xy42JUWZWv5M5s2PcN7s3duMMYDN/i7Unr7+89WVT678/Inn33evgtYoJXW/4bWN8Bwv00W1sMykfnJr/klRT1PvkBfz5dB9mPS7EfR7UcBpdN+QsAYCyqxF2tQjX5jqXGr0TgJNb9GflYWv2+/eqb9Knmr1gua9QhQTrrWnXaZJsrgmyj6x5rDjDbYHSY32hzeytijnsaEBsfqMAuEYbjVqJhjUXENTib23zvf4vMFpdR85Xobqd1bui05sjW9m5sRnRGTEZvhzYjLiB+Q4o/2x/hj/V5/nD8+Nzo3Jjc215sblxufm1KasiK6KqYqtspbFbcivialLuVaSkzn0M5BnQMKYgpiC7wFcaUxpbGl3tK4pTFLY5d6l8aF3+4B+7NejoQsTa2TyU4z426PpaHCiYv7lpVsbGxoGND01L4z7TeY8PKGgsN5U09M/O9rQmZx+eSyjw+ljWxftqe48NS24yedFat69NiTktKmYXWUsNouu8jTeKC3PwIbLXZjY7h7tb0hakMEOJ1Dwi2yEpkTrSmH77q+Ibui+Zi3rmYcLohZGlMXg0Rnp20RqUx30bRvJFpTKBRm4hcvP/vsy1pp/22fA+XvQSDwXvmBPo2NQvqZr746Q0V4sKiQN/G/09FUWLSLqGEwL3AZvyIZRsAAfxRUsqdEW6X1KVOjQ2wMI+FFGpxWGOoaHKm2XfF1+gh+/ar6w1VNbaPUqKVRNVF1UZra6qGjg7pebt03BGMHfjXqhdzX3nrrtdwXRt2/Y1I7/4h1Z/LYbWLWvq5dL589e7lr1z2JicSQjTlZnwRCi6gSJxJ9ahCtyEawuRolZbWtgW3AMBEUYYjDaR4cre+mfb6baDXfgRYZbFCYtMkGEiC7ze/itoaGPgcePxOAwJnHD7S/Tbjt2kXY4WHhkZ9adhUVskFMoWNQIXd3wNdBVwWh5aIst9SfCG5mrFSekty7mdRoYcfCG50NltWeKLeguBUYITjtgz06ic36rlYDLxiQrwc9bNqA6NLouugPoq9FSwNgABsgDHAPiJK6GdKVdGM3UwmUsBKhxF0SZZw0TwM4Tt9S6NjqHpcUwKCDbhAr2g5azh6Z+fbkKR/M4tf52yyt7XNmaBB2PLWp0SY8MvHE23ffvb9LN9abmVgIu49/1rzh0P4tmndNJ/X8O2EdAvl+j6Qyi7JbZlWwwSY3mYQQCq9GSbHazSNd2o7VpG2EzNpGaES9Ta9rG6R+zW39mpudukFfId+mXqU9HsXCw353rrvOTSHDTURGs6ArScjK1IxL+Hv9lPtZOv+wsb5+/3HZtTF3+pTqtnT8sHrUsb0a1nycOJGwNtPuerg/IcISbXRWhoQ22rExOaEhpcnYaD8eGZ0cAYpliOx0egen6XE5qA7NV4IKwS9oSGeTVnRZ2qWuy89sKEwVbu3U+rMOVaF8LDQsKxO37Vi/bseOdet3NHDeWrhv9OgtD/7uUPbBx//Q1vaHxw9mNwj93/n003fe/vTTb/nn/JvomNe6dTn++kNTJrM+TNvD9Jk8ZY9m+acI5MWEL5Jf7kp++YR4AJoEiSki5ChqG4UpDbU2LQxoXjfXWECel+wphLRX2+SeaqCfWHCjTnZ9Q/MFPuHj9PnMYIdBfo9ZMIDthMVQJR2HJssBVVEl+QErUyyQo+qzX8l23tqj6GDQQg6/I9dR4Ch1BBdyyR25Q3DBl36Xc9eMkfqqqz86ublwo5z6DWnKLU7ij8AGgSmQczML9ltVyS/lSgVSqXSNQrJOPpEuu35q0bSMArkhmqQZDxP9ybLTGG4HOdrgtlRFe7EhqilCNYDDrihyrkOx53rCyRUm6HGyjSKlnhH263flup5GOMOIg5CMxNzE0sSaxDo6Xk+8mBhINJJsdT/tJtyCQr69kunWH4ppg08uf/VE4/yF1Tsb5z/6zM7GxgH1i5fsxZWPL/rh8/aHhS0vbj6xvb1K2LLt+ddfaq8SC/ZPm/x4BwdiEXEQAj3JfxsBbUyusjkaLE0mJigwSotwOXpw1913P03/dGIPFbjfdwuaP/oH5BQ1PP74+n2NjQNfW3jqLWG7RsDWLRoBtPDUou87fM5C3Q7CyA5C5EYnNFoatHcKTvtodLoH/+ydgj9hQEQ5lMsVhgqlwlhhqjCXWyqsFbYKe4Va4Sh31kVci3Dcueu/49VD2bp9e9ev3bdv7TXm5Fev/YV/zxx48avTp7/6+p23v9nM3+Et/DtyMNnkR1ystxbbyFK3E4Wat77HH9XprRtsq9lxbIomTz1E99m3RTdKnzodtt8Y9Nh/jhHZpKSb0HSEtjtCXllj463IJvTujHe72vfLpj23xTb2bafLDsoNhxN1DqAUSDaTnpmxytZgbDKYZNr85Dg1t6HbIPnp8+9pjvlQbsjWEE1iwYh2S1xhODx2WLfNLxMdR1eE9PDgIafjzIn2gySs4imSRKuVUDx9m1ZLga/8/awWwWYeExujGAWDaUxsbMxAkzkmVnRTnF0puirdK8O1OJtEcTY1xmSOjTLAg1GKzaC44genalSdb7miGWx2dmfg/UELvJpO6dta23e0iTToZ9rbQoq2t53jMXnMHksPCh/dzN0sfY19TX3NfS1mL3hZopBqSjV3CUl3pbu7hKbGpMamedPiElMqTZXmSkulVXujywRBNslmtKAVbWhHFSMwEqPQI0YbU9LTBqT9Kq0ibWlaTVpd2rW0cNoYz7sV92P1dxJywu3JbzrTkqCehB2uGrVr4sqVk9cNaN7x439MfHN28VuFy1dP3evf+9yf/1B8SBywPzU1L88/LM7WZePKzYcTEk5kZeWPHpGbZE9cv3zLPj1z7EUO8K/SFrJB2hXYJMWOu8HBmpQqk5kwJh1TnTbNBvWA5NPjUUvQa5DbO/iqm2lWqEUhV2hfLSYlZ2nRyMEeZeV8xYiy48cvbKuqkrbwN6rb61aO2rT1j0JBNbtH8+L7yQon6Nbvgr5+zy37X21iTa4GC1m/yzyK/ECOWzPH7KBGXfHddAIl7pOaEwihTUnQ7G7uTpLZfs0JvNLQcN+BhafeYe+zo8LO9sKtW09sF8pv1O0rnnINd2nc9ycPVCEWUN58w5+ip8wCJUOSdkFBBpk5AOSBlBS9LskSCkwSwaC969ODNgSDtitPe/emvdoA/ZVTWPBtW/atFFvpKHqq7f/tUGGmUC5UCJXCUmGNsF1QtIWMaCQtppwJI8VkSGZpmCZ6lSzIYn2wj5ih5ADlUjhMzJGGyn5lHIxj+Zgv5irFUMxm4AxxmjRdLlAWwgJWjuXiQmmJvAJWsJW4UlwpVcq1UMs2CJvwOfE5aYO8S3pZrldOKheVgHKPlollGlkmS+j/JnuEPfImf7hVLGjLw3036gihvoTQYkLIzO7z50hakig6UDRoF0mkvBEdgsDMDu2/kA6jiWkXs8mgGIwORTEMNBlEJiqEntBRI2YtQQC1V0APjahXtZNDh0/uxFOrB99eNgdzOoq5Yf8Qz3+E73MmUTRFim5Tsqm/eJdprDjeMMFUbFrEloiLDAtMz4jLTRvFreIGw7OmGtNOtlt8VdxheMlUZ/KYUJQko8kciW7JbYw0p2GylGTsYvZa+7Bs7CXdbehpzDZnWIdhjjTYONzst+ZrchDycbw0Ts43jFPGGfPNudYS62Oswvo8W2fYy7Yb6q3vWy9aA9Z07T2bkGBk9EeAi0V8FtvzMT/Kj37MXuPzP2ZpLE0saL/Yfoo18KHCcCGUz2PVupZSNNC01M5W+e8zKILRAXYNZgC7zWEHu9VhsYJ2sVlNZpPFYTabBlrNRhXMUhUet5mbVJvVYjLKCIpdtJvVTgEoOuzm22A3B1+a6qirZGotHRnkP4Re0d4ih/k0zK/JICmyEa2hpjCrak2wZlmHmR4wjbJONE40zTRVWZda11qdJiAizJLFbDPbw5hbUEVVCjO5zC5LpC3SngKJ5E+9oldKU1KNSaZEc6IlxdrF1sXudfQiO8gSMsQMqbepp7mnpbc125Ztz3DcC37mF/zoF/2SX/Yb/MpA42DTEOsw2zC735EHo9loYSzmirkkn7Ekn/HG8aax5rGWfFu+PddRzIqF6aYZthn2Ake58pjtMftKeNq4wrzCstK60rbSvtG43rzessm2yb7dvN2y17bXXu9433HREXBMJVlKNhbc4g1gmjwzhbWj1j2+dvbIvMw43jdoStPfWbJpaGWeOKptHc7WJDmBIucnJEkjvOCPVILv6chYBiq7oQl3SwoyEJls6nzhawnaRfCNaUhep400+zre47X84kWef6AkhArJwhBhmEEyK3ZzOEYpXRWvuSdmKxlmDa3BOlr3KeMxX/mVuYAVCMVYIBZIk5UK81Lzq+aojjd82tt9FjcPZ7aPFA61PSEcap8qFuxq+2TtLkzqyHKkPbR3TYSh/pBkPamxxIVbYxSHJU51jUzSYoZPixpqPy2Xab4L/A6j1bHbKURWQfgGOdbZZLan9/vS5+P9rvoowfFl3JHU3Eps9MBi0B5ocU/a05nlcIOe6Oyvn5KSzH66I+PpzHo2pqZOnxLMftK1/6sQvW7KzB/0O6JyIEwJtbtERcFQkzwy8ha9vB/tp/xOhUKhWmULPxF6wLbBCE0S06i9yvX3vz7a8wcoTa+hdF3Vk/VfkkwUM9r9i8ODlL7yWqNG+U+NjVru0UnjkVc1otnBbzow7aCxl98elkMZjcmiKKrotI0M1egLkqdRR/njbqNISaXD2GQVNMK4ThXToft5akjeBBr5kVvZoZZESOU/yw9pdbmNVk+D7pRThKfnhHVVuqhRbiWyixFiZSUxxhifPLLHLaCafdq5TYcrLCo2YXeig/Lc7ie6HFBhQ6ghsSkiOk5LM3w+zZeoLT76C0q5Q5q9eva6CVWnzG9LaCVCT0tqNfGO96SMepJEfb9wUIOzQ/ZIaJLYg1Ie6wlJ0cDsBLeTNaETW527ZBjlD03NsSpqaLhLUY3aP1niooyxCSNTbuNMZ0xXg3CPd3ecQ6iyJG9wG+Ka7JExQZau9/slPz0zfwb/z7LzoJ7eLosOPm7ysPd2udyUjfZVgpC/MeFSpPdX9n4/QKyif4Tw4a9tVzqvP37UNtKWb9S+GlJufrNA4wxzeDSAjf/4UetoW/4vvm/oJp7VvxkAQUutV+nlEykMqqhcolJLZTOVIipbqFRT2UVlFZVlUhucFs9RuRxoFb+CcskFR8VimEfXeWILzBM+hHStLmTDKSE78Il2NZyBo5KT+n2h9zuKw6neFUowAXrR/f1iE/Sn0le/ZsIE6UmawxQshiaSYvod9LtgHDzDLGwp+0zoJ7wstOBYfF0MEceIq6Q4aY7UIIvyXPmoIcaQY3jS8KIiKPco1cqfjG5js4mZ7jOdN9dayi2XrNnWVdZ3bdm253SEumMedIXpYCGrUGGjhqjoFkLpqn3PYICJ2v+wRSMBnKF//6HVGYRSK1gXQGE5HXW87b54W12CcDaqoy6DixXDfVACpbAY5sMMmEarLwAvpMIUskcv+CCDjkyqTaYeXhhIfRZAGZX5MBUKYQ50o7vDYC7170G1e2E2HV548OZcZXprKl2n0phFdC6inqZ/Y9WeN1fVvqBZRGtpXy3Mpd4aHYU05n+34iCqzaRx42Ah9ZhCfQv12abqIwp1jrw0y1w6l1KfyTTvDOrnpfEltHqh/uzn84zRZykjikromEV3tVXLqG+JPpOP1s6ErDtGdY4RgsoU+LX+zdMvf911W9K+gHPo+Ypb/9ounCJJJERRjn4XzdyLZh4MOTAEhpIchsNIuB8egFwYTRiMgbG03niK//nwEDwMk5gAJ+GU9lbLsHDujJyBGRkd18yO690NwlJ/4AbHVhf+lIR/9+GPtfg3G/7A8TrH/07Cv9rwL7V4LQm/f/pe6XuOV2vxu1psacVvW/G/OH7TB78eiF9x/NKHX1wZI31Ri1eo45UxePnzdOlyK36ejpc4/pnjRR/+yYWf1eKnHD9x4n8+gR8fw//g+BF1/+gJvHB+iHThCTw/BM/9MUo6x/GPUfghxw84vs/xDxzP1uJ7Z2Kk9zieicF3fXia41srHNJbHvx9KDZzfJPjGxxPcTzJ8XWOJzge59jE8RjHow5srEySGjk2HDkmNXA8cniSdOQYHlkqHv5dknR4kj+Ah/3i75LwEMfXavEgxwMc6zm+ynF/Eb5iw317k6R9Rbh3j1Pam4R7nLibiN7dirs4vsxxJ8cdTtzO8aVtNuklH26z4YtFWEdd6mpxK8ctL1goM8UXLLj5+QhpcxE+v0mVno/ATSpuNOFzHDfUWqUNHGutuJ4Gra/FdWtt0rpUXGvDZ1txTc0xaQ3HmupJUs0xrFkqVv82SaqehNV+8bdJ+AzH1at6SKs5ruqBTxObT9+LK58ySytd+JQZq+hGVRFWElKVSbjCgb/h+ORyh/Qkx+UOXMZxKccKjv7Ar594Qvo1xyeewMeLsDzPLZUn4RKOizk+ZsNHLbjIhAs5LmjFslac34rzWrGUYwnHuRxnx+EsjjMdA6WZY3AGx+lP4DRqFHOcyrGI4xSOkzkW9sGCVnzEgpM4PsRxIsf8CSYpvxUnmHB8aIQ03ofjOI6llccOxDw3jmGqNCYcH3Th6OEh0miOuWZ8gOOo+1VpFMf7VRzJcQQ9GcFx+DBVGh6Cw6Kt0jAVh1pxCMecWhxci4M43id0l+5rxYHH8N4R6Oc4gOM9/Z3SPS7s388u9Xdiv75WqZ8/YMe+VuzDMZtj714uqXcr9uqpSr1c2DPLLPVUMcuMd8dgphV9d5klH8e7zJiRbpYyrJhuxh7djVIPFbsbsZsPu3ZJkroWYZc0p9QlCdOcmJqSJKXeiylJmJxklpLtmGTGRI4JHOPtGEd8xjnRW4SxrRhDLMQUYbQVPYSgh2NUK0YOxAhqRHAML8IwQiqMYygNCo1AN0cXxxCOTurg5JQjd5ccA1F9Au1FaONotYRKVo4W6m0JRTNHk4pGjgp1UzgaXCgXoUgPRdIAN9Jd5ChQW+iOTEXgyBpY0YpnWNf/H37w/5qA/+sv+n8AOz17nwplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9MZW5ndGggODMgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicpYxHCoBQDAUH7N1v773c/4Zm4VYRfDCPCQmBn9FedrpgYGJh4+Di4RMQfv4dCfHt6vEqISUjp6CkkrmmkW7p6BkYxSdmFlY28Z2D8wKAxQMKCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0xlbmd0aCAzNTQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicXVI9b4MwFNz5FR7TISIQMI2EkKp0YeiHSjtVGQh+REjFWIYM/PvaPptKtQSnO797PvCLz/VzLYeFxe966hpaWD9IoWme7rojdqXbIKMkZWLoFs/cuxtbFcXG3KzzQmMt+ykqSxZ/mM150SvbPYnpSg8RYyx+04L0IG9s93VuIDV3pX5oJLmwQ1RVTFBv2r206rUdicXOvK+F2R+WdW9sfxWfqyKWOp4gUjcJmlXbkW7ljaLyYFbFyt6sKiIp/u0n3nbtt/rU1gO+gRcnE2TyMkE+ovp4ghxoAkgBR0AGyAEcUAAeQxvXNcNhmT8s84fl6J77aLmPlqOxhW+gkzlC8AJyoMjEkYnDypGJ8+BAgxNo6xtsFLudo4UP4ykXAHwA70ONsxQ4u+CwBIooBaIUeaiBBT+m8BkCPQX1Yi813J69XzuM2/B0d63N3LiJdQNjR2WQtA21mpR12ecXz/vKVQplbmRzdHJlYW0KZW5kb2JqCjE1IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9TdWJ0eXBlIC9DSURGb250VHlwZTIgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucwovQ0lEU3lzdGVtSW5mbyA8PCAvUmVnaXN0cnkgKEFkb2JlKSAvT3JkZXJpbmcgKElkZW50aXR5KSAvU3VwcGxlbWVudCAwID4+Ci9Gb250RGVzY3JpcHRvciAxNCAwIFIgL1cgMTkgMCBSIC9DSURUb0dJRE1hcCAxNyAwIFIgPj4KZW5kb2JqCjE2IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9TdWJ0eXBlIC9UeXBlMCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zCi9FbmNvZGluZyAvSWRlbnRpdHktSCAvRGVzY2VuZGFudEZvbnRzIFsgMTUgMCBSIF0gL1RvVW5pY29kZSAyMCAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9Gb250RGVzY3JpcHRvciAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zIC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Bc2NlbnQgOTI5IC9EZXNjZW50IC0yMzYgL0NhcEhlaWdodCAwCi9YSGVpZ2h0IDAgL0l0YWxpY0FuZ2xlIDAgL1N0ZW1WIDAgL0ZvbnRGaWxlMiAxOCAwIFIgL01heFdpZHRoIDk3NCA+PgplbmRvYmoKMTkgMCBvYmoKWyAzMiBbIDMxOCBdIDQ2IFsgMzE4IF0gNDggWyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgXSA3OApbIDc0OCBdIDgwIFsgNjAzIF0gODQgWyA2MTEgXSA5NyBbIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSBdIDEwNQpbIDI3OCAyNzggXSAxMDggWyAyNzggOTc0IDYzNCA2MTIgNjM1IF0gMTE0IFsgNDExIDUyMSAzOTIgNjM0IDU5MiBdIDEyMApbIDU5MiA1OTIgNTI1IF0gXQplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAwIC9jYSAxID4+Ci9BMiA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggNDYyIC9IZWlnaHQgNDYzCi9Db2xvclNwYWNlIFsgL0luZGV4ZWQgL0RldmljZVJHQiAyOAooCDBr9/v/9fn+9Pn+8vf97/X87PP76/P75O/54e343uv33On22+n21eX0z+Hyqs/losviWqPPLX279vr+7/b87fT73er21OTzzuDxnsrherbZVJ7NCEyWKQpdCi9CaXRzUGVyQ29tcG9uZW50IDggL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDQ2MiAvQml0c1BlckNvbXBvbmVudCA4ID4+Ci9MZW5ndGggMjEgMCBSID4+CnN0cmVhbQp4nO3S0VHCUAAAwQDRKKKAxKho/3VagzM372uvhb3pt+xUNpdtZUvZrWzCiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceIcwflcNpU9lq1lD2X7Mpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnziGcn2U/Zd9lT2Up564MJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIkTJ06cOHHixIlzCOelLH1jLruWbWX7Mpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4ceLEiRMnTpw4/8v5VfZStpYdy97LPspw4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDiHcL6W3csOZUvZW9m5DCdOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJcwTnH/mEWSoKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iagoxMDM4CmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagoyMiAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My45LjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My45LjApCi9DcmVhdGlvbkRhdGUgKEQ6MjAyNTA1MzAxNjU4MzgrMDgnMDAnKSA+PgplbmRvYmoKeHJlZgowIDIzCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDEzMjQ3IDAwMDAwIG4gCjAwMDAwMTE2MjggMDAwMDAgbiAKMDAwMDAxMTY2MCAwMDAwMCBuIAowMDAwMDExNzU5IDAwMDAwIG4gCjAwMDAwMTE3ODAgMDAwMDAgbiAKMDAwMDAxMTgwMSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMTU0MiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDE1MjEgMDAwMDAgbiAKMDAwMDAxMTgzMyAwMDAwMCBuIAowMDAwMDExMTU1IDAwMDAwIG4gCjAwMDAwMTA3OTUgMDAwMDAgbiAKMDAwMDAxMTAwOCAwMDAwMCBuIAowMDAwMDEwMjEzIDAwMDAwIG4gCjAwMDAwMDE1NjIgMDAwMDAgbiAKMDAwMDAxMTM3OSAwMDAwMCBuIAowMDAwMDEwMzY4IDAwMDAwIG4gCjAwMDAwMTMyMjYgMDAwMDAgbiAKMDAwMDAxMzMwNyAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDIzIC9Sb290IDEgMCBSIC9JbmZvIDIyIDAgUiA+PgpzdGFydHhyZWYKMTM0NjQKJSVFT0YK", "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2025-05-30T16:58:38.032007\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.0, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix\n", "\n", "def plot_confusion_matrix(y_preds, y_true, labels):\n", " cm = confusion_matrix(y_true, y_preds, normalize=\"true\")\n", " fig, ax = plt.subplots(figsize=(6, 6))\n", " disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)\n", " disp.plot(cmap=\"Blues\", values_format=\".2f\", ax=ax, colorbar=False)\n", " plt.title(\"Normalized confusion matrix\")\n", " plt.show()\n", " \n", "plot_confusion_matrix(y_preds, y_valid, labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们可以看到，`anger` 和 `fear`经常会被误解为`sadness`，这与我们在可视化嵌入时所做的观察相符。而`love` 和 `surprise`也经常被误解为快乐。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在接下来的部分中，我们将探讨微调方法，这会带来更优越的分类性能。然而，重要的是要注意，这样做需要更多的计算资源，比如GPU，在你的组织中可能没有。在这种情况下，基于特征的方法可能是传统机器学习和深度学习之间的一个好折中方案。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 微调（Fine-Tuning） Transformers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "现在让我们来探讨如何逐步端到端的优化一个transformer。通过微调的方法，我们不使用隐藏状态作为固定特征，而是训练它们。这需要分类头是可微分的，这就是为什么这种方法通常使用神经网络进行分类的原因。\n", "\n", "\"encoder-tuning\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "训练用来作为分类模型输入的隐藏状态将有助于避免使用可能不适合分类任务的数据的问题。相反，在训练过程中，初始隐藏状态会适应以减少模型损失，从而提高其性能。\n", "\n", "我们将使用 Transformers的`Trainer` API来简化训练循环。让我们看看我们需要设置一个的所需材料！" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 加载预训练模型" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "首先我们需要的是一个预训练的DistilBERT模型，就像我们在基于特征的方法中使用的那个一样。唯一的细微修改是我们使用`AutoModelForSequenceClassification`模型而不是`AutoModel`。区别在于`AutoModelForSequenceClassification`模型在预训练模型输出的顶部有一个分类头，可以很容易地与基本模型一起训练。我们只需要指定模型需要预测多少个标签（在我们的情况中是六个），因为这会决定分类头输出的数量：" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:38.279709Z", "iopub.status.busy": "2025-05-30T08:58:38.279286Z", "iopub.status.idle": "2025-05-30T08:58:38.353033Z", "shell.execute_reply": "2025-05-30T08:58:38.352454Z", "shell.execute_reply.started": "2025-05-30T08:58:38.279688Z" } }, "outputs": [], "source": [ "from transformers import AutoModelForSequenceClassification\n", "\n", "num_labels = 6\n", "classify_model = (AutoModelForSequenceClassification\n", " .from_pretrained(model_ckpt, num_labels=num_labels)\n", " .to(device))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "您将看到警告，说明模型的某些部分是随机初始化的。这是正常的，因为分类头部还没有被训练过。接下来的步骤是定义我们将用来在微调过程中评估模型性能的指标。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 定义性能指标（Metrics）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "为了在训练过程中监控性能指标，我们需要为 `Trainer` 定义一个 `compute_metrics()` 函数。该函数接收一个 `EvalPrediction 对象`（一个具有 `predictions` 和 `label_ids` 属性的命名元组），并需要返回一个字典，将每个指标的名称映射到其值。对于我们的应用程序，我们将计算模型的 $F_1$-score 和 accuracy ，方法如下：" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:38.354141Z", "iopub.status.busy": "2025-05-30T08:58:38.353875Z", "iopub.status.idle": "2025-05-30T08:58:38.357607Z", "shell.execute_reply": "2025-05-30T08:58:38.357065Z", "shell.execute_reply.started": "2025-05-30T08:58:38.354120Z" } }, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score, f1_score\n", "\n", "def compute_metrics(pred):\n", " labels = pred.label_ids\n", " preds = pred.predictions.argmax(-1)\n", " f1 = f1_score(labels, preds, average=\"weighted\")\n", " acc = accuracy_score(labels, preds)\n", " return {\"accuracy\": acc, \"f1\": f1}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "有了数据集和指标准备就绪，我们在定义`Trainer`类之前只需要处理最后的事情：\n", "\n", "1. 为训练运行定义所有超参数。\n", "\n", "我们将在下一节中完成这些步骤。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 训练模型" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "定义训练参数，我们使用`TrainingArguments`类。这个类存储了大量信息，并且让你对训练和评估有着精细的控制。要指定的最重要参数是`output_dir`，这是训练过程中所有工件的存储位置。这里有一个`TrainingArguments`的全貌示例：" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "ExecutionIndicator": { "show": true }, "execution": { "iopub.execute_input": "2025-05-30T08:58:38.358671Z", "iopub.status.busy": "2025-05-30T08:58:38.358371Z", "iopub.status.idle": "2025-05-30T08:58:39.661484Z", "shell.execute_reply": "2025-05-30T08:58:39.660838Z", "shell.execute_reply.started": "2025-05-30T08:58:38.358651Z" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/site-packages/transformers/training_args.py:1539: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n", " warnings.warn(\n" ] } ], "source": [ "from transformers import Trainer, TrainingArguments\n", "import os\n", "\n", "#TODO: 请大家试着调整模型参数，得到分类效果更优的模型，并试着探究不同参数对模型训练效果的影响。\n", "\n", "batch_size = 100\n", "logging_steps = len(emotions_encoded[\"train\"]) // batch_size\n", "model_name = \"./fintuned_emotion/\"\n", "\n", "training_args = TrainingArguments(output_dir=model_name,\n", " num_train_epochs=5,\n", " learning_rate=0.0001,\n", " per_device_train_batch_size=batch_size,\n", " per_device_eval_batch_size=batch_size,\n", " weight_decay=0,\n", " evaluation_strategy=\"epoch\",\n", " disable_tqdm=False,\n", " logging_steps=logging_steps,\n", " push_to_hub=False, \n", " log_level=\"error\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在这里我们还设置了批处理大小、学习率和训练周期的数量，并指定在训练结束时加载最佳模型。有了这个最后的步骤，我们可以使用`Trainer`实例化并对模型进行微调：" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T08:58:39.663394Z", "iopub.status.busy": "2025-05-30T08:58:39.662466Z", "iopub.status.idle": "2025-05-30T09:02:01.559572Z", "shell.execute_reply": "2025-05-30T09:02:01.559082Z", "shell.execute_reply.started": "2025-05-30T08:58:39.663359Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset({\n", " features: ['text', 'label', 'input_ids', 'attention_mask'],\n", " num_rows: 16000\n", "})\n", "[2025-05-30 16:58:40,072] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n", "\u001b[93m [WARNING] \u001b[0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH\n", "\u001b[93m [WARNING] \u001b[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3\n", "\u001b[93m [WARNING] \u001b[0m using untested triton version (2.3.1), only 1.0.0 is known to be compatible\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [800/800 03:19, Epoch 5/5]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EpochTraining LossValidation LossAccuracyF1
10.4776000.1854700.9260000.927223
20.1364000.1434460.9400000.940217
30.0925000.1307740.9410000.941325
40.0675000.1350430.9450000.944411
50.0441000.1552830.9420000.941754

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from transformers import Trainer\n", "print(emotions_encoded[\"train\"])\n", "trainer = Trainer(model=classify_model, args=training_args, \n", " compute_metrics=compute_metrics,\n", " train_dataset=emotions_encoded[\"train\"],\n", " eval_dataset=emotions_encoded[\"validation\"],\n", " tokenizer=tokenizer)\n", "trainer.train();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "从日志中我们可以看到，我们的模型在验证集上的$F_1$-score约为92％ - 这是比基于特征的方法明显的改进！\n", "\n", "我们可以通过计算混淆矩阵来更详细地查看训练指标。要可视化混淆矩阵，首先需要获得验证集的预测值。`Trainer`类的`predict()`方法返回一些有用的对象，我们可以用于评估。" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:01.560493Z", "iopub.status.busy": "2025-05-30T09:02:01.560292Z", "iopub.status.idle": "2025-05-30T09:02:03.005658Z", "shell.execute_reply": "2025-05-30T09:02:03.005021Z", "shell.execute_reply.started": "2025-05-30T09:02:01.560473Z" }, "tags": [] }, "outputs": [ { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "PredictionOutput(predictions=array([[ 7.6792216, -2.162257 , -2.1811943,\n", "-1.960974 , -1.8862439,\n", " -2.7976341],\n", " [ 7.6636066, -2.0327802, -2.5261304, -1.9544529, -1.6736916,\n", " -2.691135 ],\n", " [-2.6337335, 4.506431 , 4.2308693, -3.120035 , -3.6785464,\n", " -2.9058154],\n", " ...,\n", " [-2.4311984, 7.5307884, -1.8220116, -2.1549919, -2.6359184,\n", " -1.6617825],\n", " [-2.9374564, 5.9516797, 3.191363 , -3.2803156, -3.7939613,\n", " -2.8530388],\n", " [-2.616915 , 7.3969355, -2.4167778, -2.4979432, -1.948712 ,\n", " -0.8968798]], dtype=float32), label_ids=array([0, 0, 2, ..., 1, 1, 1]),\n", "metrics={'test_loss': 0.15528260171413422, 'test_accuracy': 0.942, 'test_f1':\n", "0.9417538510197682, 'test_runtime': 1.4365, 'test_samples_per_second': 1392.258,\n", "'test_steps_per_second': 13.923})\n" ] } ], "source": [ "# hide_output\n", "preds_output = trainer.predict(emotions_encoded[\"validation\"])\n", "print(preds_output)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`predict()`方法的输出是一个`PredictionOutput`对象，包含了`predictions`和`label_ids`的数组，以及我们传递给训练器的指标。例如，验证集上的指标可以通过以下方式访问：" ] }, { "cell_type": "code", "execution_count": 59, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:03.006955Z", "iopub.status.busy": "2025-05-30T09:02:03.006625Z", "iopub.status.idle": "2025-05-30T09:02:03.010920Z", "shell.execute_reply": "2025-05-30T09:02:03.010180Z", "shell.execute_reply.started": "2025-05-30T09:02:03.006924Z" } }, "outputs": [ { "data": { "text/plain": [ "{'test_loss': 0.15528260171413422,\n", " 'test_accuracy': 0.942,\n", " 'test_f1': 0.9417538510197682,\n", " 'test_runtime': 1.4365,\n", " 'test_samples_per_second': 1392.258,\n", " 'test_steps_per_second': 13.923}" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preds_output.metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "它还包含每个类别的原始预测。我们可以使用 `np.argmax()` 贪婪地解码预测。这将产生预测的标签，并与 Scikit-Learn 模型在基于特征的方法中返回的标签具有相同的格式：" ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:03.011991Z", "iopub.status.busy": "2025-05-30T09:02:03.011587Z", "iopub.status.idle": "2025-05-30T09:02:03.015151Z", "shell.execute_reply": "2025-05-30T09:02:03.014709Z", "shell.execute_reply.started": "2025-05-30T09:02:03.011972Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 7.6792216 -2.162257 -2.1811943 -1.960974 -1.8862439 -2.7976341]\n", " [ 7.6636066 -2.0327802 -2.5261304 -1.9544529 -1.6736916 -2.691135 ]\n", " [-2.6337335 4.506431 4.2308693 -3.120035 -3.6785464 -2.9058154]\n", " ...\n", " [-2.4311984 7.5307884 -1.8220116 -2.1549919 -2.6359184 -1.6617825]\n", " [-2.9374564 5.9516797 3.191363 -3.2803156 -3.7939613 -2.8530388]\n", " [-2.616915 7.3969355 -2.4167778 -2.4979432 -1.948712 -0.8968798]]\n" ] } ], "source": [ "y_preds = np.argmax(preds_output.predictions,axis=1)\n", "print(preds_output.predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "根据预测，我们可以再次绘制混淆矩阵：" ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:03.016038Z", "iopub.status.busy": "2025-05-30T09:02:03.015752Z", "iopub.status.idle": "2025-05-30T09:02:03.326061Z", "shell.execute_reply": "2025-05-30T09:02:03.325550Z", "shell.execute_reply.started": "2025-05-30T09:02:03.016019Z" } }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDE4LjM5OTM3NSAzOTcuOTMwNjI1IF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nM1ZTW8bNxDlmb+Cx+SQEYffPMZNKzSHonYE9FD0oCqya0O2aymJ2/76DldSREbLDe01DAtYe3c0+95w5g3JXU3eLb9cLpZn0xPxwwc+OVwtNhzFFR0XQoorOu4FiikdF1zS1TU3GEDHqL2ly1V+qaOHqKVTluyyvPyL83M+eUswG7ptyrkPYG13m0EwtgNGwJiZVrlJewMyku1w41dTB34nvoXUWoEz+3/rpfhN3IjJW5VioDHScZ9iEeXo7/Y3pOFqrUG7I+TFtZj8jOLdrTjlp+JuDyopaQlYQthBk4WjdKB8MdTMpClJaaj8hHJ+z+/orxRvJOEEhIgKAzkpC1olWn4y45OfUKASs/OuILOP/Hfxim3YnH1kN2xJZ5vX4g8xe89/nPFT3sXC0SF4LGM4mOoxoFUQVFsIV+yW/XtMrdADlpXOTHVqJQ1Yr41uIl8R+Re27KH3Cmws6Q+mAXprAXVopJ9T6i8o+evjALQKIHURQGaqB6DRgQype75Pf07U817yoMGU2stMA+Q2gkarsYl+wz6zNfubjks6L4pQFFSCituCRvBDgL++FjT6qLwMOn3IRINTNCa9/5BpSYq/ZAv2qTsTpIE5+5POV8d5yCYN40B7yoO3IAtTrwqS/JMHQpB+TAceIlBR0rxYRLAz9UVgNXgfujoECtMNB1HpwYxcWzCxJN+a+shTYrrm1zJV43EdeCDHIIm0IN+ZejXowHUu5G3CMPlA/2X0yoDDkn5r6h27B7dtACThSTMcQLUDD/QuApbC21p6dadBddyO4vuu8Aa6L4G9SbDUT6mLMcUTZYc9gDmjBjS2bLc18SxrbabE++1+oVv6yhX1eKnvWcP5h+MNwHXvBoA8mzYPhd/u7iqiTPFDdDThemMxndPcj9rTOkApXF8cdgx1r7MpH8TIKxzV19WdpkRSxHCRJQMWWSj1Jbfg1LwSMARqZBVN4jUYSTWJPEVec+SF49m06vgtYrFDMLT/sw8biGSyukbQdBefDI1WMaUfioYVNI3Ui/7J0Dxl+IECqI80UxPt1mhKezDYsPgHRZ9/ywdFn4ulNc7I3MtTfa7T9oTrBp0+snwVnT4BWqbT8Wi5TukpC/2YFirU1IiGTD276guxNMYZmH+Bqs91Orp8hU6b0Zp0Ohot0yl6B16Pae9cp+1otbTlampHe/65PhdLa5yR2Zen+lyn48uX6/SR5avoVCE9XreA2RadNqO1bOWeAC1XUzPa88/1hVga4wwsvkDV5zptTnhtCsx06uixctTOOZdpM1hN87lKm8FaRDoaLFdSIxiWOXsewWc6aQwzlDKR3U8N23flRZnpOR67nw90ekkUc1gDJgPukH9ht+n9C3rQDncf/opdszlbsUv23+4l4YLcbtg5+8w2ZE3novP51L0/+YcC4/vATvn/tNjqvgplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjEwNzMKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTggMCBvYmoKPDwgL0xlbmd0aDEgMTIyNDggL0xlbmd0aCA4NDI5IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nNV6eXwUVfbvuXWqqqv36k531k7S2QlbYscAYZGWJaxikLAExUkghJ0EAiigw6IkIjAJAkEQISogmxiQgQQCgmZABxl1AN/4xB+iuDBGZOaHOobk9jtVnUDQmTfz++t9Xldu1b1Vdznne7Z7KgUMAJx0EsE7aMDAbLgPRgKwznTXMyjnwVFRX6Wco/ZgAKFy0KjR/TZ8tut9APwLPb/5wP25g419M56mwe9R+8cHR6X5pv1cvAJAaqD2mEmzCkqE05YXAGQPzXHvpAXzvDAtOgvAkENtXlQyZdacexdMBzBSG/ZOKSgtAQMdYGymtmXKzIVFD50a+DqASSFyFkydXFCoTHirDCB5KT3vNpVuWLcZnqd2LbUTp86a9/ip6SH9qX2J2vtnFk8qWL3qaQ6Qspra42cVPF4ibpLnUPsbantnF8yanPJdn3qADjQ/u1BSXDrv80e/3wCQqo1/rmTu5JJehr9RtWMJ8TQVNKwsEPwJdCAk070U6ER17ZkJukJvEAZkD88F28yCebMhnHClXyAAcLum9WQzJs+dDYpW04tIM2hXBQR2SOvJ3mOc5nMD8FP6uO36+Yh+Pk/lUuAIlUtarZUiCFi1Fn+NU18e7K+P5dv5ef4aXd/VW6fgP/oF5w3M+Pd92tPwn8/8b/oEuR7Q2jpPvJ6/TdMlrX27p85VoOB/QkEbFYF2WAR+hQv/T7liJCk7qCStUPBANMRCAiRCEt1X6Q673U/TmDs/ESQ6y60tA8neqGtCfLsR7X/aaE2HJBoT7G2ilpmuFiDJg02nwQFOCAFXqz5uhX3g0vVxUcHcgolQWTB31myonDi3YBpUTiqYXUrnqZPn0nnh3JlQOWVyMdWnzJ08AyqnFsymPlMnT6Q7MwpmF0DlzIJir3YmvX56VsG8qVA5e4Z2p3hKwSyonDt/NvWcVzR7Cp2navP/C93XEZs5bUrBXfovQlD/GXTTrxJx5SI046ED2RcSjpHQUb+GQyphEUn3Baqn0FnDmLUibYLP4BvIgd4WMHyqw7acls8n2C62wp5/N6xtbTa1tT76PxC4jeZd8O/7wfhgX+16u95ujrvuj2/XnnunLvQAuFVB9d63h3YgLk3BSYJn0cYqNW2SMqRN1IwJXvF/QZHgpBnMMqIiCoJ4e0TrL6doYCH4wQuLZBd3sc2GWeyLu/pgayH/ra+4nlpMb4vwOF0jSeOQnnjJ82VAFowg5HNhCkyHElgAi3TZa8/SIbP1WQFMg5kwFx4PBAJfBD4KXAxcCJwMvBmoD9QEXg/sD7wW2Pcv9D/4C/rf6nYtb2vR6OzaWjRO06lkAOhWlkmF4g7ZChAdwaJhntNa7FRyW4tKRfMmU6g4qEyjQjGKdA2IciDONL4BNBlpOhBLRcNiEZVElgm1cJaOU7AHtrCd1Cqi+3PoTrVwEFbAfLrzNjvLVgpd6N5OuAHnqWc5nMU9IrChRPFZ6v+xJMBNlguHaI4s5mJZBplMZIR4SHxIrBW/Fs9Bd7FUPCfmi6UsA1+Wxkg7qWThH0je7xJFtewylMJRvIYZWC8OEG1wGc/hHviSVtGwOQsVsB0WEy0uVgxLhMXCQ3TnjHQONtNRTM/Psa3sPFF3lD0FF+F5FIXBsJVdJL7Owo/wFOYKSwjyDKGI6D9Dc52j8ZuhlJzaRWYCLnSie0Q9rTVRP0djF+miftyAJbRyLmyXa2WXIYFW0RDbyd5mjfI6kux5fATn4CdshZgg7hIHQ0UQAcyHCpp7szZGLmILiXftWKzNLjwm5rM9cE3MN0ykuf+gcURrHhIeIo6KoJ7KY7JKPPViK3AlUao9jYZzhqFiGo2nGQxPEtcAxZhJsi6m5/vhIHTBKqigmXR+5e7SjzRyi3iFeK5ga4Qf4RwOIF9UJF4nrMlZQRXAEYMsiSgw6OxVa4SkIYU1/pHjvO/kxXXp/IumVzV4ayCnxrrQWxsI5IwTo6S8GslTg0lKjZiUcOVfPbzSpfOwnHHempaBA1pnHZg/gO6NGkdVrUW36f7AAfozbdEaKYn+huTXeCdN9T6rPpvQ81l1cs8umv0I2o6DrFSz6k/I8PcDp7rjCDwlsAgIF9XmxnsgrTG9e4Y74ZPz5zmnEeWBL8QKkqIZwiDBHyJXO6Hasta5Otzoscegxx0VTqNu0jj16s1G9Xo6ixccqjPD53SoQooPHCokxGtnYdWWF1+kvxdfvMWM/Kdbt/hPzCjl8HP8PSrnWAYd97KMal7Ky3g5L2Vr2EK2iK3R/NMVMvHx5NFN4Pe7+2G1KFRLywxQbVRiZQ9CLDOrF4bV2HPH1VFnf4+8xoYgI76bjRca0wmrvHh2yI52UZjQPc4hZSZlOOLccZwN5ZvY5D+yoc3b94ilg2sHN13cQxOQXMWhxLEHtvpTIiKjMNzjkERwSJLYT33Jsd5a7Vorkn2DahKYyROmohytNg+rcecOqwnNfXhYjSv3YaIEAyd75DVcaDx50uHMaqXmpk6NQZW+M0jfsRqP6gjLItr8vtHiGGmMYZG4SFoQVR5hIOuPECNJDTzzYIE8P7I0ap5nOZRFLI9cHrXcswt2RTkmwIQkYiKzG3S/j2Xem5wQLxsy72MZPtHtkg0ykMs51TycYMwoeODVst+cf3zRhXHfMNfAhyP4zT179jzG1vactXHIY1X9+r93j++btx7ZURLNvyXut5C8S4n7DlDi7wruEFOZMbbMG1LttlYb18meau+6hLXyavcrqaGeEEBXhCfZq3rQFWuUUzUQQnPb+Dfq/BMANxuJS0JAbbx682qj+tV1VT8IlXTmNxbGFMQWeAvjRJjAYpjbJcbFJ6dkxhAj3YirTiwzWLmLPey79hX+Af/m0TPTc9+ZdeJM3Y79hzdsfeX5USfmlr6b9xWz/A6TYhsqP/17UtLb9/iqKp7esPOxktLFicmHvN4PDz6xV9OprcRnIelUNDzqTxAjDY4yNTqy2uCqVldahWpYZl1t2B4T5mEm9IBJlWPUZtaeO1XjrlXnVE3niFG14bpmBpodEJO8IchjCEnJoVEObhfcxZzG06cY0VLdeVznJpbIL/DvH3176viTM1774x9fG/lSrnRxD3/ObufX//o3/oPXe/ae9MNbthxOTCarrCDqq3SrTIRx/sQQGaxlFqgOlas9oTvUasvK+LWe1UmWeKMnIibEg3GxUUlkpiSKq7qhXm2+ekcIfhdFMHZOOIfnxLPSWZn4PhgjTGATWLzsdoUGaWXuriwhXsA2RhK8mlHH+UKF7c9s2/YMFWYc/sLwd87bex2ccYVJ/MbnvIVfZzksavgL2Ovoyy8dO/bSy0eFhbWJyfzv/PuxE/j3337F/6qb+US2I0aTyS6yvKkkExkm+cMlh4ACOkSyOonkgRIykYFsUJvfa3Bo+pTWzpqokB1pAhp3nLbLfpqMUkuSkaN7jzy/c5zAZIyUsqTB0hSsgRrZQOZDgmEJLG4Xnmz5/DzjLRnSxTFNy6RO2t5iFeG7Ssc3AdKgvz8pnNBNkatjulQ718asTnklPdyS2NHjTvTYjeQDyRHa46LS1eaGxpsNjTqwbRqvt7JI1duBmdSVLDYxwxeqmaqu9AnxiZn3dgtp60CaIayq3LGjsnLnDr5j+VoI/NdlvnbZc6/wn376if+0ffDap5avW7f8qbXCHzaXl29+oax88xjvwaVvfPDBG0sPeuNPV3z8zTcfV5xmBfOWL59HhTRmGXFUThyF6xqTYIiNYGUQUW3aIVbDytDYanVt6Ookg8cTFxID8fEeq64wRH6bZ/+K/9CmL6ENEW9Fnow66TkZ/VZMQ6xhj7Peec2JpDHddd12hthIVyDzXsgIakl8MmtjizC4MnzLMNKTngdnfsZvMfVzhszBD/Avh29h97XqUixpCbMy55hHmP3br1ioHhS28YdjhI1tmqTpy7v6Hvgi7fMW+mMMDiYwwUG+r59BQHhdkWRmEDxiNwNtX03ETHNjUG16N/pIHrrmiN9pJS/+YKyZsQn+9G5CD8NgYZBhmlBkWCoYZGaU3SxSzmZD5LFsnDyZTZMXyivYs/IG2plsM6u6FjEHhRPmSHiXqUJVA7/RMr1BungrVrzS1Em8ciuWsCc6xTVEp0K5Wk9/ONugwgbjMqdqUmiTLkVY+zrAYxRdRKOvWXOWum5T5PKb7e5Yd1/3b9yvuyWyR0erdiTFaW5Q7ESLsnV8zebNa3gP9s4txnjgFv+jlNby/nPlZc/t/OKTTz9v2QUs0ETrX6P1DTDUb5OFDbBMZH7ya35JUS+QL9DX86WT/Zg0+1F0+1FAabOfEDDGgkrsxRpUo99YYtxmNE5Aza8lOOJk8fuW62dbrpO3arqoWY8Ai0nXutBezURZcT3F0FhzmNEGu8PkOpvDWxZ71FOXUOtYHWaBMAy3GhVzLCqugcnE/nsXGn2+oJQart5sJrU7rduSI0vTu9np0ekx6bHp3vS49Pi+Kf5of4w/1u/1x/njc6JzYnJic7w5cTnxOSklKSuiy2PKY8u95XEr4itTqlNupMS0DW0b1DYgPyY/Nt+bH1cSUxJb4i2JWxqzNHapd2lceHsP2Id1dyRkamqdTHaaEdc+IoUKJy7vW1a8qa62tm/9M/vOttxiwqsb8w/nTj4x/r9vCBlFiyeWfnwodXjLsj1FBadePn7SuWRV1657UlKaNayOElbbZRd5Gg/08EdgncVurAt3r7bXRm2MAKdzULhFViKzozXl8N3UtzVXNR9z+nr64fyYpTHVMUh0ttkWkcp0F027L6I1xZ3gyMAvX33uuVe10vK7ngcWv0eZ+XuLD/SsqxPSzn799VkqwkOFBbye/4OO+oLCXbL2PmFO4Av8mmQYAX39UVDGnhFtZdZnTHUOsS6MhBdpcFphsGtgpNp81dfmI/jN6+oP1zW1jVKjlkZVRlVHaWqrh45W6rq7dd8QjB349YgXc944ffqNnBdHPLBjQgv/iHVh8uiXxcx9nTp9ce7cF5067UlMJIZszMl6JhBaRJU4nuhTg2hF1oHNVScpq221bCOGiaAIgxxO88BofU/q891Gq+EutMhgg8KkrSqQAFk7v4sv19b2PPDE2QAEzj5xoOUM4bZrF2GHh4VHf27cVVjABjCFjgEF3N0KXytdSwgtF0TRvikR3MxYpjwjuXczqc7CjoXXOWstqz1RbkFxKzBMcNoHenQSG/S9oQZeMCDfDHrY1L7RJdHV0R9E34iW+kJf1lfo6+4bJXU2pClpxs6mYihmxUKxuzjKOGGOBnCcvqXQsdU9LimAQQfdIC5pPmg5d2T6mYmTPpjBb/IzLLX5c2aoFXY8s7nOJjw6/sSZe+/d37Ez68FMLIT15582bDy0f6vmXdNIPf9BWIdAnt8jqcyi7JZZOWy0yfUmIYTCq1FSrHbzcJe27zNpGyGzthEaVmPT69oGqXdDc++GBqdu0FfJt6nXfU4tFh72u3Pc1W4KGW4iMpoFXUlCZoZmXMI/aiY9wNL4h3U1NfuPy65NOVMnVTSn4YcVI47t1bDmY8TxhLWZ9qhD/QkRlmijsywktM6OdckJtSn1xjr78cjo5AhQLINkp9M7MFWPy0F1aLgaVAh+UUM6i7Si49KO1R1/YUNhqnBnp9aHtaoKZTWhYZmUgO/YsH7HjvUbdtRy3lSwb+TIrQ/9/lDWwSf+1Nz8pycOZtUKfd65dOmdM5cufcs/59eiY97o3PH4mw9Pmsh6Mm0P03PipD2a5Z8ikBcSvkh+uRP55RPiAagXJKaIkK2ozRSmNNSatTCged0cYz55XrKnENJeB2Vop2rpJ+bfqpZd12i+wCd8jD6fGewwwO8xCwawnbAYyqXjUG85oCqqJD9oZYoFslV99qtZzjt7FB0MWsjhd+Q48h0ljuBCLrl1Bx5c8JXfZ98zbbi+6uqPTm4p2CR3uEaacoeT+COwUWAKZN/OJf1WVfJLOVK+VCLdoJCsk0+ky66fGzUto0BuiCZpxsN4f7LsNIbbQY42uC3l0V6sjaqPUA3gsCuKnONQ7DmecHKFCXqcbKZIqedVvXtfvdm7odHncIYRByHpiTmJJYmVidV0vJl4OTGQaCTZ6n7aTbgFhdy+kuHWH4qpA08uf/1E3dz5FTvr5j62ZmddXd+ahYv24sonFvzwecsjwtaXtpzY3lIubH35hTdfaSkX8/dPmfhEKwdiIXEQAt3IfxsBbUwutzlqLfUmJigwQotw2Xpw1913b03/dGIP5bvfdwuaP/on5BTWPvHEhn11df3emH/qtLBdI2DbVo0AWnhy4fetPme+bgdhZAchcp0T6iy1WmbutI9Ep3vgLzJzf0LfiMWwWF5iWKIsMS4xLTEvtiyxLrEtsS9RlzgWO6sjbkQ47t7135XAl67ft3fDun371t1gTn79xt/498yBl79+992vv3nnzLUt/B3eyL8jB5NFfsTFemixjSx1O1Goeev7/FFt3rrWtpodx/po8tSDdJ/dLrpR+tTmsP3GoMf+LEZkE5JuQ9Ma2u4KeaV1dXcim9CjLd7tatkvm/a0i23s2zaXHZQbDiXqHEApkGwmPTNjua3WWG8wybT5yXZqbkO3QfLTF97THPOhnJBtIZrEghHtjrjCcGjskM5bXiU6jq4I6erBQ07H2RMtB0lYRZMkiVYrpnh6hlZLga/9va0WwWYeFRujGAWDaVRsbEw/kzkmVnRTnF0pusrcK8O1OJtEcbZDjMkcG2WAh6IUm0FxxQ/soFF1ofGqZrBZWW2B9wct8Go6pW9rbd/RJtKgn2lvCyna3naWx+QxeyxdKXx0Nne29DL2MvUy97KYveBliUIHUwdzx5A0V5q7Y2iHmA6xqd7UuMSUMlOZucxSZnVqb4kFQTbJZrSgFW1oRxUjMBKj0CNGG1PSUvum/iZ1SerS1MrU6tQbqeG0MZ5zJ+7H6pm9nNA++U1jWhLUjbDDVSN2jV+5cuL6vg07fvrL+LdnFp0uWL568l7/3uc/+1PRIbHv/g4dcnP9Q+JsHTet3HI4IeFEZmbeyGE5SfbEDcu37tMzx+7kAP8ubSUbpF2BTVLsuBscrF4pN5kJY9Ix1WnTbFAPSD49HjUGvQa5vYOvu5lmhVoUcoX20mJScqYWjRzsMbaYrxhWevz4xZfLy6Wt/K2KluqVIzZv+7OQX8Hu07z4frLCcbr1u6CX33PH/lebWL2r1kLW7zKPID+Q7dbMMSuoUVd9t51Asfuk5gRCaFMSNLvbu5Nktl9zAq/V1vY/MP/UO+x9dlTY2VKwbduJ7cLiW9X7iibdwF0a933IAy0R8ylvvuVP0VNmgZIhSbugIIPMHAByP0qK3pRkCQUmiWDQ3pjpQRuCQduVq73B0l5tgP7iJiz4zirrToqttBY91fb/brAwXVgsLBHKhKXCWmG7oGgLGdFIWkw5E0aKyZDMUjFV9CqZkMl6Yk8xXckGyqVwiJgtDZb9yhgYw/IwT8xRiqCITcNp4hRpqpyvzId5bDEuFudLi+QVsIKtxJXiSqlMroIqtlHYjM+Lz0sb5V3Sq3KNclK5rASU+7RMLMPIMlhCn7fZo+zRt/kjTWJ+cy7uu1VNCPUihBYSQmbW358taUmi6EDRoF0kkfJGdAgCMzu0/+05jCamXcwmg2IwOhTF0M9kEJmoEHpCa42YtQQB1F4BPTysRtVODh0+uQ1PrR58B9gQzOko5ob9Uzz/Gb7Pm0TRFCm6TcmmPuI9ptHiWMM4U5FpAVskLjDMM60Rl5s2idvEjYbnTJWmnWy3+Lq4w/CKqdrkMaEoSUaTORLdktsYaU7FZCnJ2NHstfZkWdhdutfQzZhlTrcOwWxpoHGo2W/N0+Qg5OFYaYycZxijjDHmmXOsxdbH2RLrC2y9YS/bbqixvm+9bA1Y0whpo5BgZPRHgIuFfAbb8zE/yo9+zN7gcz9mqSxVzG+53HKK1fLBwlAhlM9hFbqWUjTQtNTOVvn7GxTB6AC7BjOA3eawg93qsFhBu9isJrPJ4jCbTf2sZqMKZqkcj9vM9arNajEZZQTFLtrNapsAFB12czvYzcFXjzrqKplaY2sG+U+hV7R3sWE+DfMbMkiKbERrqCnMqloTrJnWIaYHTSOs443jTdNN5dal1nVWpwmICLNkMdvM9jDmFlRRlcJMLrPLEmmLtKdAIvlTr+iVUpUOxiRTojnRkmLtaOto9zq6kx1kCuliutTD1M3czdLDmmXLsqc77gc/8wt+9It+yS/7DX6ln3GgaZB1iG2I3e/IhZFspDAac8Qcks9oks9Y41jTaPNoS54tz57jKGJFwlTTNNs0e75jsfK47XH7SnjWuMK8wrLSutK20r7JuMG8wbLZttm+3bzdste2117jeN9x2RFwTCZZSjYW3OL1ZZo8M4R1I9Y/sW7m8NyMON4raEpT31m0eXBZrjiieT3O1CQ5jiLnJyRJI7zoj1SC7+nIWPopu6Eed0sKMhCZbFIvDasxk4AsQbsIvjENyW2zkQZf63u8xl+9yPP3k4RQIVkYJAwxSGbFbg7HKKWT4jV3wywl3ayhNVBHq78yFvOU35jzWb5QhPlivjRRWWJean7dHNX6hk97R87i5uD0luHCoeYnhUMtk8X8Xc2frNuFSa1ZjrSH9q6JMNgfkqwnNZa4cGuM4rDEqa7hSVrM8GlRQ+2t5TIN94DfYbQ6djuFyHII3yjHOuvN9rTeX/l8vPd1HyU4vvS7kpo7iY0eWAzaAy3uSXvashxu0BOd/TWTUpLZz3dlPG1Zz6YOHaZOCmY/adp/J4heN2XmD/kdUdkQpoTaXaKiYKhJHh55h17em/ZTfqdCoVAtt4WfCD1g22iEeolp1F7n+vtfH+35A5SmV1K6rurJ+q9JJooZ7f7FoUFKX3ujTqP857o6Lfdoo/HI6xrR7OC1Vkxbaezut4dlU0ZjsiiKKjptw0M1+oLkadRR/rjbKFJS6TDWWwWNMK5TxXTofpkakjeBOn7kTnaoJRHS4l/kh7S63Eyrp0IXyinC07LDOikd1Si3EtnRCLGykhhjjE8e3vUOUA0+7dyswxUWFZuwO9FBeW6XEx0PqLAx1JBYHxEdp6UZPp/mS9RGH/0Fpdwqze7dut+Gqk3m7RJaidDTklpNvGM9KSOeIlE/IBzU4GyVPRKaJPaglEd7QlI0MNvAbWNNaMNW5y4ZRvhDO2RbFTU03KWoRvDISlyUMTZheEo7znTGdDUI93h3xzmEckvyRrchrt4eGRNk6WbvX/PTLeMX8P8iOw/qaXtZtPJxm4e97eVyWzbBbw2EvE37wh/4+2/svX+AWEX/x/6Hv7Vdbbv+9FHzcFueUfsaR7n9HQCNM8zi0QA2/tNHTSNteb/6aqCTeE7/3zsIWnK9Si+fSGFQTuUKlSoqW6hspVJBZReVVVSWSc3wrnieyheBJvFrWCy54KhYBHPoOkdshDnCh5Cm1YUsOCVkBT7RroazcFRyUr8v9X5HcSjVO0ExJkB3ur9frIc+VHrp1wwYJz1Fc5iCxVBPMky7i3YXjIE1zMKWssuk4hXCP3AiHsYfxa5ig6RKD9M+xyFPkE/J1wzJhgGGHYYrik9ZptQapxi3G78yPWbOssRYnrZcsna1zrJesvVqxbkz5kInmAoWsgkVNmloim4hlK7aNwEGGK/9H1g0Uud0/XsKrc4glFrBugAKy26tY7v7Yru6BOFsRGtdBhcrgv5QDCWwEObCNJhCq88DL3SASWSNXvBBOh0ZVJtIPbzQj/rMg1Iqc2EyFMAs6Ex3h8Bs6t+VavfDTDq88NDtuUr11mS6TqYxC+hcSD1N/8Gq3W6vqn2VsoDW0v7zP5t6a3QU0Jj/2YoDqDadxo2B+dRjEvUt0GebrI8o0Dny0iyz6VxCfSbSvNOon5fGF9PqBfqzX84zSp+llCgqpmMG3dVWLaW+xfpMPv3Lm8y7RrWNEYLKFPit/i3Rr3+ddb3Qvitz6NmKW/+6KgIiIYry83to3u4070DIhkEwmKQwFIbDA/Ag5MBIQmAUjKbVxlLsz4OH4RGYwAQ4Cae0N1qG+bOnZfdLT2+9ZrRe7wWoFZb6A7c4Nrnw5yT8hw9/qsIfbfgDx5sc/zsJ/27Dv1XhjST8/tn7pe85Xq/C76qwsQm/bcK/crzWE7/ph19z/MqHX14dJX1ZhVep49VR+MXnadIXTfh5Gl7h+BnHyz78Lxd+WoWXOH7ixP/9JH58DP/C8SPq/tGTePHCIOnik3hhEJ7/c5R0nuOfo/BDjh9wfJ/jnzieq8L3zsZI73E8G4N/9OG7HE+vcEinPfiHUGzg+DbHtzie4niS45scT3A8zrGe4zGORx1YV5Yk1XGsPXJMquV45PAE6cgxPLJUPPz7JOnwBH8AD/vF3yfhIY5vVOFBjgc41nB8neP+QnzNhvv2Jkn7CnHvHqe0Nwn3OHE3Eb27CXdxfJXjTo47nLid4ysv26RXfPiyDV8qxGrqUl2F2zhufdFCeSm+aMEtL0RIWwrxhc2q9EIEblZxkwmf57ixyipt5FhlxQ00aEMVrl9nk9Z3wHU2fK4J11Yek9ZyrKyYIFUew8qlYsXvkqSKCVjhF3+XhGs4rl7VVVrNcVVXfJbYfPZ+XPmMWVrpwmfMWE43yguxjJAqS8IVDnya41PLHdJTHJc7cBnHpRyXcPQHfvvkk9JvOT75JD5RiItz3dLiJFzEcSHHx234mAUXmHA+x3lNWNqEc5twThOWcCzmOJvjzDicwXG6o580fRRO4zj1SZxCjSKOkzkWcpzEcSLHgp6Y34SPWnACx4c5jueYN84k5TXhOBOODY2QxvpwDMfRtPLofpjrxlFMlUaF40MuHDk0RBrJMceMD3Ic8YAqjeD4gIrDOQ6jJ8M4Dh2iSkNDcEi0VRqi4mArDuKYXYUDq3AAx/5CF6l/E/Y7hvcPQz/Hvhzv6+OU7nNhn952qY8Te/eySr39ATv2smJPjlkce3R3ST2asHs3Veruwm6ZZqmbiplmvDcGM6zou8cs+TjeY8b0NLOUbsU0M3btYpS6qtjFiJ192KljktSpEDumOqWOSZjqxA4pSVKH+zElCZOTzFKyHZPMmMgxgWO8HeOIzzgnegsxtgljiIWYQoy2oocQ9HCMasLIfhhBjQiO4YUYRkiFcQylQaER6Obo4hjC0UkdnJwy5C6Sox+qT6K9EG0crZZQycrRQr0toWjmaFLRyFGhbgpHgwvlQhTpoUga4Ea6ixwFagtdkKkIHFktK1yxhnX6/+EH/68J+L/+ov8P7mxSwwplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9MZW5ndGggODMgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicpYxHCoQAEAQLXHXNOef4/yfaB6/KwjZUU8MMA3/GeNl9hImFzVfm4OLhE/z8OxTR7fHjVUJKRk5BSaW5plG3dPQMjPKJmYWVTb5zcF517ALjCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0xlbmd0aCAzNTcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicXVLLboMwELzzFT6mh4iEh2kkhFSlFw59qLSnKgdiLxFSMZYhB/6+xmNTqUhkNbMz3iHe+Fw/16qfWfxuRtHQzLpeSUPTeDeC2JVuvYqOCZO9mD1yv2JodRRbc7NMMw216saoLFn8YZvTbBa2e5LjlR4ixlj8ZiSZXt3Y7uvcgGruWv/QQGpmh6iqmKTOHvfS6td2IBY7876Wtt/Py97a/hSfiyaWOHxEJDFKmnQryLTqRlF5sE/Fys4+VURK/uvbz3G2a7fpk1WP8o16cTSBJk8T6BTqNAUd4BElCU1oc8CT13rIUQqUx6BxlgxTMz8181NzjMl9xtxnzDPQmacz0BxpeAE6QITjKQqsHJk4Dw4ccAJs/QEbRFc4WPgwHnKJgg/gXdA4S4HZBYclQEQpEKXIgwYW/DGFzxDgKbCX9XbDNa4XvW7ltkXiboxdILe6bnPWnekVbdutR7261vcXxDvMWQplbmRzdHJlYW0KZW5kb2JqCjE1IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9TdWJ0eXBlIC9DSURGb250VHlwZTIgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucwovQ0lEU3lzdGVtSW5mbyA8PCAvUmVnaXN0cnkgKEFkb2JlKSAvT3JkZXJpbmcgKElkZW50aXR5KSAvU3VwcGxlbWVudCAwID4+Ci9Gb250RGVzY3JpcHRvciAxNCAwIFIgL1cgMTkgMCBSIC9DSURUb0dJRE1hcCAxNyAwIFIgPj4KZW5kb2JqCjE2IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9TdWJ0eXBlIC9UeXBlMCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zCi9FbmNvZGluZyAvSWRlbnRpdHktSCAvRGVzY2VuZGFudEZvbnRzIFsgMTUgMCBSIF0gL1RvVW5pY29kZSAyMCAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9Gb250RGVzY3JpcHRvciAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zIC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Bc2NlbnQgOTI5IC9EZXNjZW50IC0yMzYgL0NhcEhlaWdodCAwCi9YSGVpZ2h0IDAgL0l0YWxpY0FuZ2xlIDAgL1N0ZW1WIDAgL0ZvbnRGaWxlMiAxOCAwIFIgL01heFdpZHRoIDk3NCA+PgplbmRvYmoKMTkgMCBvYmoKWyAzMiBbIDMxOCBdIDQ2IFsgMzE4IF0gNDggWyA2MzYgNjM2IDYzNiA2MzYgXSA1MyBbIDYzNiA2MzYgNjM2IDYzNiA2MzYgXQo3OCBbIDc0OCBdIDgwIFsgNjAzIF0gODQgWyA2MTEgXSA5NyBbIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSBdIDEwNQpbIDI3OCAyNzggXSAxMDggWyAyNzggOTc0IDYzNCA2MTIgNjM1IF0gMTE0IFsgNDExIDUyMSAzOTIgNjM0IDU5MiBdIDEyMApbIDU5MiA1OTIgNTI1IF0gXQplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAwIC9jYSAxID4+Ci9BMiA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggNDYyIC9IZWlnaHQgNDYzCi9Db2xvclNwYWNlIFsgL0luZGV4ZWQgL0RldmljZVJHQiAxNgoo9/v/9fn+9Pn+8vf98ff93+v3CDRxCEaMCDh39vr+8Pb87fT77PT74Oz3CE6XXHJYoQgwaykgXQovQml0c1BlckNvbXBvbmVudCA4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyA0NjIgL0JpdHNQZXJDb21wb25lbnQgOCA+PgovTGVuZ3RoIDIxIDAgUiA+PgpzdHJlYW0KeJzt0tlRGEAQQ0HANz5w/tESAXy4VJaY6hfBanof/iZ7mO1xtuxOnOWyO3GWy+7EWS67E2e57E6c5bI7cZbL7sRZLrsTZ7nsTpzlsjtxlsvuxFkuuxNnuexOnOWyO3GWy+7EWS67E2e57E6c5bI7cZbL7sRZLrsTZ7nsTpzlsjtxlsvuxFkuuxNnuexOnOWyO3GWy+7EWS67E2e57E6c5bI7cZbL7sRZLrsTZ7nsTpzlsjtxlsvuxFkuuxNnuexOnOWyO3GWy+7EWS67E2e57E6c5bI7cZbL7sRZLrsTZ7nsTpzlsjtxlsvuxFkuuxNnuexOnOWyO3GWy+7EWS67E2e57E6c5bI7cZbL7sRZLrsTZ7nsTpzlsjtxlsvuxFkuuxNnuexOnOWyO3GWy+7EWS67E2e57E6c5bI7cZbL7sRZLrsTZ7nsTpzlsjtxlsvuxFkuuxNnuexOnOWyO3GWy+7EWS67E2e59mXe6Wuy52Tty3zIcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngrnqXCeCuepcJ4K56lwngpnu8dkX5L9SfYjWdvsnXDixLkaTpw4V8OJE+dqOHHiXA0nTpyr4cSJczWcOHGuhhMnztVw4sS5Gk6cOFfDiRPnajhx4lwNJ06cq+HEiXM1nDhxroYTJ87VcOLEuRpOnDhXw4kT52o4ceJcDSdOnKvhxIlzNZw4ca6GEyfO1XDixLkaTpw4V8OJE+dqOHHiXA0nTpyr4cSJczWcOHGuhhMnztVw4sS5Gk6cOFfDiRPnajhx4lwNJ06cq+HEiXO1z8mifyM683uyp2TRmThx4sSJE2csnDhx4sSJMxZOnDhx4sSZCidOnDhx4oyFEydOnDhxpsKJEydOnDhj4cSJEydOnKlw4sSJEyfOWDhx4sSJE2cqnDhx4sSJMxZOnDhx4sSZCidOnDhx4oyFEydOnDhxpsKJEydOnDhj4cSJEydOnKlw4sSJEyfOWDhx4sSJE2cqnDhx4sSJMxZOnDhx4sSZCidOnDhx4oyFEydOnDhxpsKJEydOnDhj/UqWfVqyT8m+JXtOhhMnTpzdcOLEuRpOnDhXw4kT52o4ceJcDSdOnKvhxIlzNZw4ca6GEyfO1XDixLkaTpw4V8OJE+dqOHHiXA0nTpyr4cSJczWcOHGuhhMnztVw4sS5Gk6cOFfDiRPnajhx4lwNJ06cq+HEiXM1nDhxroYTJ87VcOLEuRpOnDhXw4kT52o4ceJcDSdOnKvhxIlzNZw4ca6GEyfO1XDixLkaTpw4V8OJE+dqOHHiXA0nzg/I+ZTsZ7L2D3i738lekuH8h3DixLkaTpw4V8OJE+dqOHHiXA0nTpyr4cSJczWcOHGuhhMnztVw4sS5Gk6cOFfDiRPnajhx4lwNJ06cq+HEiXM1nDhxroYTJ87VcOLEuRpOnDhXw4kT52o4ceJcDSdOnKvhxIlzNZw4ca6GEyfO1XDixLkaTpw4V8OJE+dqOHHiXA0nTpyr4cSJczWcOHGuhhMnztVw4sS5Gk6cOFfDiRPnajhx4lwNJ06cq+HE+V84XwGRZ652CmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKMTMyMwplbmRvYmoKMiAwIG9iago8PCAvVHlwZSAvUGFnZXMgL0tpZHMgWyAxMSAwIFIgXSAvQ291bnQgMSA+PgplbmRvYmoKMjIgMCBvYmoKPDwgL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuOS4wLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuOS4wKQovQ3JlYXRpb25EYXRlIChEOjIwMjUwNTMwMTcwMjAzKzA4JzAwJykgPj4KZW5kb2JqCnhyZWYKMCAyMwowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxMzM0MSAwMDAwMCBuIAowMDAwMDExNDcyIDAwMDAwIG4gCjAwMDAwMTE1MDQgMDAwMDAgbiAKMDAwMDAxMTYwMyAwMDAwMCBuIAowMDAwMDExNjI0IDAwMDAwIG4gCjAwMDAwMTE2NDUgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ0IDAwMDAwIG4gCjAwMDAwMDE1MTMgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAxNDkyIDAwMDAwIG4gCjAwMDAwMTE2NzcgMDAwMDAgbiAKMDAwMDAxMDk5NiAwMDAwMCBuIAowMDAwMDEwNjM2IDAwMDAwIG4gCjAwMDAwMTA4NDkgMDAwMDAgbiAKMDAwMDAxMDA1MSAwMDAwMCBuIAowMDAwMDAxNTMzIDAwMDAwIG4gCjAwMDAwMTEyMjAgMDAwMDAgbiAKMDAwMDAxMDIwNiAwMDAwMCBuIAowMDAwMDEzMzIwIDAwMDAwIG4gCjAwMDAwMTM0MDEgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSAyMyAvUm9vdCAxIDAgUiAvSW5mbyAyMiAwIFIgPj4Kc3RhcnR4cmVmCjEzNTU4CiUlRU9GCg==", "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2025-05-30T17:02:03.083571\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.0, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "

" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "[0 0 1 ... 1 1 1]\n" ] } ], "source": [ "plot_confusion_matrix(y_preds, y_valid, labels)\n", "print(y_preds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这个矩阵更接近理想的对角混淆矩阵。`love` 的类别仍然经常被误认为是`joy`，这似乎很自然。`surprise`也经常被误认为是`joy`，或与`fear`混淆。总的来说，模型的性能似乎相当不错，但在我们结束之前，让我们深入了解一下我们的模型可能会犯的错误类型。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 错误分析" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "在继续之前，我们应该进一步调查我们模型的预测。 一个简单而有效的技术是通过模型损失对验证样本进行排序。 在前向传递标签时，损失会自动计算并返回。 这里是一个返回损失和预测标签的函数：" ] }, { "cell_type": "code", "execution_count": 62, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:03.327040Z", "iopub.status.busy": "2025-05-30T09:02:03.326776Z", "iopub.status.idle": "2025-05-30T09:02:03.331155Z", "shell.execute_reply": "2025-05-30T09:02:03.330690Z", "shell.execute_reply.started": "2025-05-30T09:02:03.327019Z" } }, "outputs": [], "source": [ "from torch.nn.functional import cross_entropy\n", "\n", "def forward_pass_with_label(batch):\n", " # Place all input tensors on the same device as the model\n", " inputs = {k:v.to(device) for k,v in batch.items() \n", " if k in tokenizer.model_input_names}\n", "\n", " with torch.no_grad():\n", " output = classify_model(**inputs)\n", " pred_label = torch.argmax(output.logits, axis=-1)\n", " loss = cross_entropy(output.logits, batch[\"label\"].to(device), \n", " reduction=\"none\")\n", "\n", " # Place outputs on CPU for compatibility with other dataset columns \n", " return {\"loss\": loss.cpu().numpy(), \n", " \"predicted_label\": pred_label.cpu().numpy()}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "再次使用`map()`方法，我们可以将这个函数应用到所有样本上，得到损失。" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:03.332041Z", "iopub.status.busy": "2025-05-30T09:02:03.331808Z", "iopub.status.idle": "2025-05-30T09:02:06.047405Z", "shell.execute_reply": "2025-05-30T09:02:06.046918Z", "shell.execute_reply.started": "2025-05-30T09:02:03.332021Z" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4b97e327195a4f37a3db7c046eb4f620", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map: 0%| | 0/2000 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabelpredicted_labelloss
1801i feel that he was being overshadowed by the supporting characterslovesadness9.555587
177im sure much of the advantage is psychological the feeling ive out clevered the competition who are now hopelessly burdened with their big chainring jumpsadnessjoy9.286954
1322i hate being the party girl because i feel like such a hypocrite because i always hated themsadnessanger8.678489
1509i guess this is a memoir so it feels like that should be fine too except i dont know something about such a deep amount of self absorption made me feel uncomfortablejoyfear8.167973
318i felt ashamed of these feelings and was scared because i knew that something wrong with me and thought i might be gayfearsadness7.644805
929i feel food smarter already and slightly annoyed calories counting is so annoyingangerjoy7.544228
1304i feel not just attracted to but completely enthralled and captivated by him like hes some kind of other worldly creature with nothing inside him but a really bright lightsurprisejoy7.508989
1124someone acting stupid in publicangersadness7.204728
1581i feel stronger clearer but a little annoyed not quite sure whyangerjoy7.154798
1683i had applied for a job and they had assured me that the exams would take place a few months later a week later i went to obtain some more information and they told me that the exams had already taken placeangerjoy6.923202
\n", "" ], "text/plain": [ " text label predicted_label loss\n", "1801 i feel that he was being overshadowed by the supporting characters love sadness 9.555587\n", "177 im sure much of the advantage is psychological the feeling ive out clevered the competition who are now hopelessly burdened with their big chainring jump sadness joy 9.286954\n", "1322 i hate being the party girl because i feel like such a hypocrite because i always hated them sadness anger 8.678489\n", "1509 i guess this is a memoir so it feels like that should be fine too except i dont know something about such a deep amount of self absorption made me feel uncomfortable joy fear 8.167973\n", "318 i felt ashamed of these feelings and was scared because i knew that something wrong with me and thought i might be gay fear sadness 7.644805\n", "929 i feel food smarter already and slightly annoyed calories counting is so annoying anger joy 7.544228\n", "1304 i feel not just attracted to but completely enthralled and captivated by him like hes some kind of other worldly creature with nothing inside him but a really bright light surprise joy 7.508989\n", "1124 someone acting stupid in public anger sadness 7.204728\n", "1581 i feel stronger clearer but a little annoyed not quite sure why anger joy 7.154798\n", "1683 i had applied for a job and they had assured me that the exams would take place a few months later a week later i went to obtain some more information and they told me that the exams had already taken place anger joy 6.923202" ] }, "execution_count": 65, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.set_option('display.max_columns', 1000)\n", "pd.set_option('display.width', 1000)\n", "pd.set_option('display.max_colwidth', 1000)\n", "\n", "df_test.sort_values(\"loss\", ascending=False).head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TODO: 分析：分类错误的样本为什么会被分错？" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们再看看损失最小的数据样本：" ] }, { "cell_type": "code", "execution_count": 66, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:06.255433Z", "iopub.status.busy": "2025-05-30T09:02:06.255248Z", "iopub.status.idle": "2025-05-30T09:02:06.262402Z", "shell.execute_reply": "2025-05-30T09:02:06.261942Z", "shell.execute_reply.started": "2025-05-30T09:02:06.255413Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabelpredicted_labelloss
1600i enjoy going to churches acquired there feeling is always so peaceful and tranquil thats why ive had a wish to visit pochayiv monastery and without comments it was really worthyjoyjoy0.000243
1569i obviously wasn t feeling particularly friendly and neither was i but we said yes anywayjoyjoy0.000244
71i must say to get to this point where i feel nothing but just friendly feelings towards him takes alot of timejoyjoy0.000245
798i said as five years of pain and futility lifted from my shoulders and took wing around me in angelic style i feel all jolly againjoyjoy0.000245
1819i feel like texans are some of the most friendly and genuinely kind people you will ever meet but i still have my issues of old attitudes that just don t change nearly quickly enough for my tastejoyjoy0.000250
4i can have for a treat or if i am feeling festivejoyjoy0.000251
1815i don t feel i need to stop being festivejoyjoy0.000252
1730i know i sure do and i m feeling festivejoyjoy0.000252
540i think im mad at myself for just feeling this jaded after only five months of nursingsadnesssadness0.000252
787i nearly barfed on the day before came inside to ask me how i was feeling and as i assured her i was better and it was most likely something i ate she winked at me and said well you know there is something else that can make young women sick like that as welljoyjoy0.000252
\n", "
" ], "text/plain": [ " text label predicted_label loss\n", "1600 i enjoy going to churches acquired there feeling is always so peaceful and tranquil thats why ive had a wish to visit pochayiv monastery and without comments it was really worthy joy joy 0.000243\n", "1569 i obviously wasn t feeling particularly friendly and neither was i but we said yes anyway joy joy 0.000244\n", "71 i must say to get to this point where i feel nothing but just friendly feelings towards him takes alot of time joy joy 0.000245\n", "798 i said as five years of pain and futility lifted from my shoulders and took wing around me in angelic style i feel all jolly again joy joy 0.000245\n", "1819 i feel like texans are some of the most friendly and genuinely kind people you will ever meet but i still have my issues of old attitudes that just don t change nearly quickly enough for my taste joy joy 0.000250\n", "4 i can have for a treat or if i am feeling festive joy joy 0.000251\n", "1815 i don t feel i need to stop being festive joy joy 0.000252\n", "1730 i know i sure do and i m feeling festive joy joy 0.000252\n", "540 i think im mad at myself for just feeling this jaded after only five months of nursing sadness sadness 0.000252\n", "787 i nearly barfed on the day before came inside to ask me how i was feeling and as i assured her i was better and it was most likely something i ate she winked at me and said well you know there is something else that can make young women sick like that as well joy joy 0.000252" ] }, "execution_count": 66, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_test.sort_values(\"loss\", ascending=True).head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "分析：分类正确的样本为什么会被分对？" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 保存和分享模型" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "我们还可以使用微调过的模型来对新推文进行预测。由于我们已经将模型推送到了 Hub，现在我们可以使用 `pipeline()` 函数来使用它，就像在<>中所做的那样。首先，让我们加载 pipeline:" ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:02:06.267971Z", "iopub.status.busy": "2025-05-30T09:02:06.267636Z", "iopub.status.idle": "2025-05-30T09:02:06.320493Z", "shell.execute_reply": "2025-05-30T09:02:06.319948Z", "shell.execute_reply.started": "2025-05-30T09:02:06.267951Z" } }, "outputs": [], "source": [ "from transformers import pipeline\n", "import os\n", "\n", "model_id = os.path.join('./', 'fintuned_emotion', 'checkpoint-800')\n", "classifier = pipeline(\"text-classification\", model=model_id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "使用以下pipeline可以直接对任意推文进行情感分类：\n", "\n", "TODO：构造5条对抗样本，尽可能的让你的模型无法正确的判断其情感类别，并试着解释原因？" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "ExecutionIndicator": { "show": true }, "execution": { "iopub.execute_input": "2025-05-30T09:06:21.971131Z", "iopub.status.busy": "2025-05-30T09:06:21.970770Z", "iopub.status.idle": "2025-05-30T09:06:22.016457Z", "shell.execute_reply": "2025-05-30T09:06:22.015909Z", "shell.execute_reply.started": "2025-05-30T09:06:21.971105Z" }, "tags": [] }, "outputs": [], "source": [ "custom_tweet = \"Just found out I won a free vacation… too bad it’s to a deserted island with no WiFi. Life’s hilarious!\" # 修改这句话\n", "preds = classifier(custom_tweet, return_all_scores=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "最后，我们可以在条形图中绘制每个类别的概率。显然，模型估计最有可能的类别是喜悦。" ] }, { "cell_type": "code", "execution_count": 80, "metadata": { "execution": { "iopub.execute_input": "2025-05-30T09:06:23.306187Z", "iopub.status.busy": "2025-05-30T09:06:23.305848Z", "iopub.status.idle": "2025-05-30T09:06:23.548235Z", "shell.execute_reply": "2025-05-30T09:06:23.547715Z", "shell.execute_reply.started": "2025-05-30T09:06:23.306163Z" }, "tags": [] }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNzQxLjQ2MjUgMjcxLjQyNTYyNSBdIC9Db250ZW50cyA5IDAgUiAvQW5ub3RzIDEwIDAgUiA+PgplbmRvYmoKOSAwIG9iago8PCAvTGVuZ3RoIDEyIDAgUiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJy9l8tOZDcQhs/ab5CdM1IiWIy7qnxfDpMMCsqGDNIsZmbRQHNT0x1oApq3z283cHxEc3pWIJlj/1TZn29VZvLH7P7yZPbP/p7++FlN+tbJSrG+QjnXpK9QHjTrfZRzRWhdq+jYuCAejXnfkIiK+FKdw3DQvFDqTE0+oIsVnPaVErLGWi5u3iR8rpW3yXArzVvJReNT0RrPZ612f6NfdGqtM0mLsIlB3870F73A8GScs2yjeJfRiOsaib7FhNXIX4HNyaDlrPTcEsh4S0FsDz7UMnvrXZEH7s9yxT/UbzIB7AqJz8L9BGBsiCKz9BNoNRuLS8FvnJ/FN4S3Yo0ncS178MYlG4gb9qEWQsipyAPvJ/Ut8SP2PKaUc8/vOJociHKP30jBcEpeKn3j3MhviO8kGedxJVODH+GC9Y4t/0DLHmcoFnng/iy/5QQSmWw5xdgEHRHcQ9zU1ESdRivMTLbwN969+ow/+SDr4HaOQImgCfSHEuqKosRm3HsiaUZuNTG2jqz2EHUf1A1+k35P6EnYG46SbM6Svc4m6pNrtXekJp9Y43IendWYfHSqvuqdbtVNu9Nu0c1QW+3q7/roQP15pA5VxVGSxRAHz6HBaLQRjGQNtiZldh4LsAXjqlt2P14Ob50zLluiJui32uvD41yYTJFCzMnnbcPPMfx9N9sAkANuVna5BWi0EYCIzaLkfSRGANkCMMUWnGMTbl8iOJdxRH1OrrlEjfY6gkPEdVkSMp5DWtmCcIbhp5sAPGEpBanTNpeg0UYAYjTRJ8fIC/QTR/G/7rb7F+US9Q2bseEFwDmbNHwBvMRA6DMhU31xsJHyHcGgsYFDMhEH2stg7F7dPDxyIIycYS+EUEHejgHIOAIzm0BYUzdgaOQRCCYkYccwhGUYo3BbKHy9hYjZQ4peHqPA8UVgZB8C+zRGEbZQ5GhssoLMN6Do5TEKxLEkOJkpCfMYRRqnEEdGosTAwxPZyxspojWOMrI0nkYeB4gklOLGSLijIcuNKn29L71yQFxadxidCanWRvr6iLA3LWG/0+XW7SKfgvzpBwbL7hgGx7iOc5S77gcMv+10v3XfdlsE0QfrV39NYMP/CjZc2U2vcfX55Vv+euNbHpY/94/A0PDR/9U+qc5hnYi5puHzdq+iKQ9dW+2bFcVZb9a0Luq77gBhbIXV0oioS9QXyK+61or2F8oDWgt8p9WmLDweLMR45+Anqx0E4hn+dA+DE5Q7LD4c9O/Q7lBb4ntc07Yu26J/qTt4V/XS5WlN5sgl0GaPVqu612uUh7qZF6gtqs+XXTUAwCwuu0+7Gk/3gKi2PhKqiAbWf+OLVPE46kU9HNMas5d14r9273A41NPhOFT/A/Jjy7sKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoxMDMxCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE3IDAgb2JqCjw8IC9MZW5ndGgxIDEzNzY0IC9MZW5ndGggOTI3NyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJzVenl8FFW28Ll1qqqr9yXd2ZfOnrAlJAQIoLQRwipGVkHRhISwKCQQUDEyYZHECA4gEAQVMk5QCIgRGUkgMi64IDCjD5gZB964RdExIuMwOkJy+51b3QmBee/NzPvn+31VfevWvXXvuWc/91QXMABw0UUG76gRI/NhLMwDYH2oN3pUwe2T4hel/4naowGwYNSkKXl1ybu/p8FP0fPLt90yeXSsfcaHAMo6GjPx9kkZWfOu2PsDqJ/Q86nFC4rKk0IObAUwDKbnZ4ofWOKFeTG5AEYLtXlp+ZwFiwY8MB/ATG3YO6eoohwMdIK5mdqWOfcvK33763DRPgbgDZ87u6hEm/lmNUDv8/R84FzqsO40NAD0cVM7ae6CJQ9991nMcmrTevCn+8uKiz5K+zANoG84tRcvKHqoXN6lLqI24QvehUULZqd+e1MrtfcJ/MrLKpacz7yN5masoOd/LF88u3yo4S90m3k30TgXBK8EpuKQ6ETIor5sGET34pkJ+sEwkEbkj58MtvuLliyEcOIrHX4/QPedGMnum714IWjiTi8yQRC1BpJUKkZKj0pHwQph8G8e/FDXnX+Eft2gX+/Tr4eC9YZrz/8vR9dM//kefed71mKE39P9bLl+fUy/ng7c+xsCGN0I51/E4L5/OiKAx/n/A+x/a0ZPLvpP03qne6x9+kZY/qL/K+x/fvB/B29GuuUAJ4SAGzwQCVEQDwmQBGmQDr3oqZP6hGYKHZe7Zymg9oBh0PXXGGyZwEy2waidoM+8dggYQvsxOF/MM+rjzbo1WcEGdqoFPq4gRqFBe9oB+8Ct29PDRYuLZsGGosULFsKGWYuL5sGG4qKFFXSdO3sxXZctvh82zJldRvdzFs++DzbMLVpIY+bOnkU99xUtLIIN9xeVecWV7PLRBUVL5sKGhfeJnrI5RQtgw+KlC2nkktKFc+g6V8D/H2xX59/98+YUXWe/MgTsl8FAvVaINjdEEzfSyD8gcdhLT0QdCznEEy8MoGsseQ+J+hidUUGaP4GvoACGWcDwnzr7VtHyhcT8s4GlxX3Po6vN5gbvp/wL4rcR3Af++TiYERgr6u77HjCu65/Ro7342r1E/vTqerof1j01jSQeqnu8SMgjqiVdJwT1XbqGkCHGGZfStR+NTzM+Qc/70h0Y664bl9U9rn/3uMzucYSKbGMbSBqgZCvbqBkbqPH3UCq5CDuziqjJkiSvgOv1tqB0ZAn4wOtNUd3czbYbFrDPoecYDJZoEFIDOEMtprdlEPGrF8lYpicJkExW1Y8wHUAaMBhuBRFlx8M0mA/3Qzk8CMu8Ed4UXc+8NDaNxmb2GDtGH1tEYxfC4q6x/s/9n/o/9p/z/9H/kf8P/t/7T/lP+I/73/Mf9f/Kf9D/iv+A/+VPXJ+oN9ji/37osa17glvHBwj/AJ3JwaLqEgSiSdiykE+gCE+QSSUryI8BwWKlMjBYhJ4MDhZh87cGi4NKfrA4qYyhMpYKSYioDxSBz7RgEbFF+FLaO5AmAfEmUAS+C6mUU4mnIvTwQSpJVJYFSy8aFhEo0JvqlECBPiyH5HaCzjegEZ5hz1OrlPoXUU+9dADWwFLqeYudYLVSX+p7Hi7BaRpZAyewUQY2lmz5BI3/SJHgMpsMBwlGLnOzXINK7mGCfFCeKDfLF+RTMEiukE/JhXIFy8bnlKnK81Ry8W3Sx+MQB83sY6iAw/g1ZmOrPEK2wcd4ChvhC1pF6P0JWA8NUEm4uFkZVEmV0kTqeVc5BdvpLKPnp9gOdpqwO8xWw1l4CmVpNOxgZ4muE/ADrMbJUhWJNJt2GqfhXYJ1iuZvhwqZfAwzAZd6Ux9hT2vN0q8x2Fc5q5+XoIpWngwNarPqNiTSKoJjz7O3WLu6CerhNN6Ni/AcWyMnyrvl0bA+wAEshPUEe7uYo5ayZUS7OCsFdOlBuZA1wtdyoWEWwX5bUERrHpQmEkWl0ErlQdVBNA1la7CWMBVPY+CUYaycQfMJgmE5UQ1QhjmkD2X0fD8cgL5YB+sJkk6vOkj5gWY+I39KNK9nT0g/wCkcQRpcKl8kXgvVIm9xyKAqMkoM+ngdTVLymJIm3x13et+bHt+3zw1Nr8PgbYKCJusyb7PfX3CnHKVMb1KimzBZa5KTEz/9nx5+2rfPuII7vU2dI0cEoY4sHEF9k+6kW9GibuofOUJ/JhZtUpLpN6awyVs81/u44/HEIY87Zg/pK+xTErtF8hoIEmkuNSlCIFlhjs9ugNXySkkzKAxJY0yOjnFN5sl3tlAIe33w9GHtWbm5/SGjreNkJnsVzF5zgRlnJmd7Ep3ZTkxElnPixAn3cx7OlbOdi/jTbLYA/jY2Sl/rKxggz2dX2KMyrNRkRQVFkjWxgmvyuKaQyXfRMpJYpuN8e39wXKbVMpnPnmn0GQuMhcZyY73x10bDTOZMdMZTYWWNUlIjLXRW6i0KeepF/s8N6aRpZtr9JkIqWVWtb2CSPTklOcWempSaB09aYp/s90T4k0nqk5YnUlzr0pI2DkiNj0o2otVjM1rt8dbetiirvb95gEDLOFkQf9e4Jou4uMXFJi5WHdX+AtVj7ZfbHRd/uOjMdbqIM462rMvD2vQex8VMEo7BoXwryvQElqB63KHZWQNzBqSkZoV63GpiQkrOgIE9+5OzWWJIj2fKh9OKi6dNKS6esuPwkWfrDx/p2Dq1eNa0acUl2L++Y0Z93I7WIzt3thyWNm5+dNWWLatWb6k6f+TIuXNHWs9JRVtWPbp586Mr66p++qtqPXfktT+eaz18XjjpJv69VKm6yLsO8tnVp2CrzWoAdKkQYrI5zgtJEHUmQd24Jrt+D77B09uy2p1C+I6O9kymSh63KywxRcoZ4BokVVavWr2mvm7L5q2q60t+84ULfOgX37B3PvmYHWun9RpovTJ9vTjSL7GegYHZJYdoQOsNu3wNbkh2qMvjlgyJA105A6QGArmlrn7N6tWqq50P+/gTPuSbL9jbFy6wNwnqOSJlP3DSKuchWC2xCAiXCQZpZ3vmINLIc6dPc07aV+P/XF7fpRO+ELXeBfWWja514cZoeyxGe6LCadZloXBtQpSZLEFyOlzZWS6nQ0rNAqcDEhPEVVr7zLPP0u/ZZ68yI//x6lX+IzMqBfwUP0nlFMumcwDLrucVvJrX8Ar2BFvGHmZPCI5/StuOGWQBxEifJw/rZaleWWmAeqMWp0YjxDGz40yQ10zwuv1YgJCsy+1n2oUakfoctKNdlmYOincqOcnZznhPPGdj+TY2+302tqOhUa4Y3Tz6ytlGAkD+SB5LFEfDDl9qRGQUhkc7FRmciiLnOX7h3Gytd2+UKS6BwyQxU3SYA9UYoe4eUvfQoKYTJqjr95n211/XVVvH5nJ7t1Ir37KmaIczLJdw82VNkacqUw0Pyw8rD0TVRBgoakXIkeS+opfAA+rSyIqoJdGroDpiVeSqqFXRu2F3lHMmzEwmInIGwqCbGel+YoJqyLmZZWfJpP4G2n3USm90jCc2Zhfd9kL1vacfevjMnV8x98i7IvjlxsbGB9nGIQu2jnmwLu/Wk/2zvnrz7l3lMfwbkncJUd9AvJZI31b6YpgVrYBozQM0G+rJq600MosJolVNtujabp7cZdAWQfCZYcfas5y6kztDDsiVGyBYPk5EHxek9jLTNmA0TId5tD14HAyhrDeksN44kE1gt1tut05lpWwpexjXMCuRaGTxmO0UDpKcVg6qXGI8h589e7zzHiW543M81ZG9m9ezwrcI3/WkqXW6pibBnb6kEBWs1RaoD1Xro0N3OeottQkbo9clWxKM0RGxIdEYHxeVTKpLMmnTlbeto81xUZwkrkyfm3Yj7JR0Ck/JJ5QTqqEh9kCsNJPN7OlumKcfS0yQUHAdPG5I9ApFj88KlRoe27nzMSrMOP7p8e+dtg89cN+nTOGXPuOd/CIrYFHjn8ahh5/7xZEjv3jusLSsOSmFf8+/mzaTf/fNl/zPuurPYrtixe5vLVG1VqcqkXbmt/qSw4mmVLU+tm+9a2PsutRfZoZbknpFe5Ki7UayRjJJe3xUpqODPOuxdp0c3Tu069bpuJibm3mdx+xHupOUrftMQyzzuOXEhCTynCFdA0itpLUbdu3asOH5XXzXqo3g/9PHfOPKJ3/Jf/zxR/5jw+iNq1dt2rRq9Ubp7e01Ndufrq7ZPtV7YMUrH3zwyooD3oR31n/01VcfrX+HFS1ZtWoJFbKvCv/nSipRFAEDfZHWX9j2m7Y42S9gv7wlbKNzXaQhwgqZbkckySYr6FYCMSLzoD0qLkoiGQi+B3k9cJDH1t0IVVJLL6zyA7/EHAxWXSid/+2j/EX+MKtmk6q/VWadvfce/i7/A/+Iv3vPvadHj2Y72Rw2l+0cRVgdJ71PJL03QrrPpW6RpS2wUn5RU5gBKa0wCWzOHBNmrHu5A3YzYzNDyACFJ3EmHscZneVSQWfT+8rZRj66sXMQBGCyTXrk7ncIXpIYAZKFp3DoOwLdU+n+lvmsmYpPKVAKlfXKTkWdGZJNIN8nWFdom8z8dbxUh2Om+B9ippxti9IEKy2KpuYKmJYbYLbpEm/rEIFBh231WQushdb11p1WHbZD7c1yhHyPv3/q89uGVy+khTbx7y831r0Z5MR8Wk2BZJ9FZwOLxiEgq8JahGG3Z/qMmYYCwwpcIctMACQeHH9f+n3HvQRI96EjaG/4PMGwsErfGCVSVYwmoxxpMmKkyWySIplkNptUp4H2SU5Z0TSD5ETJQqMpA7HkmRQJVYSXzJrFbDJqga2h2QBWx5mTYcK1kGPJCsu9tj/QgkW5VgdupyccABsJyve9KqmKZKIIYnKZ0pQkk9d0s3SzMsCUaRov3abkmXym6dJ86T5ljqnQVClVSY8oVcoKU520RYkxgFHSkOhXxGsUZpA10AxGMMomkwVskeiRPVqExWHzyvGKV/UavFqiMcmUbPbavLZh0hDMkbOVTG2gMdc83JJpy4d8NlbyySNJ4HlqniFP82k+4wjTbRafzWe7U5qqTbcU2EqlOVgkz1IK1UJDoVZiLDGVmB+EB1il9BA+KC9RlqnLDA9q5dpDlipLla1aqsHH5FpljfFx83rbVnmn7SXbXeQ9Q7KNTPxYopEljjjJhrDcz8XlFK/l/G3+Jm0zr7rki6Jc6a04rlwiL1pJ/qYv5QcmyjhbKf7FmcOMNtgTprbYnN7quMPRLYnNznVhFgjDcKtRM8eh5h6ZQqpx8gxJRXc2GcfaLneQmbyjex9nrvCnCzNjMmMz4zK9mfGZCcNTfTG+WF+cz+uL9yUUxBTEFsQVeAviCxIKUstT18TUxNbE1Xhr4tckbEitT72UGts1tWtS14TC2MK4Qm9hfHlseVy5tzx+ReyKuBXeFfHhPT31TWyQMzGHvIS+J8yO7xksQ6WjH+9bWbatpbl5eOtj+050XmXSC1sLX508++iMv16SsksrZ1V8dDB9fOfKxtKiN5577XVX1dp+/RpTUzvEzvww8apBdZNdRsNgXwS2WOzGlnDPOntz1NYIcLlGhVtULTI/RjiPrMu6rbcJr/zOxcxXC2NXxNbHIuGpoxNAlemhhHZOhGuqyArwixeefPIFUTp/PuTlypPg95+sfHlIS4uUceLChRNUpIklRbyV/53O1qKS3YQNE/t4vEAyjIDhviioZo/JtmrrY6YWp9wS1ixcrMsKo90jycW2dblYB7980fG3i5k+sz3KEbUiakNUfZQSdLVd2OmuNiHoavHChGcLXnnnnVcKnp1w266Znfx3rC9Tpzwn5+zr3fvzU6c+7927MSmJCLIxFxuSSNwirOQZhJ8jwK3IFrC5WxRtna2ZbcUwGTRplNNlHhmj7yezsrq5dew6blGmFBAmbTOBBMh6RCp8rrl5yMuPnPCD/8QjL3e+S3zbvZt4h69K9/zUvrukiI1gGp0jirgnyL4gXlXELTdEQbkvCTzMWK09pnj2MKXFwo6Et7iaLeuiozyS5tFgnOSyj4zWUTwWTFloA6FvHC4Hdg7pw2PKY+pjPoi5FKMMh+FsuDTcMzxK6WPI0DKMfUxlUMbKpDJPWZRx5iLB4PhYlt0zjJECGHSmG+SqjgOWU4fmvzur+IP7+GX+Lkvv+IwZmqVdj21vsUn3zDj67oAB+3v1YYOZiYWwW/l/Htt6cP8OEXkySD3/TrwOgem+aMXBLNoeldVQ6qC2mqQQAxiMima1m8e7RegwidBhFqFDJGjiXk9Wj3UMO3bMFUhYszoo6tFujuLJqz5Pgafeg4Q6IRnD9PhHMskWxiX9van4NpbBP2xpatr/mureVjC3eH1HBn64fsKRvYLXfKo8g3hthjQY60uMsMQYXdUhoS12bElJbE5tNbbYX4uMSYkAzTJKdbm8I9P1nUxAHY61BRSCnxWcziWt6LWiV32vG2wozCHFk/Xk6Gy9iQVVhTKS0LCcbHxu15bNu3Zt3rKrmfMrRfvuuGPHxF8dzD3wyG86On7zyIHcZumm986ff+/d8+e/4Z/xr2NiX+nT67Vf31U8i9wmMpkNmVXcKPh7mGJkic7fAWT3RkAbU2tszmbLVhOTNJggPGO+W9i9bvbDxGbM6aKYdaDQw5iegwdQdorUX/dEcknzI49s2dfSkvfK0jfekRo675Z27NxxtKGzRnV37phd8p3wOG/Q4stoXfEuoLfPph6VX4ZWSWGaDPmao4MCo/56gSK02dGd9yt6jE4U2+g3mumQC6/Wq+6vCZ7/HJ+qwzODHUb4os2SAWxHLYYa5TVotbzs0ByKeruVaRbId+jQ23Jd13aTuhBoIafPWeAsdJY7Awu5u3YXgQV/+av8/vPG66uu+93rzxRtU9O+Jg5eoyThEGyVmAb53fmnz+oIbofKlUuKGkCfUFfdP7V3c5+0KIR2kNdxv/W/535bF/cPFnp+65Fu5L/nn/BfLtwv2B/wFEt17Q0j7Q1RW1zQYmkWubDLfge6PCNvyIV9icMjKqFSrTJUaVXGKlOVudJSZa2yVdmrHFXOSld9xKUI5/U5xXUpc8XmfXu3bNq3b9Ml5uIXL/2Ff8ec+PGF48cvfPXeu18/w9/j7fxbcgu5ZP1uNlhEJLKvBsJQ+NibfVFdPrbZto69hq0x5F9H6Z62R0xytLV1uVmfMeBnP4mlbV1yN2uCAem6QFXR0nItHkmDu6LU7s79qqmxR0Ri33Q52oDccCxh5wRKsFTa0TnNWGNrNrYaTCptqvJdwth1DSbveuakcKcHC0J2hgiJBeLQNXGF4di4MX2eeYHwOLwmpF80HnQ5TxztPEDCKi1WFFqtjKLgu7RaKlzwDbNaJJt5UlysZpQMpklxcbF5JnNsnOyh6Fgru6s9teEiOiZTdEyLNZnjogwwMUqzGTR3wsg0gdWZ9jah7rm5XeHybyJcurp2obZvKZM36Fex50wVe84F0aZoc7SlHzn9PuY+lqHGoaah5qEWsxe8LElKM6WZe4VkuDM8vULTYtPi0r3p8Ump1aZqc7Wl2irevjNJUk2qGS1oRRva0YERGIlRGC3HGFMz0oen35telb4ifUN6ffql9HDa7i26Fq3jmEjmxGuwoCMk1cpgItkbSLzDtRN2z6itnbV5+LFdP/5hxlv3l75TtGrd7L2+vU998pvSg/Lw/Wlpkyf7xsTbem2rfebVxMSjOTnT7xhXkGxP2rJqx75YIUtKcKTvlR1kgxTLbYpmxz3gZK1ajclMPCYdc7hswgb1MBJ47dl+OfBygLzgSwEvKGKHO3SoiCQpOSKGONmDrJKvGVfx2mtnn6upUXbwN9d31tdO2L7zP6TC9exm4QP3kxXeqVu/G4b6oq/Z/zoTa3U3W8j63eYJ5AfyPcIccwMa1ZbV7QTKPK8LJxBCnjdgdt17ihS2XziBF5ubb3156Rvvsd+yw9LznUU7dx5tkCqv1u8rLb6EuwX1N5EHqpILQYWrvlSkHEaWnExSRIWSSlkCJTJqnoTwa0VVKHtRZDCId1Sm7ixNvCYS74zu6no5fCws8Jbov81rxJuTn4+mDEXkJtXSCmmj1CBpYiEjGkmLPSwSI+UUSGHpmC57tRzIYUNwiJypiXxjDI6R85XRqk+bClPZdJwuF2ilUMrm4Tx5jjJXLdSWwhJWiZXyUuVhdQ2sYbVYS/lEtVoHdWyrtB2fkp9Stqq7lRfUJu117WPNr93clV+wxJveYvewe97id1+RCzsm476r9cShqcSAHOKQhX1DGeCUQAY4hTLAKSIDnPIvZYC//m8yQMHDcU1O8Z7NJS4h4mIOsFHwlYnXzforKZETd3P3304cmc+vSKFSqJJgyjGNkcYo+ZQj3iXdpUwxFZgWSguVUtMyksUyyhNrpG3SU8pmU6vUqvxGehd/q8QokhFV2ayYNLORKotHisBQOVKJ0qKMbrPHkgzJLFFKxXg5WUlQEwzJWirljPHmREsuDpQHarkiU5RGY77sk/MUn+oz+LQRlCWOMIssUUhxqlQg36FMVCcaCrRJxsmmKeZiKGGzpfk4m7L3+ep8w0JjkXmOpcy2FJayZdJyfEheTtKtUh82VBke0pYZq4yVpgfMyy010mMKZY2wlW2WNuEz8tPKU+pThm2aL6POstP2PDzPGqQG3CvvVfaoewx7tQbLS7ZfSS/ja/IRpdn4a9sx6S08Kb+vLNM1IoqJH0s0s8SpzV9+8dGXXzTzcx/95fuPSDfqcL4oV+uxrmO+sCKKVsKK7Gyt71aDJhmdYDc5zSYAu81pB7vVabGCqGxWUhqLk1Qmz2o2OsCs1OBrNnOrw2a1mIykKZpdtpsdXdqh6XI3d5lZ14722DFnWPBPhvZgXvo/aYLybViW0IFLKiiaakRrqCnM6rAmWnOsY0y3myZYZxhnmOabaqwrrJusLhMQEiRls81sD2MeySE7lDCT2+y2RNoi7amQRP7eK3uVdC3NmGxKMidZUq29bL3sXucgstMcKVPOVAabBpoHWgZbc2259kznLeBjPsmHPtkXlH6ecaRplHWMbYzd55wMd7A7pClYIBcoU9UphqnaNOM00oAplum26fYCZykrleaa5tnm2QudldpDtofstfC4cY15jaXWWmurtW8zbjFvsWy3bbc3mBsse2177U3O3zo/dvqds0mGio0FNnDDmf72QNo0YfMjm+4fPzk7ng8NmPrc9x7ePrp6sjyhYzPeL/zhAeTs9/o7o3CfGXfAShVlFgGq43z7mZPiz6j2TJbtwcQQJp9uWDlROfsUf4MNF+83aUNncOv/QxjEO06fk7ayq02qJEcoEMGMEZrFcabjDMVfYb7CfjPF/2VdBevOnJH2nD175cWzZ7l+6P/QdHZggeFJggcsMccR4grJBiQHXfFW/eHn63nHjKrOjj/jVvaZlMmw86+8uvNix7eBeazCUCv+cxc7zqa33jLU/lAR+M/HTRBrdYgeATBRIoA5rKJq04n6DzbUG2r/3LGLT+NuPpuNZBelFMz6842Y5DjIiWW7QlxOj/RE1QzeUf/84XqBSIjkIgO18c7OD3hsx7w/U+R/Qp9XK762EHvVTCzQUdG/seiJS7aHgGKiACo98Vb9hg/qT2yqEsic6jzHXfwIe5q1sz14VzAnVBppx50Eo30hKXoKaIkPt8ZqTku8wz0+WcTqLBGtHcNE5nesP/icRqtzj0uKrIHwrWqcq9Vszxj2ZVYWH3Yxi9LBrMzrUsBraaAe0A3igdhvKI1dOSE36Gnh/qbi1BT203X5YVeOuC0tbW5xIFfMEP/DqOKbpAiY6HNG5UOYFmp3y5qGoSZ1fOQ1fPkw0g6fS6MtiKPGFn409GXbViO0Kkxge5Hrb/WzKFPxR9VHbYhaEeXQX238I8qEMXE0UR4bwPTFV1oE5j+1tIiMqQvHQy8JpNmBr4M8DeI4yGcPy6c8zGTRNIfsso0PFfgF0BPYUba9xyhTCu40tlolgRjXsWI6625MpKWxErTwQ9dyaZH6KJU3ZNO0utpBq6dDX5jhC8/ID+ut9XJEebTIXkaIU7WkWGNCyvh+1xh1LEtcO3R2hUXFJe5JcrIa6Hu018sO2BpqSGqNiIknK2vLyhI+0tGe5WgPSjkozUEDB3WzqkvmPdJ/hbgnXgEI8U6LTp2wmkR9m3RAsDMoeyRuktgDUp4SHZIqmNnF3C7SpC7e6tSlwARfaFq+VXOEhrs1h1H8+RQfZYxLHJ/agzKdMF0NwqO9e+KdUo0lZavHEN9qj4wNkCT+EL+RnoHZN7D/hncZAT3tKYsgHd007O0pl27Z6F/wTd9WfDF53b32YX+DOE3/sOfDn9nauuoff9cx3jbdKL4d1Lq/A6J5hgU8BsDGf/zdlTts0//hK6JB8in92xiQGqmsJRdKEQT3w9uG7bAI46CJSgP1n1PCoIbKp1TqqJRQWU9lrVIHFfJ2OC7dLYq/Tl4Kxw1fwwj5AlQqbjgsl8IiqhfJ7bBI+hAyxL3igsNSLrwh5frPiVq05S/0cYdxLN33hjJMhEHUv19uhZvUtTBV1ITHASKundxY0/VFekLUymqCbwoUQytJPOM6WvsTpT+x/uxuVs4OsK+ludLbmInbsE22yVnyPHmtMlfh6lh1i/qpYZxhq+GqNki7V3tGO6z9YBxlrDOeMj1iajFdNa+z7LCuttZb37d+af2rzW7rbcu1TbG9aPPbb7a/6LAEP70ajJOhN8wFC1mWA7YJqcgeKZRqWf/eaob4ykw20uBM/bsscc8glFqBewk0lh+8xx79co97io9sQvBeBTcrhVuhDMphGSyGeTCHVl8CXkiDYrJpL2RBJp3ZdDeLRnghj8YsgQoqi2E2FMEC6EO9Y2Ahje9Hd7fA/XR6YWI3rAq9NZvq2TTnAbqW0EjTv7DqwO5VJ9NKD9Ba4gufhTRa4FFEc/69FUfQ3XyaNxWW0ohiGlukQ5utzyjSKfISlIV0LacxswjuPBrnpflltHqR/uxGOJN0KBWEURmd91GvWLWCxpbpkLJo7WzIuW5W1xwpoGT+n+nfS/7jMVjXi8B3uAppgfhmtuu73TCKSVHkmdJJAn1Ja4fQCiMhH0bBaJLHWBgH4+E2uB3uIF5Mgim07jS4E6bTSnfDIdjLJP1tosJUZmAavA5vMCMzGZYunJeVnZsXrG8N1iOC9chgnR+ob8kUdX5eZledHawHADRLK3z+qxyvuPGnZPx7Fv5Yhz/Y8G8cL3P8azJ+b8O/1OGlZPzu8VuU7zherMNv67D9Cn5zBf/M8esh+FUeXuD4ZRZ+0TZJ+aIO22hg2yT8/LMM5fMr+FkGfsrxE44fZ+Gf3PifdXie4zkX/nE5fnQE/8DxdzT8d8vx7JlRytnleGYUnv6PKOU0x/+Iwg85fsDxtxx/w/FUHZ48Eauc5HgiFt/PwuMc31njVN6JxrdD8RjHtzi+yfENjq9z/DXHoxxf49jK8QjHw05sqU5WWjg2H6I8hOOhV2cqh47goRXyq79KVl6d6fPjqz75V8l4kOMrdXiA48scmzi+xHF/Cb5ow317k5V9Jbi30aXsTcZGF+4hpPdcwd0cX+D4PMddLmzg+MvnbMovs/A5G/6iBOtpSH0d7uS441mLsoPjsxZ85ukI5ZkSfHq7Q3k6Arc7cJsJn+K4tc6qbOVYZ8UtNGlLHW7eZFM2p+EmGz55BTduOKJs5Lhh/UxlwxHcsEJe//NkZf1MXO+Tf56MT3Bct7afso7j2n74OJH5+C1Y+5hZqXXjY2asoY6aEqwmTlUn4xonPspx9SqnsprjKieu5LiCYxVHn/9ny5crP+O4fDk+UoKVkz1KZTI+zHEZx4ds+KAFHzDhUo5LrmDFFVx8BRddwXKOZRwXcrw/Hu/jON+Zp8yfhPM4zl2Oc6hRynE2xxKOxRxncSwagoVX8B4LzuR4F8cZHKffaVKmX8E7TTgtNEKZloVTOU6hlafk4WQPTmIOZVI4TnTjHWNDlDs4Fpjxdo4TbnMoEzje5sDxHMfRk3Ecx45xKGNDcEyMVRnjwNFWHMUxvw5H1uEIjrdKfZVbr2DeEbxlHPo4Dud4800u5WY33jTMrtzkwmFDrcown9+OQ604hGMux8GD3MrgKzhooEMZ5MaBOWZloANzzDggFrOtmNXfrGRx7G/GzAyzkmnFDDP262tU+jmwrxH7ZGHvXslK7xLsle5SeiVjugvTUpOVtFswNRlTks1Kih2TzZjEMZFjgh3jic54F3pLMO4KxhIJsSUYY8Vo4mA0x6grGJmHEdSI4BhegmHEqTCOoTQpNAI9HN0cQzi6aICLo5NodeahYznaS9DG0WoJVawcLTTaEopmjiYHGjlqNEzjaHCjWkK5l0ORSQM8SL3IKe9wKFJfZA4EjqyZlax5gvX+/+GA/9cI/K9HzH8B3j158QplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9MZW5ndGggMTQzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nO3PyU5CQRAF0JugqKA4MQiiAjIo8P/fR+UtXD0Xho2Lc5Jb1d3V6aSTM3Vykctm1W3qVa5/ZjeVXvq5rX5XGVTuW195aOpjU5/yXHXYem/UcjbOJC+ZZpbXzPOW93zU6aKyzCqf1dfZZJtdvvJdu/3fPggAAAAAAAAAAAAAAAAAAAD/0uHXyfEEtCYD3AplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9MZW5ndGggNDE0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nF2Ty66bMBCG9zyFl6eLI8Ah40aKkKrTTRa9qGlX6CyIGSKkxiCHLPL2tf2PaVUkGP3fXDw24/Lt9PnkplWV3/1sz7yqcXKD5/v88JbVha+TK2qthsmuotLX3vqlKEPy+Xlf+XZy41wcj6r8EZz31T/Vy6dhvvCHQilVfvMD+8ld1cuvtzPQ+bEsv/nGblVV0bZq4DGU+9IvX/sbqzIlv56G4J/W52tI+xvx87mw0knXaMnOA9+X3rLv3ZWLYxWeVh3H8LQFu+E/v66Qdhm3eB3jg9Gt6v6RdabvKWgPuZegveCPwAfBm4SXIVm8DLzDCjtZMMs6UwQ1kI0ENYIJmASTYCwcTQebcLNLOJoOFhhFGinSSJHmANwL3iS8FtKK1wLvTcLRdLAJE3ZDUitLDYO2CFskHC2hKUI9wobokMugKnog6YGkB8I5m0owJI2ZpiCDtY00miVaMWjFoBVDORSZaMLITzbp7+qqjhimg01YU8KagHU83DCHeeDiSMb7s827fXgfRj1dsjTjcbonx9s9XOYlZsX3D+Hm8CoKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvQ0lERm9udFR5cGUyIC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMKL0NJRFN5c3RlbUluZm8gPDwgL1JlZ2lzdHJ5IChBZG9iZSkgL09yZGVyaW5nIChJZGVudGl0eSkgL1N1cHBsZW1lbnQgMCA+PgovRm9udERlc2NyaXB0b3IgMTMgMCBSIC9XIDE4IDAgUiAvQ0lEVG9HSURNYXAgMTYgMCBSID4+CmVuZG9iagoxNSAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvVHlwZTAgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucwovRW5jb2RpbmcgL0lkZW50aXR5LUggL0Rlc2NlbmRhbnRGb250cyBbIDE0IDAgUiBdIC9Ub1VuaWNvZGUgMTkgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9Gb250RmlsZTIgMTcgMCBSIC9NYXhXaWR0aCAxMDAwID4+CmVuZG9iagoxOCAwIG9iagpbIDMyIFsgMzE4IDQwMSA0NjAgXSAzNyBbIDk1MCBdIDQwIFsgMzkwIDM5MCBdIDQ2IFsgMzE4IF0gNDgKWyA2MzYgNjM2IDYzNiBdIDUyIFsgNjM2IF0gNTQgWyA2MzYgXSA1NiBbIDYzNiBdIDY3IFsgNjk4IF0gNzAgWyA1NzUgXSA3MwpbIDI5NSAyOTUgXSA3NiBbIDU1NyBdIDg3IFsgOTg5IF0gOTcKWyA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggXSAxMDggWyAyNzggXSAxMTAgWyA2MzQgNjEyIDYzNSBdCjExNCBbIDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IF0gMTIxIFsgNTkyIF0gODIxNyBbIDMxOCBdIDgyMzAgWyAxMDAwIF0gXQplbmRvYmoKMyAwIG9iago8PCAvRjEgMTUgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAwIC9jYSAxID4+Ci9BMiA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8ID4+CmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagoyMCAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My45LjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My45LjApCi9DcmVhdGlvbkRhdGUgKEQ6MjAyNTA1MzAxNzA2MjMrMDgnMDAnKSA+PgplbmRvYmoKeHJlZgowIDIxCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDEyNjc0IDAwMDAwIG4gCjAwMDAwMTI0ODAgMDAwMDAgbiAKMDAwMDAxMjUxMiAwMDAwMCBuIAowMDAwMDEyNjExIDAwMDAwIG4gCjAwMDAwMTI2MzIgMDAwMDAgbiAKMDAwMDAxMjY1MyAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDIgMDAwMDAgbiAKMDAwMDAwMTQ2OSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDE0NDggMDAwMDAgbiAKMDAwMDAxMTkxOCAwMDAwMCBuIAowMDAwMDExNTU4IDAwMDAwIG4gCjAwMDAwMTE3NzEgMDAwMDAgbiAKMDAwMDAxMDg1NSAwMDAwMCBuIAowMDAwMDAxNDg5IDAwMDAwIG4gCjAwMDAwMTIxNDMgMDAwMDAgbiAKMDAwMDAxMTA3MSAwMDAwMCBuIAowMDAwMDEyNzM0IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgMjEgL1Jvb3QgMSAwIFIgL0luZm8gMjAgMCBSID4+CnN0YXJ0eHJlZgoxMjg5MQolJUVPRgo=", "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2025-05-30T17:06:23.362788\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.9.0, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "preds_df = pd.DataFrame(preds[0])\n", "plt.bar(labels, 100 * preds_df[\"score\"], color='C0')\n", "plt.title(f'\"{custom_tweet}\"')\n", "plt.ylabel(\"Class probability (%)\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 4 }