Upload 99 files

.gitattributes

@@ -109,3 +109,5 @@ Transformer[[:space:]]Mechanism/Transformer_Implementation/home/jovyan/work/W4A1
 Transformer[[:space:]]Mechanism/Transformer_Implementation/home/jovyan/work/W4A1/encoder.png filter=lfs diff=lfs merge=lfs -text
 Transformer[[:space:]]Mechanism/Transformer_Implementation/home/jovyan/work/W4A1/self-attention.png filter=lfs diff=lfs merge=lfs -text
 Transformer[[:space:]]Mechanism/Transformer_Implementation/home/jovyan/work/W4A1/transformer.png filter=lfs diff=lfs merge=lfs -text
+Transformer[[:space:]]Mechanism/Transformer[[:space:]]Pre-Processing/home/jovyan/work/W4A4_UGL_POS/glove/glove.6B.100d.txt filter=lfs diff=lfs merge=lfs -text
+Transformer[[:space:]]Mechanism/Transformer[[:space:]]Pre-Processing/home/jovyan/work/W4A4_UGL_POS/preprocessing.png filter=lfs diff=lfs merge=lfs -text

Transformer Mechanism/Named Entity Recognition/tf/.Trash-0/files/W4A2-UGL-NER.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c76298397280a20118c061ffdb3d9abfa2e52e6b833fcdfbc0dab89837635fd
+size 245286524

Transformer Mechanism/Named Entity Recognition/tf/.Trash-0/info/W4A2-UGL-NER.tar.gz.trashinfo

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:779e266c8830e8c475b5b4d1eddf4b3cd85148af4686f9eef77319d458a6c9a8
+size 71

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/.ipynb_checkpoints/Transformer_application_Named_Entity_Recognition-checkpoint.ipynb

@@ -0,0 +1,715 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Transformer Network Application: Named-Entity Recognition\n",
+    "\n",
+    "Welcome to Week 4's second ungraded lab. In this notebook you'll explore one application of the transformer architecture that you built in the previous assignment.\n",
+    "\n",
+    "**After this assignment you'll be able to**:\n",
+    "\n",
+    "* Use tokenizers and pre-trained models from the HuggingFace Library.\n",
+    "* Fine-tune a pre-trained transformer model for Named-Entity Recognition"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Table of Contents\n",
+    "\n",
+    "- [Packages](#0)\n",
+    "- [1 - Named-Entity Recogniton to Process Resumes](#1)\n",
+    "    - [1.1 - Data Cleaning](#1-1)\n",
+    "    - [1.2 - Padding and Generating Tags](#1-2)\n",
+    "    - [1.3 - Tokenize and Align Labels with 🤗 Library](#1-3)\n",
+    "        - [Exercise 1 - tokenize_and_align_labels](#ex-1)\n",
+    "    - [1.4 - Optimization](#1-4)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name='0'></a>\n",
+    "## Packages\n",
+    "\n",
+    "Run the following cell to load the packages you'll need."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import tensorflow as tf\n",
+    "import json\n",
+    "import random\n",
+    "import logging\n",
+    "import re\n",
+    "\n",
+    "tf.get_logger().setLevel('ERROR')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name='1'></a>\n",
+    "## 1 - Named-Entity Recogniton to Process Resumes\n",
+    "\n",
+    "When faced with a large amount of unstructured text data, named-entity recognition (NER) can help you detect and classify important information in your dataset. For instance, in the running example \"Jane vists Africa in September\", NER would help you detect \"Jane\", \"Africa\", and \"September\" as named-entities and classify them as person, location, and time. \n",
+    "\n",
+    "* You will use a variation of the Transformer model you built in the last assignment to process a large dataset of resumes.\n",
+    "* You will find and classify relavent information such as the companies the applicant worked at, skills, type of degree, etc. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name='1-1'></a>\n",
+    "### 1.1 - Dataset Cleaning\n",
+    "\n",
+    "In this assignment you will optimize a Transformer model on a dataset of resumes. Take a look at how the data you will be working with are structured."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_data = pd.read_json(\"ner.json\", lines=True)\n",
+    "df_data = df_data.drop(['extras'], axis=1)\n",
+    "df_data['content'] = df_data['content'].str.replace(\"\\n\", \" \")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_data.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_data.iloc[0]['annotation']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def mergeIntervals(intervals):\n",
+    "    sorted_by_lower_bound = sorted(intervals, key=lambda tup: tup[0])\n",
+    "    merged = []\n",
+    "\n",
+    "    for higher in sorted_by_lower_bound:\n",
+    "        if not merged:\n",
+    "            merged.append(higher)\n",
+    "        else:\n",
+    "            lower = merged[-1]\n",
+    "            if higher[0] <= lower[1]:\n",
+    "                if lower[2] is higher[2]:\n",
+    "                    upper_bound = max(lower[1], higher[1])\n",
+    "                    merged[-1] = (lower[0], upper_bound, lower[2])\n",
+    "                else:\n",
+    "                    if lower[1] > higher[1]:\n",
+    "                        merged[-1] = lower\n",
+    "                    else:\n",
+    "                        merged[-1] = (lower[0], higher[1], higher[2])\n",
+    "            else:\n",
+    "                merged.append(higher)\n",
+    "    return merged"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_entities(df):\n",
+    "    \n",
+    "    entities = []\n",
+    "    \n",
+    "    for i in range(len(df)):\n",
+    "        entity = []\n",
+    "    \n",
+    "        for annot in df['annotation'][i]:\n",
+    "            try:\n",
+    "                ent = annot['label'][0]\n",
+    "                start = annot['points'][0]['start']\n",
+    "                end = annot['points'][0]['end'] + 1\n",
+    "                entity.append((start, end, ent))\n",
+    "            except:\n",
+    "                pass\n",
+    "    \n",
+    "        entity = mergeIntervals(entity)\n",
+    "        entities.append(entity)\n",
+    "    \n",
+    "    return entities"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_data['entities'] = get_entities(df_data)\n",
+    "df_data.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def convert_dataturks_to_spacy(dataturks_JSON_FilePath):\n",
+    "    try:\n",
+    "        training_data = []\n",
+    "        lines=[]\n",
+    "        with open(dataturks_JSON_FilePath, 'r') as f:\n",
+    "            lines = f.readlines()\n",
+    "\n",
+    "        for line in lines:\n",
+    "            data = json.loads(line)\n",
+    "            text = data['content'].replace(\"\\n\", \" \")\n",
+    "            entities = []\n",
+    "            data_annotations = data['annotation']\n",
+    "            if data_annotations is not None:\n",
+    "                for annotation in data_annotations:\n",
+    "                    #only a single point in text annotation.\n",
+    "                    point = annotation['points'][0]\n",
+    "                    labels = annotation['label']\n",
+    "                    # handle both list of labels or a single label.\n",
+    "                    if not isinstance(labels, list):\n",
+    "                        labels = [labels]\n",
+    "\n",
+    "                    for label in labels:\n",
+    "                        point_start = point['start']\n",
+    "                        point_end = point['end']\n",
+    "                        point_text = point['text']\n",
+    "                        \n",
+    "                        lstrip_diff = len(point_text) - len(point_text.lstrip())\n",
+    "                        rstrip_diff = len(point_text) - len(point_text.rstrip())\n",
+    "                        if lstrip_diff != 0:\n",
+    "                            point_start = point_start + lstrip_diff\n",
+    "                        if rstrip_diff != 0:\n",
+    "                            point_end = point_end - rstrip_diff\n",
+    "                        entities.append((point_start, point_end + 1 , label))\n",
+    "            training_data.append((text, {\"entities\" : entities}))\n",
+    "        return training_data\n",
+    "    except Exception as e:\n",
+    "        logging.exception(\"Unable to process \" + dataturks_JSON_FilePath + \"\\n\" + \"error = \" + str(e))\n",
+    "        return None\n",
+    "\n",
+    "def trim_entity_spans(data: list) -> list:\n",
+    "    \"\"\"Removes leading and trailing white spaces from entity spans.\n",
+    "\n",
+    "    Args:\n",
+    "        data (list): The data to be cleaned in spaCy JSON format.\n",
+    "\n",
+    "    Returns:\n",
+    "        list: The cleaned data.\n",
+    "    \"\"\"\n",
+    "    invalid_span_tokens = re.compile(r'\\s')\n",
+    "\n",
+    "    cleaned_data = []\n",
+    "    for text, annotations in data:\n",
+    "        entities = annotations['entities']\n",
+    "        valid_entities = []\n",
+    "        for start, end, label in entities:\n",
+    "            valid_start = start\n",
+    "            valid_end = end\n",
+    "            while valid_start < len(text) and invalid_span_tokens.match(\n",
+    "                    text[valid_start]):\n",
+    "                valid_start += 1\n",
+    "            while valid_end > 1 and invalid_span_tokens.match(\n",
+    "                    text[valid_end - 1]):\n",
+    "                valid_end -= 1\n",
+    "            valid_entities.append([valid_start, valid_end, label])\n",
+    "        cleaned_data.append([text, {'entities': valid_entities}])\n",
+    "    return cleaned_data  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = trim_entity_spans(convert_dataturks_to_spacy(\"ner.json\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tqdm.notebook import tqdm\n",
+    "def clean_dataset(data):\n",
+    "    cleanedDF = pd.DataFrame(columns=[\"setences_cleaned\"])\n",
+    "    sum1 = 0\n",
+    "    for i in tqdm(range(len(data))):\n",
+    "        start = 0\n",
+    "        emptyList = [\"Empty\"] * len(data[i][0].split())\n",
+    "        numberOfWords = 0\n",
+    "        lenOfString = len(data[i][0])\n",
+    "        strData = data[i][0]\n",
+    "        strDictData = data[i][1]\n",
+    "        lastIndexOfSpace = strData.rfind(' ')\n",
+    "        for i in range(lenOfString):\n",
+    "            if (strData[i]==\" \" and strData[i+1]!=\" \"):\n",
+    "                for k,v in strDictData.items():\n",
+    "                    for j in range(len(v)):\n",
+    "                        entList = v[len(v)-j-1]\n",
+    "                        if (start>=int(entList[0]) and i<=int(entList[1])):\n",
+    "                            emptyList[numberOfWords] = entList[2]\n",
+    "                            break\n",
+    "                        else:\n",
+    "                            continue\n",
+    "                start = i + 1  \n",
+    "                numberOfWords += 1\n",
+    "            if (i == lastIndexOfSpace):\n",
+    "                for j in range(len(v)):\n",
+    "                        entList = v[len(v)-j-1]\n",
+    "                        if (lastIndexOfSpace>=int(entList[0]) and lenOfString<=int(entList[1])):\n",
+    "                            emptyList[numberOfWords] = entList[2]\n",
+    "                            numberOfWords += 1\n",
+    "        cleanedDF = cleanedDF.append(pd.Series([emptyList],  index=cleanedDF.columns ), ignore_index=True )\n",
+    "        sum1 = sum1 + numberOfWords\n",
+    "    return cleanedDF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cleanedDF = clean_dataset(data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Take a look at your cleaned dataset and the categories the named-entities are matched to, or 'tags'."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cleanedDF.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name='1-2'></a>\n",
+    "### 1.2 - Padding and Generating Tags\n",
+    "\n",
+    "Now, it is time to generate a list of unique tags you will match the named-entities to."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "unique_tags = set(cleanedDF['setences_cleaned'].explode().unique())#pd.unique(cleanedDF['setences_cleaned'])#set(tag for doc in cleanedDF['setences_cleaned'].values.tolist() for tag in doc)\n",
+    "tag2id = {tag: id for id, tag in enumerate(unique_tags)}\n",
+    "id2tag = {id: tag for tag, id in tag2id.items()}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "unique_tags"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, you will create an array of tags from your cleaned dataset. Oftentimes, your input sequence can exceeds the maximum length of a sequence your network can process, so it needs to be cut off to that desired maximum length. And when the input sequence is shorter than the desired length, you need to append zeroes onto its end using this [Keras padding API](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tensorflow.keras.preprocessing.sequence import pad_sequences"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MAX_LEN = 512\n",
+    "labels = cleanedDF['setences_cleaned'].values.tolist()\n",
+    "\n",
+    "tags = pad_sequences([[tag2id.get(l) for l in lab] for lab in labels],\n",
+    "                     maxlen=MAX_LEN, value=tag2id[\"Empty\"], padding=\"post\",\n",
+    "                     dtype=\"long\", truncating=\"post\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tags"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name='1-3'></a>\n",
+    "### 1.3 - Tokenize and Align Labels with 🤗 Library\n",
+    "\n",
+    "Before feeding the texts to a Transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use must match the Transformer model type you are using! In this exercise, you will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. Notice this matches the maximum length you used when creating tags. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "deletable": false,
+    "edittable": false
+   },
+   "outputs": [],
+   "source": [
+    "gpus = tf.config.list_physical_devices('GPU')\n",
+    "if gpus:\n",
+    "    for gpu in gpus:\n",
+    "        tf.config.experimental.set_virtual_device_configuration(gpu,[tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import DistilBertTokenizerFast #, TFDistilBertModel\n",
+    "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the lists of tags and the list of labels generated by the selected tokenizer with a `tokenize_and_align_labels()` function.\n",
+    "\n",
+    "<a name='ex-1'></a>\n",
+    "### Exercise 1 - tokenize_and_align_labels\n",
+    "\n",
+    "Implement `tokenize_and_align_labels()`. The function should perform the following:\n",
+    "* The tokenizer cuts sequences that exceed the maximum size allowed by your model with the parameter `truncation=True`\n",
+    "* Aligns the list of tags and labels with the tokenizer `word_ids` method returns a list that maps the subtokens to the original word in the sentence and special tokens to `None`. \n",
+    "* Set the labels of all the special tokens (`None`) to -100 to prevent them from affecting the loss function. \n",
+    "* Label of the first subtoken of a word and set the label for the following subtokens to -100. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "label_all_tokens = True\n",
+    "def tokenize_and_align_labels(tokenizer, examples, tags):\n",
+    "    tokenized_inputs = tokenizer(examples, truncation=True, is_split_into_words=False, padding='max_length', max_length=512)\n",
+    "    labels = []\n",
+    "    for i, label in enumerate(tags):\n",
+    "        word_ids = tokenized_inputs.word_ids(batch_index=i)\n",
+    "        previous_word_idx = None\n",
+    "        label_ids = []\n",
+    "        for word_idx in word_ids:\n",
+    "            # Special tokens have a word id that is None. We set the label to -100 so they are automatically\n",
+    "            # ignored in the loss function.\n",
+    "            if word_idx is None:\n",
+    "                label_ids.append(-100)\n",
+    "            # We set the label for the first token of each word.\n",
+    "            elif word_idx != previous_word_idx:\n",
+    "                label_ids.append(label[word_idx])\n",
+    "            # For the other tokens in a word, we set the label to either the current label or -100, depending on\n",
+    "            # the label_all_tokens flag.\n",
+    "            else:\n",
+    "                label_ids.append(label[word_idx] if label_all_tokens else -100)\n",
+    "            previous_word_idx = word_idx\n",
+    "\n",
+    "        labels.append(label_ids)\n",
+    "\n",
+    "    tokenized_inputs[\"labels\"] = labels\n",
+    "    return tokenized_inputs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that you have tokenized inputs, you can create train and test datasets!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test = tokenize_and_align_labels(tokenizer, df_data['content'].values.tolist(), tags)\n",
+    "train_dataset = tf.data.Dataset.from_tensor_slices((\n",
+    "    test['input_ids'],\n",
+    "    test['labels']\n",
+    "))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a name='1-4'></a>\n",
+    "### 1.4 - Optimization\n",
+    "\n",
+    "Fantastic! Now you can finally feed your data into into a pretrained 🤗 model. You will optimize a DistilBERT model, which matches the tokenizer you used to preprocess your data. Try playing around with the different hyperparamters to improve your results!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import TFDistilBertForTokenClassification\n",
+    "\n",
+    "model = TFDistilBertForTokenClassification.from_pretrained('model/', num_labels=len(unique_tags))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)\n",
+    "model.compile(optimizer=optimizer, loss=model.hf_compute_loss, metrics=['accuracy']) # can also use any keras loss fn\n",
+    "model.fit(train_dataset.batch(4),\n",
+    "          epochs=10, \n",
+    "          batch_size=4)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = \"Manisha Bharti. 3.5 years of professional IT experience in Banking and Finance domain\"\n",
+    "inputs = tokenizer(text, return_tensors=\"tf\", truncation=True, is_split_into_words=False, padding=\"max_length\", max_length=512 )\n",
+    "input_ids = inputs[\"input_ids\"]\n",
+    "#inputs[\"labels\"] = tf.reshape(tf.constant([1] * tf.size(input_ids).numpy()), (-1, tf.size(input_ids)))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "output = model(inputs).logits\n",
+    "prediction = np.argmax(output, axis=2)\n",
+    "print( prediction)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model(inputs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pred_labels = []"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install seqeval"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "true_labels = [[id2tag.get(true_index, \"Empty\") for true_index in test['labels'][i]] for i in range(len(test['labels']))]\n",
+    "np.array(true_labels).shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "output = model.predict(train_dataset)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions = np.argmax(output['logits'].reshape(220, -1, 12), axis=-1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "predictions.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from matplotlib import pyplot as plt \n",
+    "\n",
+    "p = plt.hist(np.array(true_labels).flatten())\n",
+    "plt.xticks(rotation='vertical')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import Counter\n",
+    "Counter(np.array(true_labels).flatten())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pred_labels = [[id2tag.get(index, \"Empty\") for index in predictions[i]] for i in range(len(predictions))]\n",
+    "p = plt.hist(np.array(pred_labels).flatten())\n",
+    "plt.xticks(rotation='vertical')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from seqeval.metrics import classification_report\n",
+    "print(classification_report(true_labels, pred_labels))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Congratulations!\n",
+    "\n",
+    "#### Here's what you should remember\n",
+    "\n",
+    "- Named-entity recognition (NER) detects and classifies named-entities, and can help process resumes, customer reviews, browsing histories, etc. \n",
+    "- You must preprocess text data with the corresponding tokenizer to the pretrained model before feeding your input into your Transformer model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/model/._tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c609005a68991082f5f7a122c44818a33fd6be0205464bbfdd514dd50eb8295f
+size 212

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/model/config.json

@@ -0,0 +1,51 @@
+{
+  "_name_or_path": "distilbert-base-uncased",
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForMaskedLM"
+  ],
+  "attention_dropout": 0.1,
+  "dim": 768,
+  "dropout": 0.1,
+  "hidden_dim": 3072,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5",
+    "6": "LABEL_6",
+    "7": "LABEL_7",
+    "8": "LABEL_8",
+    "9": "LABEL_9",
+    "10": "LABEL_10",
+    "11": "LABEL_11"
+  },
+  "initializer_range": 0.02,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_10": 10,
+    "LABEL_11": 11,
+    "LABEL_2": 2,
+    "LABEL_3": 3,
+    "LABEL_4": 4,
+    "LABEL_5": 5,
+    "LABEL_6": 6,
+    "LABEL_7": 7,
+    "LABEL_8": 8,
+    "LABEL_9": 9
+  },
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "pad_token_id": 0,
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "transformers_version": "4.5.1",
+  "vocab_size": 30522
+}

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/model/tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e18c2bfd591b3cb73cf6619c3b59870b09291057287cc4f2fabf65e06232ced8
+size 265614944

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/tokenizer/special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/tokenizer/tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "distilbert-base-uncased"}

Transformer Mechanism/Named Entity Recognition/tf/W4A2_UGL/utils.py

@@ -0,0 +1,152 @@
+import pandas as pd
+
+def mergeIntervals(intervals):
+    sorted_by_lower_bound = sorted(intervals, key=lambda tup: tup[0])
+    merged = []
+
+    for higher in sorted_by_lower_bound:
+        if not merged:
+            merged.append(higher)
+        else:
+            lower = merged[-1]
+            if higher[0] <= lower[1]:
+                if lower[2] is higher[2]:
+                    upper_bound = max(lower[1], higher[1])
+                    merged[-1] = (lower[0], upper_bound, lower[2])
+                else:
+                    if lower[1] > higher[1]:
+                        merged[-1] = lower
+                    else:
+                        merged[-1] = (lower[0], higher[1], higher[2])
+            else:
+                merged.append(higher)
+    return merged
+
+def get_entities(df):
+    
+    entities = []
+    
+    for i in range(len(df)):
+        entity = []
+    
+        for annot in df['annotation'][i]:
+            try:
+                ent = annot['label'][0]
+                start = annot['points'][0]['start']
+                end = annot['points'][0]['end'] + 1
+                entity.append((start, end, ent))
+            except:
+                pass
+    
+        entity = mergeIntervals(entity)
+        entities.append(entity)
+    
+    return entities
+
+def read_dataset()
+    df_data = pd.read_json("ner.json", lines=True)
+    df_data = df_data.drop(['extras'], axis=1)
+    df_data['content'] = df_data['content'].str.replace("\n", " ")
+    df_data['entities'] = get_entities(df_data)
+    
+    return df_data
+
+def convert_dataturks_to_spacy(dataturks_JSON_FilePath):
+    try:
+        training_data = []
+        lines=[]
+        with open(dataturks_JSON_FilePath, 'r') as f:
+            lines = f.readlines()
+
+        for line in lines:
+            data = json.loads(line)
+            text = data['content'].replace("\n", " ")
+            entities = []
+            data_annotations = data['annotation']
+            if data_annotations is not None:
+                for annotation in data_annotations:
+                    #only a single point in text annotation.
+                    point = annotation['points'][0]
+                    labels = annotation['label']
+                    # handle both list of labels or a single label.
+                    if not isinstance(labels, list):
+                        labels = [labels]
+
+                    for label in labels:
+                        point_start = point['start']
+                        point_end = point['end']
+                        point_text = point['text']
+                        
+                        lstrip_diff = len(point_text) - len(point_text.lstrip())
+                        rstrip_diff = len(point_text) - len(point_text.rstrip())
+                        if lstrip_diff != 0:
+                            point_start = point_start + lstrip_diff
+                        if rstrip_diff != 0:
+                            point_end = point_end - rstrip_diff
+                        entities.append((point_start, point_end + 1 , label))
+            training_data.append((text, {"entities" : entities}))
+        return training_data
+    except Exception as e:
+        logging.exception("Unable to process " + dataturks_JSON_FilePath + "\n" + "error = " + str(e))
+        return None
+
+def trim_entity_spans(data: list) -> list:
+    """Removes leading and trailing white spaces from entity spans.
+
+    Args:
+        data (list): The data to be cleaned in spaCy JSON format.
+
+    Returns:
+        list: The cleaned data.
+    """
+    invalid_span_tokens = re.compile(r'\s')
+
+    cleaned_data = []
+    for text, annotations in data:
+        entities = annotations['entities']
+        valid_entities = []
+        for start, end, label in entities:
+            valid_start = start
+            valid_end = end
+            while valid_start < len(text) and invalid_span_tokens.match(
+                    text[valid_start]):
+                valid_start += 1
+            while valid_end > 1 and invalid_span_tokens.match(
+                    text[valid_end - 1]):
+                valid_end -= 1
+            valid_entities.append([valid_start, valid_end, label])
+        cleaned_data.append([text, {'entities': valid_entities}])
+    return cleaned_data  
+
+def clean_dataset(data):
+    cleanedDF = pd.DataFrame(columns=["setences_cleaned"])
+    sum1 = 0
+    for i in range(len(data)):
+        start = 0
+        emptyList = ["Empty"] * len(data[i][0].split())
+        numberOfWords = 0
+        lenOfString = len(data[i][0])
+        strData = data[i][0]
+        strDictData = data[i][1]
+        lastIndexOfSpace = strData.rfind(' ')
+        for i in range(lenOfString):
+            if (strData[i]==" " and strData[i+1]!=" "):
+                for k,v in strDictData.items():
+                    for j in range(len(v)):
+                        entList = v[len(v)-j-1]
+                        if (start>=int(entList[0]) and i<=int(entList[1])):
+                            emptyList[numberOfWords] = entList[2]
+                            break
+                        else:
+                            continue
+                start = i + 1  
+                numberOfWords += 1
+            if (i == lastIndexOfSpace):
+                for j in range(len(v)):
+                        entList = v[len(v)-j-1]
+                        if (lastIndexOfSpace>=int(entList[0]) and lenOfString<=int(entList[1])):
+                            emptyList[numberOfWords] = entList[2]
+                            numberOfWords += 1
+        cleanedDF = cleanedDF.append(pd.Series([emptyList],  index=cleanedDF.columns ), ignore_index=True )
+        sum1 = sum1 + numberOfWords
+    return cleanedDF

Transformer Mechanism/QA/tf/.Trash-0/files/QA_dataset.ipynb

@@ -0,0 +1,2510 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "TBjVwYpHJ7ra"
+   },
+   "source": [
+    "# Transformer Network Application: Question Answering\n",
+    "\n",
+    "Welcome to Week 4's third, and the last lab of the course! Congratulations on making it this far. In this notebook you'll explore another application of the transformer architecture that you built.\n",
+    "\n",
+    "**After this assignment you'll be able to**:\n",
+    "\n",
+    "* Perform extractive Question Answering \n",
+    "* Fine-tune a pre-trained transformer model to a custom dataset\n",
+    "* Implement a QA model in TensorFlow and PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "SoRb7ykXJ_C4"
+   },
+   "source": [
+    "## Table of Contents\n",
+    "\n",
+    "\n",
+    "- [1 - Extractive Question Answering](#1)\n",
+    "    - [1.1 - Data Cleaning](#1-1)\n",
+    "    - [1.2 - Tokenize and Align Labels with 🤗 Library](#1-2)\n",
+    "- [2 - Training](#2)\n",
+    "    - [2.1 TensorFlow implementation](#2-1)\n",
+    "    - [2.2 PyTorch implementation](#2-2)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "C0k56ZVXLDbi"
+   },
+   "source": [
+    "<a name='1'></a>\n",
+    "## 1 - Extractive Question Answering\n",
+    "\n",
+    "Question answering (QA) is a task of natural language processing that aims to automatically answer questions. The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question. For example, when tasked with answering the question 'When will Jane go to Africa?' given the text data 'Jane visits Africa in September', the question answering model will highlight 'September'.\n",
+    "\n",
+    "* You will use a variation of the Transformer model you built in the last assignment to answer questions about stories.\n",
+    "* You will implement extractive QA model in TensorFlow and in PyTorch.\n",
+    "\n",
+    "**Recommendation:**\n",
+    "* If you are interested, check out the [Course 4: Natural Language Processing with Attention Models](https://www.coursera.org/learn/attention-models-in-nlp/home/welcome) of our [Natural Language Processing Specialization](https://www.coursera.org/specializations/natural-language-processing?=) where you can learn how to build Transformers and perform QA using the [Trax](https://trax.readthedocs.io/en/latest/) library. \n",
+    "\n",
+    "<a name='1-1'></a>\n",
+    "### 1.1 - Data preprocessing\n",
+    "\n",
+    "Run the following cell to load the [QA bAbI dataset](https://research.fb.com/downloads/babi/), which is one of the bAbI datasets generated by Facebook AI Research to advance natural language processing."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install pyarrow==6.0.0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "XxU0G_PYLSXJ",
+    "outputId": "44e7877f-5c33-45fc-ed83-3aa4920dcc40"
+   },
+   "outputs": [
+    {
+     "ename": "ModuleNotFoundError",
+     "evalue": "No module named 'fsspec.archive'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
+      "Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mdatasets\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m load_from_disk\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m# Load a dataset and print the first example in the training set\u001b[39;00m\n\u001b[1;32m      4\u001b[0m babi_dataset \u001b[38;5;241m=\u001b[39m load_from_disk(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mdata/\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/__init__.py:43\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     40\u001b[0m \u001b[38;5;28;01mdel\u001b[39;00m pyarrow\n\u001b[1;32m     41\u001b[0m \u001b[38;5;28;01mdel\u001b[39;00m version\n\u001b[0;32m---> 43\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01marrow_dataset\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Dataset\n\u001b[1;32m     44\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01marrow_reader\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ReadInstruction\n\u001b[1;32m     45\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mbuilder\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/arrow_dataset.py:63\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtqdm\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mauto\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m tqdm\n\u001b[1;32m     62\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m config\n\u001b[0;32m---> 63\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01marrow_reader\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ArrowReader\n\u001b[1;32m     64\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01marrow_writer\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ArrowWriter, OptimizedTypedSequence\n\u001b[1;32m     65\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdownload\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdownload_config\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DownloadConfig\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/arrow_reader.py:29\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpyarrow\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mpa\u001b[39;00m\n\u001b[1;32m     27\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpyarrow\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mparquet\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mpq\u001b[39;00m\n\u001b[0;32m---> 29\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdownload\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdownload_config\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DownloadConfig\n\u001b[1;32m     30\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mnaming\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m _split_re, filenames_for_dataset_split\n\u001b[1;32m     31\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mtable\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m InMemoryTable, MemoryMappedTable, Table, concat_tables\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/download/__init__.py:10\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdownload_config\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DownloadConfig\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdownload_manager\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DownloadManager, DownloadMode\n\u001b[0;32m---> 10\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mstreaming_download_manager\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m StreamingDownloadManager\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/download/streaming_download_manager.py:20\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01maiohttp\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mclient_exceptions\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ClientError\n\u001b[1;32m     19\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m config\n\u001b[0;32m---> 20\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mfilesystems\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m COMPRESSION_FILESYSTEMS\n\u001b[1;32m     21\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutils\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mfile_utils\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     22\u001b[0m     get_authentication_headers_for_url,\n\u001b[1;32m     23\u001b[0m     http_head,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     27\u001b[0m     url_or_path_join,\n\u001b[1;32m     28\u001b[0m )\n\u001b[1;32m     29\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutils\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlogging\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m get_logger\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/filesystems/__init__.py:6\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m List\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mfsspec\u001b[39;00m\n\u001b[0;32m----> 6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m compression\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mhffilesystem\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m HfFileSystem\n\u001b[1;32m     10\u001b[0m _has_s3fs \u001b[38;5;241m=\u001b[39m importlib\u001b[38;5;241m.\u001b[39mutil\u001b[38;5;241m.\u001b[39mfind_spec(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124ms3fs\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n",
+      "File \u001b[0;32m/usr/local/lib/python3.8/dist-packages/datasets/filesystems/compression.py:5\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Optional\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mfsspec\u001b[39;00m\n\u001b[0;32m----> 5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mfsspec\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01marchive\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m AbstractArchiveFileSystem\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mfsspec\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutils\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DEFAULT_BLOCK_SIZE\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28;01mclass\u001b[39;00m \u001b[38;5;21;01mBaseCompressedFileFileSystem\u001b[39;00m(AbstractArchiveFileSystem):\n",
+      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'fsspec.archive'"
+     ]
+    }
+   ],
+   "source": [
+    "from datasets import load_from_disk\n",
+    "\n",
+    "# Load a dataset and print the first example in the training set\n",
+    "babi_dataset = load_from_disk('data/')\n",
+    "print(babi_dataset['train'][0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "XJwacC3bMhZM"
+   },
+   "source": [
+    "Take a look at the format of the data. For a given story, there are two sentences which serve as the context, and one question. Each of these phrases has an ID. There is also a supporting fact ID which refers to a sentence in the story that helps answer the question. For example, for the question 'What is east of the hallway?', the supporting fact 'The bedroom is east of the hallway' has the ID '2'. There is also the answer, 'bedroom' for the question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "aizPXfGlLZ1D",
+    "outputId": "0e1d47bc-9c1a-458a-983e-22f47f8184bd"
+   },
+   "outputs": [],
+   "source": [
+    "babi_dataset['train'][102]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ewtXZUPjMm2l"
+   },
+   "source": [
+    "Check and see if the entire dataset of stories has this format."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "55BSWxwuM1hN"
+   },
+   "outputs": [],
+   "source": [
+    "type_set = set()\n",
+    "for story in babi_dataset['train']:\n",
+    "    if str(story['story']['type'] )not in type_set:\n",
+    "        type_set.add(str(story['story']['type'] ))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "bdJ8VMF1UT7S",
+    "outputId": "2b959467-75e8-4e25-e7bb-481b657a2fce"
+   },
+   "outputs": [],
+   "source": [
+    "type_set"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "JsHx1tcyMq_k"
+   },
+   "source": [
+    "To make the data easier to work with, you will flatten the dataset to transform it from a dictionary structure to a table structure."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "YxixFI-pVOK9"
+   },
+   "outputs": [],
+   "source": [
+    "flattened_babi = babi_dataset.flatten()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "kXU43CqCdX98",
+    "outputId": "e968ff5e-0db0-4e9d-e1e9-e93f965b2582"
+   },
+   "outputs": [],
+   "source": [
+    "flattened_babi"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "OQw59MgT6Luh",
+    "outputId": "ea5eac53-027e-42d3-d19f-98ed7863de2b"
+   },
+   "outputs": [],
+   "source": [
+    "next(iter(flattened_babi['train']))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4vXfmhOPMvt1"
+   },
+   "source": [
+    "Now it is much easier to access the information you need! You can now easily extract the answer, question, and facts from the story, and also join the facts into a single entry under 'sentences'."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "O5NcABwkdbrf"
+   },
+   "outputs": [],
+   "source": [
+    "def get_question_and_facts(story):\n",
+    "    dic = {}\n",
+    "    dic['question'] = story['story.text'][2]\n",
+    "    dic['sentences'] = ' '.join([story['story.text'][0], story['story.text'][1]])\n",
+    "    dic['answer'] = story['story.answer'][2]\n",
+    "    return dic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 115,
+     "referenced_widgets": [
+      "44b7bea3e09d4e5684921c66dd4c7514",
+      "6af3ec5091d74bd1a95bf02a87dd240b",
+      "7e1325e57bf9417e93d7ef180794ab3c",
+      "3dab28395f3f475d8242e4d4d45ed059",
+      "ca722dcd857c433c9058585e31a1673d",
+      "7fb1118c0b4443b6b6dbb5803e9ec2e8",
+      "58718e12f1b7459989ab5296846c4be6",
+      "63b4ebafcead4c0784b5511219a6a198",
+      "c42644a4e6184a1cbdb2b453b5dbb7d6",
+      "364ba960eb474c9084cc71851594d345",
+      "e8f1abd85f3e49f991d4c1312ffd416b",
+      "929946fdfaa04cf59d3b31cf92fc08d1",
+      "aa5c0d374889482697fc0f7ce9c81afe",
+      "ff444b253e9a40e5bec755926d83740f",
+      "89fdda6e6688476495ca297bfe010bf8",
+      "cda72c45821a4eb89f1a3ab5510b26d3"
+     ]
+    },
+    "id": "LHKNQ75afMoZ",
+    "outputId": "6ceeae5c-392c-4553-c487-14a648eb9209"
+   },
+   "outputs": [],
+   "source": [
+    "processed = flattened_babi.map(get_question_and_facts)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "KaTacKMufPba",
+    "outputId": "2433d446-e985-45cd-a200-f9805b4056bd"
+   },
+   "outputs": [],
+   "source": [
+    "processed['train'][2]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "IOrYr5LI0pbP",
+    "outputId": "8142f23c-7dab-49b9-8027-fbe7364ae4e9"
+   },
+   "outputs": [],
+   "source": [
+    "processed['test'][2]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "oN7D3fszM2hy"
+   },
+   "source": [
+    "The goal of extractive QA is to find the part of the text that contains the answer to the question. You will identify the position of the answer using the indexes of the string. For example, if the answer to some question was 'September', you would need to find the start and end string indices of the word 'September' in the context sentence 'Jane visits Africa in September.'\n",
+    "\n",
+    "\n",
+    "Use this next function to get the start and end indices of the answer in each of the stories in your dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "J1JJx3PafSyR"
+   },
+   "outputs": [],
+   "source": [
+    "def get_start_end_idx(story):\n",
+    "    str_idx = story['sentences'].find(story['answer'])\n",
+    "    end_idx = str_idx + len(story['answer'])\n",
+    "    return {'str_idx':str_idx,\n",
+    "          'end_idx': end_idx}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 115,
+     "referenced_widgets": [
+      "8968319cdaca476fb15c11a388dce39a",
+      "863c5ce96db84e3da162072c9a13c913",
+      "a725734893004a45b61194f649f5f602",
+      "c4a24656d67844e995d3b8e175c6c497",
+      "4f5b06c3a5e44c6cade5bf83634d9f69",
+      "afc33fa78b5d440192c435bfca6f7914",
+      "f37bd346f8614fec92d6c5b5e9b66d2f",
+      "b4c6a18610734036a16a14a43174c52e",
+      "07aaa9b79a744856b19d723370d6e588",
+      "afedd2328cf141f78775e4cfa7758267",
+      "b39b85d8cb05418aa92e8476ad02f755",
+      "0a8534ac52af4d48ad82b66463ad08c3",
+      "3abb36da57c841838867c56e2a3a325b",
+      "8b961844b5004905922531bd805a9d57",
+      "31fc08a1e7e04f6b9b3ea400ccfaea75",
+      "8cfbd3b14b23417993270f851a2d8ff9"
+     ]
+    },
+    "id": "4e7BdgJJhwXi",
+    "outputId": "d9c7a923-d2eb-4533-f37e-4f269f22eb89"
+   },
+   "outputs": [],
+   "source": [
+    "processed = processed.map(get_start_end_idx)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "P8ytxyfvh0kB",
+    "outputId": "c008b161-be24-40bb-a32d-47d92e624787"
+   },
+   "outputs": [],
+   "source": [
+    "num = 187\n",
+    "print(processed['test'][num])\n",
+    "start_idx = processed['test'][num]['str_idx']\n",
+    "end_idx = processed['test'][num]['end_idx']\n",
+    "print('answer:', processed['test'][num]['sentences'][start_idx:end_idx])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "VVX3TA2xM-vJ"
+   },
+   "source": [
+    "<a name='1-2'></a>\n",
+    "### 1.2 - Tokenize and Align with 🤗 Library\n",
+    "\n",
+    "Now you have all the data you need to train a Transformer model to perform Question Answering! You are ready for a task you may have already encountered in the Named-Entity Recognition lab - tokenizing and aligning your input. To feed text data to a Transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use must match the Transformer model type you are using! In this exercise, you will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "c892hk9NNF9O"
+   },
+   "source": [
+    "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word with a `tokenize_and_align()` function. In this case, since you are interested in the start and end indices of the answer, you will want to align the index of the sentence to match the index of the token for a word. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "UI-9P7VYitxv"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import DistilBertTokenizerFast\n",
+    "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "Pex-YXJnnwb9"
+   },
+   "outputs": [],
+   "source": [
+    "def tokenize_align(example):\n",
+    "    encoding = tokenizer(example['sentences'], example['question'], truncation=True, padding=True, max_length=tokenizer.model_max_length)\n",
+    "    start_positions = encoding.char_to_token(example['str_idx'])\n",
+    "    end_positions = encoding.char_to_token(example['end_idx']-1)\n",
+    "    if start_positions is None:\n",
+    "        start_positions = tokenizer.model_max_length\n",
+    "    if end_positions is None:\n",
+    "        end_positions = tokenizer.model_max_length\n",
+    "    return {'input_ids': encoding['input_ids'],\n",
+    "          'attention_mask': encoding['attention_mask'],\n",
+    "          'start_positions': start_positions,\n",
+    "          'end_positions': end_positions}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 115,
+     "referenced_widgets": [
+      "4d9152a30e824931983a425ee6d607a6",
+      "1f2773e3e80c4dd8b6b26e171bf33bc7",
+      "013f041c3e0b4e35bf2432fc345cb7bf",
+      "ef4e12f29f1e458f811a400faf21bdcc",
+      "f0e34f2bf626434fa73f0def26b3d1a5",
+      "1e6c02317171453cbd3d4d665879b0d4",
+      "5b6dbe662ca24834b7678638e101e1ff",
+      "39029f730ae140c7902fca6dac5361ad",
+      "723acefae33d448199fa5c1a9ec3f246",
+      "32a5c82c7a9845c09c11bb4e30c2f1aa",
+      "77273c2e4b4e4e4c8ee4b6b344749518",
+      "f0ac3b9b8f664479940c6ee18fc2f13e",
+      "393697738e724e9fad4d163de0a77840",
+      "e592db98c0c34c5e800f5d7b6d3c099e",
+      "568f11b4462f4b4e95f3ad5947bb275e",
+      "7fefe9e1121a43558d773500aef8935c"
+     ]
+    },
+    "id": "kKyLNWCvksOr",
+    "outputId": "7af3d914-4546-430c-c2f0-206b732e5131"
+   },
+   "outputs": [],
+   "source": [
+    "qa_dataset = processed.map(tokenize_align)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "8v5odGZBmGw0"
+   },
+   "outputs": [],
+   "source": [
+    "qa_dataset = qa_dataset.remove_columns(['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "yBHzbjffmJa8",
+    "outputId": "b0688636-fdec-4de0-c2d9-69372b1ddbac"
+   },
+   "outputs": [],
+   "source": [
+    "qa_dataset['train'][200]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "qw79BQfW4feu"
+   },
+   "source": [
+    "<font color='blue'><b>What you should remember:</b>\n",
+    "- The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question.\n",
+    "- Transformer models are often trained by tokenizers that split words into subwords.\n",
+    "  - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
+    "</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "rFfJozZvNZWG"
+   },
+   "source": [
+    "<a name='2'></a>\n",
+    "# 2 - Training \n",
+    "\n",
+    "Now that you have finished tokenizing and aligning your data, you can feed it into a pre-trained 🤗 Transformer model! You will use a DistilBERT model, which matches the tokenizer you used to preprocess your data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "8sdX5XY0Gwwc"
+   },
+   "outputs": [],
+   "source": [
+    "train_ds = qa_dataset['train']\n",
+    "test_ds = qa_dataset['test']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "Be5k3ilHsJ6q",
+    "outputId": "f2f7fea3-1394-4aaf-b159-994a38476994"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import TFDistilBertForQuestionAnswering\n",
+    "model = TFDistilBertForQuestionAnswering.from_pretrained(\"model/tensorflow\", return_dict=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "-aQVOG4ANcd2"
+   },
+   "source": [
+    "<a name='2-1'></a>\n",
+    "### 2.1 - TensorFlow implementation\n",
+    "For this assignment you will execute two implemenations, one in TensorFlow and one in PyTorch.\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "8pCRo_parYMc"
+   },
+   "source": [
+    "\n",
+    "#### Train and test datasets\n",
+    "\n",
+    "**Note:**\n",
+    "* In the TensorFlow implementation, you will have to set the data format type to tensors, which may create ragged tensors (tensors of different lengths). \n",
+    "* You will have to convert the ragged tensors to normal tensors using the `to_tensor()` method, which pads the tensors and sets the dimensions to `[None, tokenizer.model_max_length]` so you can feed different size tensors into your model based on the batch size.  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "FbpplBxNtanH"
+   },
+   "outputs": [],
+   "source": [
+    "import tensorflow as tf\n",
+    "\n",
+    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+    "\n",
+    "train_ds.set_format(type='tf', columns=columns_to_return)\n",
+    "\n",
+    "train_features = {x: train_ds[x].to_tensor(default_value=0, shape=[None, tokenizer.model_max_length]) for x in ['input_ids', 'attention_mask']}\n",
+    "train_labels = {\"start_positions\": tf.reshape(train_ds['start_positions'], shape=[-1,1]),\n",
+    "                'end_positions': tf.reshape(train_ds['end_positions'], shape=[-1,1])}\n",
+    "\n",
+    "\n",
+    "train_tfdataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels)).batch(8)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "0_Jj8Av6rEuN"
+   },
+   "source": [
+    "#### Training \n",
+    "\n",
+    "It is finally time to start training your model! \n",
+    "\n",
+    "* Create a custom training function using [tf.GradientTape()](https://www.tensorflow.org/api_docs/python/tf/GradientTape)\n",
+    "* Target two loss functions, one for the start index and one for the end index. \n",
+    "* `tf.GradientTape()` records the operations performed during forward prop for automatic differentiation during backprop. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "PtZz249vQbLn",
+    "outputId": "24cdf861-af63-4581-a0ae-2de29d1880ed"
+   },
+   "outputs": [],
+   "source": [
+    "EPOCHS = 3\n",
+    "loss_fn1 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
+    "loss_fn2 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
+    "opt = tf.keras.optimizers.Adam(learning_rate=3e-5)\n",
+    "\n",
+    "losses = []\n",
+    "for epoch in range(EPOCHS):\n",
+    "    print(\"Starting epoch: %d\"% epoch )\n",
+    "    for step, (x_batch_train, y_batch_train) in enumerate(train_tfdataset):\n",
+    "        with tf.GradientTape() as tape:\n",
+    "            answer_start_scores, answer_end_scores = model(x_batch_train)\n",
+    "            loss_start = loss_fn1(y_batch_train['start_positions'], answer_start_scores)\n",
+    "            loss_end = loss_fn2(y_batch_train['end_positions'], answer_end_scores)\n",
+    "            loss = 0.5 * (loss_start + loss_end)\n",
+    "        losses.append(loss)\n",
+    "        grads = tape.gradient(loss, model.trainable_weights)\n",
+    "        opt.apply_gradients(zip(grads, model.trainable_weights))\n",
+    "\n",
+    "        if step % 20 == 0:\n",
+    "            print(\"Training loss (for one batch) at step %d: %.4f\"% (step, \n",
+    "                                                                   float(loss_start)))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Q8ggB0JUWQuW"
+   },
+   "source": [
+    "Take a look at your losses and try playing around with some of the hyperparameters for better results!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 282
+    },
+    "id": "fK91EPvRYFcX",
+    "outputId": "6b7099dd-f918-4905-e3a3-fcce2880e506"
+   },
+   "outputs": [],
+   "source": [
+    "from matplotlib.pyplot import plot\n",
+    "\n",
+    "plot(losses)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "64OtEmyUWUiM"
+   },
+   "source": [
+    "You have successfully trained your model to help automatically answer questions! Try asking it a question about a story."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "eFniMzpp1bpz",
+    "outputId": "0ce0e2a3-3d6a-4e6e-adff-d0c16b622c9a"
+   },
+   "outputs": [],
+   "source": [
+    "question, text = 'What is south of the bedroom?','The hallway is south of the garden. The garden is south of the bedroom.'\n",
+    "input_dict = tokenizer(text, question, return_tensors='tf')\n",
+    "outputs = model(input_dict)\n",
+    "start_logits = outputs[0]\n",
+    "end_logits = outputs[1]\n",
+    "\n",
+    "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
+    "answer = ' '.join(all_tokens[tf.math.argmax(start_logits, 1)[0] : tf.math.argmax(end_logits, 1)[0]+1])\n",
+    "print(question, answer.capitalize())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f07OtnCpuKFa"
+   },
+   "source": [
+    "Congratulations! You just implemented your first QA model in TensorFlow. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "9UaM5pY9u8EW"
+   },
+   "source": [
+    "<a name='2-1'></a>\n",
+    "## 2.2 PyTorch implementation\n",
+    "\n",
+    "[PyTorch](https://pytorch.org/) is an open source machine learning framework developed by Facebook's AI Research lab that can be used for computer vision and natural language processing. As you can imagine, it is quite compatible with the bAbI dataset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "nD9akXoXxMjd"
+   },
+   "source": [
+    "#### Train and test dataset\n",
+    "\n",
+    "Go ahead and try creating a train and test dataset by importing PyTorch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "JxMYWSG173ch"
+   },
+   "outputs": [],
+   "source": [
+    "from torch.utils.data import DataLoader\n",
+    "\n",
+    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+    "train_ds.set_format(type='pt', columns=columns_to_return)\n",
+    "test_ds.set_format(type='pt', columns=columns_to_return)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "OeuzZKlPHAAQ"
+   },
+   "source": [
+    "For the accuracy metrics for the PyTorch implementation, you will change things up a bit and use the [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) for start and end indicies over the entire test dataset as the loss functions. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "aD9tDpZfJsIB"
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.metrics import f1_score\n",
+    "\n",
+    "def compute_metrics(pred):\n",
+    "    start_labels = pred.label_ids[0]\n",
+    "    start_preds = pred.predictions[0].argmax(-1)\n",
+    "    end_labels = pred.label_ids[1]\n",
+    "    end_preds = pred.predictions[1].argmax(-1)\n",
+    "    \n",
+    "    f1_start = f1_score(start_labels, start_preds, average='macro')\n",
+    "    f1_end = f1_score(end_labels, end_preds, average='macro')\n",
+    "    \n",
+    "    return {\n",
+    "        'f1_start': f1_start,\n",
+    "        'f1_end': f1_end,\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "laX5cYQRHMXb"
+   },
+   "source": [
+    "#### Training\n",
+    "\n",
+    "Now it is time to load a pre-trained model. \n",
+    "\n",
+    "**Note:** You will be using the DistilBERT instead of TFDistilBERT for a PyTorch implementation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "del model # We delete the tensorflow model to avoid memory issues"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "YXFCsNcY79jx",
+    "outputId": "09af112f-e1e9-4a47-c988-37ee2a068df2"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import DistilBertForQuestionAnswering\n",
+    "\n",
+    "pytorch_model = DistilBertForQuestionAnswering.from_pretrained(\"model/pytorch\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "xCUdMmCxHP6_"
+   },
+   "source": [
+    "Instead of a custom training loop, you will use the [🤗 Trainer](https://huggingface.co/transformers/main_classes/trainer.html), which contains a basic training loop and is fairly easy to implement in PyTorch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 329
+    },
+    "id": "1htmS3TV-2Bk",
+    "outputId": "cc21bfbb-da09-47f9-ee16-7db0096d35e7"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import Trainer, TrainingArguments\n",
+    "\n",
+    "training_args = TrainingArguments(\n",
+    "    output_dir='results',          # output directory\n",
+    "    overwrite_output_dir=True,\n",
+    "    num_train_epochs=3,              # total number of training epochs\n",
+    "    per_device_train_batch_size=8,  # batch size per device during training\n",
+    "    per_device_eval_batch_size=8,   # batch size for evaluation\n",
+    "    warmup_steps=20,                # number of warmup steps for learning rate scheduler\n",
+    "    weight_decay=0.01,               # strength of weight decay\n",
+    "    logging_dir=None,            # directory for storing logs\n",
+    "    logging_steps=50\n",
+    ")\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=pytorch_model,                 # the instantiated 🤗 Transformers model to be trained\n",
+    "    args=training_args,                  # training arguments, defined above\n",
+    "    train_dataset=train_ds,         # training dataset\n",
+    "    eval_dataset=test_ds,\n",
+    "    compute_metrics=compute_metrics             # evaluation dataset\n",
+    ")\n",
+    "\n",
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 207
+    },
+    "id": "lDzbm7vzAiPJ",
+    "outputId": "7cd62f51-a04b-4583-bc0e-e459813d3103"
+   },
+   "outputs": [],
+   "source": [
+    "trainer.evaluate(test_ds)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "QAgrcs2pHvVu"
+   },
+   "source": [
+    "Now it is time to ask your PyTorch model a question! \n",
+    "* Before testing your model with a question, you can tell PyTorch to send your model and inputs to the GPU if your machine has one, or the CPU if it does not. \n",
+    "* You can then proceed to tokenize your input and create PyTorch tensors and send them to your device. \n",
+    "* The rest of the pipeline is relatively similar to the one you implemented for TensorFlow.   \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "yfBe9AFABqUr",
+    "outputId": "b5ca6039-8ce2-4e75-9161-1c96a0f39425"
+   },
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
+    "\n",
+    "pytorch_model.to(device)\n",
+    "\n",
+    "question, text = 'What is east of the hallway?','The kitchen is east of the hallway. The garden is south of the bedroom.'\n",
+    "\n",
+    "input_dict = tokenizer(text, question, return_tensors='pt')\n",
+    "\n",
+    "input_ids = input_dict['input_ids'].to(device)\n",
+    "attention_mask = input_dict['attention_mask'].to(device)\n",
+    "\n",
+    "outputs = pytorch_model(input_ids, attention_mask=attention_mask)\n",
+    "\n",
+    "start_logits = outputs[0]\n",
+    "end_logits = outputs[1]\n",
+    "\n",
+    "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
+    "answer = ' '.join(all_tokens[torch.argmax(start_logits, 1)[0] : torch.argmax(end_logits, 1)[0]+1])\n",
+    "\n",
+    "print(question, answer.capitalize())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "eGzuHkMZ4q9I"
+   },
+   "source": [
+    "### Congratulations!\n",
+    " \n",
+    "You've completed this notebook, and can now implement Transformer models for QA tasks!\n",
+    "\n",
+    "You are now able to:\n",
+    "* Perform extractive Question Answering \n",
+    "* Fine-tune a pre-trained transformer model to a custom dataset\n",
+    "* Implement a QA model in TensorFlow and PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "G8tAV-584vKE"
+   },
+   "source": [
+    "<font color='blue'><b>What you should remember</b>:\n",
+    "- Transformer models are often trained by tokenizers that split words into subwords.\n",
+    "  - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
+    "- PyTorch is a relatively light and easy to implement framework that can make rapid prototyping easier, while TensorFlow has advantages in scaling and is more widely used in production\n",
+    "  - `tf.GradientTape` allows you to build custom training loops in TensorFlow\n",
+    "  - The `Trainer` API in PyTorch gives you a basic training loop that is compatible with 🤗 models and datasets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%javascript\n",
+    "let element = document.getElementById('submit-notebook-button-group');\n",
+    "if (!element) {\n",
+    "    window._save_and_close = function(){\n",
+    "        IPython.notebook.save_checkpoint();\n",
+    "        IPython.notebook.session.delete();\n",
+    "        window.onbeforeunload = null\n",
+    "        setTimeout(function() {window.close();}, 1000)\n",
+    "    }\n",
+    "    let header = document.getElementById('maintoolbar-container');\n",
+    "    element = document.createElement(\"div\");\n",
+    "    element.setAttribute(\"class\", \"btn-group\");\n",
+    "    element.setAttribute(\"id\", \"submit-notebook-button-group\");\n",
+    "    element.setAttribute(\"align\", \"right\");\n",
+    "    element.setAttribute(\"style\", \"float:right\")\n",
+    "    element.innerHTML = '<button class=\"btn btn-default\" title=\"Save and close this notebook.\" style=\"background-color:rgb(42, 115, 204); color:white; padding:4px 8px\" onclick=window._save_and_close()>Save and close</button>'\n",
+    "    header.appendChild(element); \n",
+    "}                    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "collapsed_sections": [],
+   "name": "QA-dataset.ipynb",
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "013f041c3e0b4e35bf2432fc345cb7bf": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_1e6c02317171453cbd3d4d665879b0d4",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_f0e34f2bf626434fa73f0def26b3d1a5",
+      "value": 1000
+     }
+    },
+    "07aaa9b79a744856b19d723370d6e588": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_b39b85d8cb05418aa92e8476ad02f755",
+       "IPY_MODEL_0a8534ac52af4d48ad82b66463ad08c3"
+      ],
+      "layout": "IPY_MODEL_afedd2328cf141f78775e4cfa7758267"
+     }
+    },
+    "0a8534ac52af4d48ad82b66463ad08c3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_8cfbd3b14b23417993270f851a2d8ff9",
+      "placeholder": "​",
+      "style": "IPY_MODEL_31fc08a1e7e04f6b9b3ea400ccfaea75",
+      "value": " 1000/1000 [01:40&lt;00:00,  9.90ex/s]"
+     }
+    },
+    "1e6c02317171453cbd3d4d665879b0d4": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "1f2773e3e80c4dd8b6b26e171bf33bc7": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "31fc08a1e7e04f6b9b3ea400ccfaea75": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "32a5c82c7a9845c09c11bb4e30c2f1aa": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "364ba960eb474c9084cc71851594d345": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "39029f730ae140c7902fca6dac5361ad": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "393697738e724e9fad4d163de0a77840": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "3abb36da57c841838867c56e2a3a325b": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "3dab28395f3f475d8242e4d4d45ed059": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_63b4ebafcead4c0784b5511219a6a198",
+      "placeholder": "​",
+      "style": "IPY_MODEL_58718e12f1b7459989ab5296846c4be6",
+      "value": " 1000/1000 [00:10&lt;00:00, 97.35ex/s]"
+     }
+    },
+    "44b7bea3e09d4e5684921c66dd4c7514": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_7e1325e57bf9417e93d7ef180794ab3c",
+       "IPY_MODEL_3dab28395f3f475d8242e4d4d45ed059"
+      ],
+      "layout": "IPY_MODEL_6af3ec5091d74bd1a95bf02a87dd240b"
+     }
+    },
+    "4d9152a30e824931983a425ee6d607a6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_013f041c3e0b4e35bf2432fc345cb7bf",
+       "IPY_MODEL_ef4e12f29f1e458f811a400faf21bdcc"
+      ],
+      "layout": "IPY_MODEL_1f2773e3e80c4dd8b6b26e171bf33bc7"
+     }
+    },
+    "4f5b06c3a5e44c6cade5bf83634d9f69": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "568f11b4462f4b4e95f3ad5947bb275e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "58718e12f1b7459989ab5296846c4be6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "5b6dbe662ca24834b7678638e101e1ff": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "63b4ebafcead4c0784b5511219a6a198": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "6af3ec5091d74bd1a95bf02a87dd240b": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "723acefae33d448199fa5c1a9ec3f246": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_77273c2e4b4e4e4c8ee4b6b344749518",
+       "IPY_MODEL_f0ac3b9b8f664479940c6ee18fc2f13e"
+      ],
+      "layout": "IPY_MODEL_32a5c82c7a9845c09c11bb4e30c2f1aa"
+     }
+    },
+    "77273c2e4b4e4e4c8ee4b6b344749518": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_e592db98c0c34c5e800f5d7b6d3c099e",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_393697738e724e9fad4d163de0a77840",
+      "value": 1000
+     }
+    },
+    "7e1325e57bf9417e93d7ef180794ab3c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_7fb1118c0b4443b6b6dbb5803e9ec2e8",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_ca722dcd857c433c9058585e31a1673d",
+      "value": 1000
+     }
+    },
+    "7fb1118c0b4443b6b6dbb5803e9ec2e8": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "7fefe9e1121a43558d773500aef8935c": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "863c5ce96db84e3da162072c9a13c913": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "8968319cdaca476fb15c11a388dce39a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_a725734893004a45b61194f649f5f602",
+       "IPY_MODEL_c4a24656d67844e995d3b8e175c6c497"
+      ],
+      "layout": "IPY_MODEL_863c5ce96db84e3da162072c9a13c913"
+     }
+    },
+    "89fdda6e6688476495ca297bfe010bf8": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "8b961844b5004905922531bd805a9d57": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "8cfbd3b14b23417993270f851a2d8ff9": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "929946fdfaa04cf59d3b31cf92fc08d1": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_cda72c45821a4eb89f1a3ab5510b26d3",
+      "placeholder": "​",
+      "style": "IPY_MODEL_89fdda6e6688476495ca297bfe010bf8",
+      "value": " 1000/1000 [00:08&lt;00:00, 123.32ex/s]"
+     }
+    },
+    "a725734893004a45b61194f649f5f602": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_afc33fa78b5d440192c435bfca6f7914",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_4f5b06c3a5e44c6cade5bf83634d9f69",
+      "value": 1000
+     }
+    },
+    "aa5c0d374889482697fc0f7ce9c81afe": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "afc33fa78b5d440192c435bfca6f7914": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "afedd2328cf141f78775e4cfa7758267": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b39b85d8cb05418aa92e8476ad02f755": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_8b961844b5004905922531bd805a9d57",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_3abb36da57c841838867c56e2a3a325b",
+      "value": 1000
+     }
+    },
+    "b4c6a18610734036a16a14a43174c52e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "c42644a4e6184a1cbdb2b453b5dbb7d6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_e8f1abd85f3e49f991d4c1312ffd416b",
+       "IPY_MODEL_929946fdfaa04cf59d3b31cf92fc08d1"
+      ],
+      "layout": "IPY_MODEL_364ba960eb474c9084cc71851594d345"
+     }
+    },
+    "c4a24656d67844e995d3b8e175c6c497": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_b4c6a18610734036a16a14a43174c52e",
+      "placeholder": "​",
+      "style": "IPY_MODEL_f37bd346f8614fec92d6c5b5e9b66d2f",
+      "value": " 1000/1000 [01:41&lt;00:00,  9.86ex/s]"
+     }
+    },
+    "ca722dcd857c433c9058585e31a1673d": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "cda72c45821a4eb89f1a3ab5510b26d3": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "e592db98c0c34c5e800f5d7b6d3c099e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "e8f1abd85f3e49f991d4c1312ffd416b": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_ff444b253e9a40e5bec755926d83740f",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_aa5c0d374889482697fc0f7ce9c81afe",
+      "value": 1000
+     }
+    },
+    "ef4e12f29f1e458f811a400faf21bdcc": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_39029f730ae140c7902fca6dac5361ad",
+      "placeholder": "​",
+      "style": "IPY_MODEL_5b6dbe662ca24834b7678638e101e1ff",
+      "value": " 1000/1000 [01:25&lt;00:00, 11.68ex/s]"
+     }
+    },
+    "f0ac3b9b8f664479940c6ee18fc2f13e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_7fefe9e1121a43558d773500aef8935c",
+      "placeholder": "​",
+      "style": "IPY_MODEL_568f11b4462f4b4e95f3ad5947bb275e",
+      "value": " 1000/1000 [01:24&lt;00:00, 11.77ex/s]"
+     }
+    },
+    "f0e34f2bf626434fa73f0def26b3d1a5": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "f37bd346f8614fec92d6c5b5e9b66d2f": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "ff444b253e9a40e5bec755926d83740f": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

Transformer Mechanism/QA/tf/.Trash-0/files/W4A2.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2078c9a6640abf78f244c07b6e5863cfd8b3e9b3d563010e40353df03bc2abdb
+size 448771063

Transformer Mechanism/QA/tf/.Trash-0/files/W4A3UGLQA.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a356ecbc0ab59f15b6bb3708be0f1a5ad495b1db61be7b97781374b509f3c9d6
+size 490112767

Transformer Mechanism/QA/tf/.Trash-0/info/QA_dataset.ipynb.trashinfo

@@ -0,0 +1,3 @@
+[Trash Info]
+Path=W4A3_UGL/QA_dataset.ipynb
+DeletionDate=2022-12-19T21:11:08

Transformer Mechanism/QA/tf/.Trash-0/info/W4A2.tar.gz.trashinfo

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2c55e5daa95f2064559c98ace901b6fa7316e88069a8cd4ada77c73c7b53100
+size 63

Transformer Mechanism/QA/tf/.Trash-0/info/W4A3UGLQA.tar.gz.trashinfo

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b25931e152b0a335d4129d856f91d75e8550688320c55ec4a0d76fd0000024e2
+size 68

Transformer Mechanism/QA/tf/W4A3_UGL/.ipynb_checkpoints/QA_dataset-checkpoint.ipynb

@@ -0,0 +1,2483 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "TBjVwYpHJ7ra"
+   },
+   "source": [
+    "# Transformer Network Application: Question Answering\n",
+    "\n",
+    "Welcome to Week 4's third, and the last lab of the course! Congratulations on making it this far. In this notebook you'll explore another application of the transformer architecture that you built.\n",
+    "\n",
+    "**After this assignment you'll be able to**:\n",
+    "\n",
+    "* Perform extractive Question Answering \n",
+    "* Fine-tune a pre-trained transformer model to a custom dataset\n",
+    "* Implement a QA model in TensorFlow and PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "SoRb7ykXJ_C4"
+   },
+   "source": [
+    "## Table of Contents\n",
+    "\n",
+    "\n",
+    "- [1 - Extractive Question Answering](#1)\n",
+    "    - [1.1 - Data Cleaning](#1-1)\n",
+    "    - [1.2 - Tokenize and Align Labels with 🤗 Library](#1-2)\n",
+    "- [2 - Training](#2)\n",
+    "    - [2.1 TensorFlow implementation](#2-1)\n",
+    "    - [2.2 PyTorch implementation](#2-2)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "C0k56ZVXLDbi"
+   },
+   "source": [
+    "<a name='1'></a>\n",
+    "## 1 - Extractive Question Answering\n",
+    "\n",
+    "Question answering (QA) is a task of natural language processing that aims to automatically answer questions. The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question. For example, when tasked with answering the question 'When will Jane go to Africa?' given the text data 'Jane visits Africa in September', the question answering model will highlight 'September'.\n",
+    "\n",
+    "* You will use a variation of the Transformer model you built in the last assignment to answer questions about stories.\n",
+    "* You will implement extractive QA model in TensorFlow and in PyTorch.\n",
+    "\n",
+    "**Recommendation:**\n",
+    "* If you are interested, check out the [Course 4: Natural Language Processing with Attention Models](https://www.coursera.org/learn/attention-models-in-nlp/home/welcome) of our [Natural Language Processing Specialization](https://www.coursera.org/specializations/natural-language-processing?=) where you can learn how to build Transformers and perform QA using the [Trax](https://trax.readthedocs.io/en/latest/) library. \n",
+    "\n",
+    "<a name='1-1'></a>\n",
+    "### 1.1 - Data preprocessing\n",
+    "\n",
+    "Run the following cell to load the [QA bAbI dataset](https://research.fb.com/downloads/babi/), which is one of the bAbI datasets generated by Facebook AI Research to advance natural language processing."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "XxU0G_PYLSXJ",
+    "outputId": "44e7877f-5c33-45fc-ed83-3aa4920dcc40"
+   },
+   "outputs": [],
+   "source": [
+    "from datasets import load_from_disk\n",
+    "\n",
+    "# Load a dataset and print the first example in the training set\n",
+    "babi_dataset = load_from_disk('data/')\n",
+    "print(babi_dataset['train'][0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "XJwacC3bMhZM"
+   },
+   "source": [
+    "Take a look at the format of the data. For a given story, there are two sentences which serve as the context, and one question. Each of these phrases has an ID. There is also a supporting fact ID which refers to a sentence in the story that helps answer the question. For example, for the question 'What is east of the hallway?', the supporting fact 'The bedroom is east of the hallway' has the ID '2'. There is also the answer, 'bedroom' for the question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "aizPXfGlLZ1D",
+    "outputId": "0e1d47bc-9c1a-458a-983e-22f47f8184bd"
+   },
+   "outputs": [],
+   "source": [
+    "babi_dataset['train'][102]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ewtXZUPjMm2l"
+   },
+   "source": [
+    "Check and see if the entire dataset of stories has this format."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "55BSWxwuM1hN"
+   },
+   "outputs": [],
+   "source": [
+    "type_set = set()\n",
+    "for story in babi_dataset['train']:\n",
+    "    if str(story['story']['type'] )not in type_set:\n",
+    "        type_set.add(str(story['story']['type'] ))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "bdJ8VMF1UT7S",
+    "outputId": "2b959467-75e8-4e25-e7bb-481b657a2fce"
+   },
+   "outputs": [],
+   "source": [
+    "type_set"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "JsHx1tcyMq_k"
+   },
+   "source": [
+    "To make the data easier to work with, you will flatten the dataset to transform it from a dictionary structure to a table structure."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "YxixFI-pVOK9"
+   },
+   "outputs": [],
+   "source": [
+    "flattened_babi = babi_dataset.flatten()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "kXU43CqCdX98",
+    "outputId": "e968ff5e-0db0-4e9d-e1e9-e93f965b2582"
+   },
+   "outputs": [],
+   "source": [
+    "flattened_babi"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "OQw59MgT6Luh",
+    "outputId": "ea5eac53-027e-42d3-d19f-98ed7863de2b"
+   },
+   "outputs": [],
+   "source": [
+    "next(iter(flattened_babi['train']))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4vXfmhOPMvt1"
+   },
+   "source": [
+    "Now it is much easier to access the information you need! You can now easily extract the answer, question, and facts from the story, and also join the facts into a single entry under 'sentences'."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "O5NcABwkdbrf"
+   },
+   "outputs": [],
+   "source": [
+    "def get_question_and_facts(story):\n",
+    "    dic = {}\n",
+    "    dic['question'] = story['story.text'][2]\n",
+    "    dic['sentences'] = ' '.join([story['story.text'][0], story['story.text'][1]])\n",
+    "    dic['answer'] = story['story.answer'][2]\n",
+    "    return dic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 115,
+     "referenced_widgets": [
+      "44b7bea3e09d4e5684921c66dd4c7514",
+      "6af3ec5091d74bd1a95bf02a87dd240b",
+      "7e1325e57bf9417e93d7ef180794ab3c",
+      "3dab28395f3f475d8242e4d4d45ed059",
+      "ca722dcd857c433c9058585e31a1673d",
+      "7fb1118c0b4443b6b6dbb5803e9ec2e8",
+      "58718e12f1b7459989ab5296846c4be6",
+      "63b4ebafcead4c0784b5511219a6a198",
+      "c42644a4e6184a1cbdb2b453b5dbb7d6",
+      "364ba960eb474c9084cc71851594d345",
+      "e8f1abd85f3e49f991d4c1312ffd416b",
+      "929946fdfaa04cf59d3b31cf92fc08d1",
+      "aa5c0d374889482697fc0f7ce9c81afe",
+      "ff444b253e9a40e5bec755926d83740f",
+      "89fdda6e6688476495ca297bfe010bf8",
+      "cda72c45821a4eb89f1a3ab5510b26d3"
+     ]
+    },
+    "id": "LHKNQ75afMoZ",
+    "outputId": "6ceeae5c-392c-4553-c487-14a648eb9209"
+   },
+   "outputs": [],
+   "source": [
+    "processed = flattened_babi.map(get_question_and_facts)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "KaTacKMufPba",
+    "outputId": "2433d446-e985-45cd-a200-f9805b4056bd"
+   },
+   "outputs": [],
+   "source": [
+    "processed['train'][2]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "IOrYr5LI0pbP",
+    "outputId": "8142f23c-7dab-49b9-8027-fbe7364ae4e9"
+   },
+   "outputs": [],
+   "source": [
+    "processed['test'][2]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "oN7D3fszM2hy"
+   },
+   "source": [
+    "The goal of extractive QA is to find the part of the text that contains the answer to the question. You will identify the position of the answer using the indexes of the string. For example, if the answer to some question was 'September', you would need to find the start and end string indices of the word 'September' in the context sentence 'Jane visits Africa in September.'\n",
+    "\n",
+    "\n",
+    "Use this next function to get the start and end indices of the answer in each of the stories in your dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "J1JJx3PafSyR"
+   },
+   "outputs": [],
+   "source": [
+    "def get_start_end_idx(story):\n",
+    "    str_idx = story['sentences'].find(story['answer'])\n",
+    "    end_idx = str_idx + len(story['answer'])\n",
+    "    return {'str_idx':str_idx,\n",
+    "          'end_idx': end_idx}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 115,
+     "referenced_widgets": [
+      "8968319cdaca476fb15c11a388dce39a",
+      "863c5ce96db84e3da162072c9a13c913",
+      "a725734893004a45b61194f649f5f602",
+      "c4a24656d67844e995d3b8e175c6c497",
+      "4f5b06c3a5e44c6cade5bf83634d9f69",
+      "afc33fa78b5d440192c435bfca6f7914",
+      "f37bd346f8614fec92d6c5b5e9b66d2f",
+      "b4c6a18610734036a16a14a43174c52e",
+      "07aaa9b79a744856b19d723370d6e588",
+      "afedd2328cf141f78775e4cfa7758267",
+      "b39b85d8cb05418aa92e8476ad02f755",
+      "0a8534ac52af4d48ad82b66463ad08c3",
+      "3abb36da57c841838867c56e2a3a325b",
+      "8b961844b5004905922531bd805a9d57",
+      "31fc08a1e7e04f6b9b3ea400ccfaea75",
+      "8cfbd3b14b23417993270f851a2d8ff9"
+     ]
+    },
+    "id": "4e7BdgJJhwXi",
+    "outputId": "d9c7a923-d2eb-4533-f37e-4f269f22eb89"
+   },
+   "outputs": [],
+   "source": [
+    "processed = processed.map(get_start_end_idx)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "P8ytxyfvh0kB",
+    "outputId": "c008b161-be24-40bb-a32d-47d92e624787"
+   },
+   "outputs": [],
+   "source": [
+    "num = 187\n",
+    "print(processed['test'][num])\n",
+    "start_idx = processed['test'][num]['str_idx']\n",
+    "end_idx = processed['test'][num]['end_idx']\n",
+    "print('answer:', processed['test'][num]['sentences'][start_idx:end_idx])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "VVX3TA2xM-vJ"
+   },
+   "source": [
+    "<a name='1-2'></a>\n",
+    "### 1.2 - Tokenize and Align with 🤗 Library\n",
+    "\n",
+    "Now you have all the data you need to train a Transformer model to perform Question Answering! You are ready for a task you may have already encountered in the Named-Entity Recognition lab - tokenizing and aligning your input. To feed text data to a Transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use must match the Transformer model type you are using! In this exercise, you will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "c892hk9NNF9O"
+   },
+   "source": [
+    "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word with a `tokenize_and_align()` function. In this case, since you are interested in the start and end indices of the answer, you will want to align the index of the sentence to match the index of the token for a word. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "UI-9P7VYitxv"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import DistilBertTokenizerFast\n",
+    "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "Pex-YXJnnwb9"
+   },
+   "outputs": [],
+   "source": [
+    "def tokenize_align(example):\n",
+    "    encoding = tokenizer(example['sentences'], example['question'], truncation=True, padding=True, max_length=tokenizer.model_max_length)\n",
+    "    start_positions = encoding.char_to_token(example['str_idx'])\n",
+    "    end_positions = encoding.char_to_token(example['end_idx']-1)\n",
+    "    if start_positions is None:\n",
+    "        start_positions = tokenizer.model_max_length\n",
+    "    if end_positions is None:\n",
+    "        end_positions = tokenizer.model_max_length\n",
+    "    return {'input_ids': encoding['input_ids'],\n",
+    "          'attention_mask': encoding['attention_mask'],\n",
+    "          'start_positions': start_positions,\n",
+    "          'end_positions': end_positions}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 115,
+     "referenced_widgets": [
+      "4d9152a30e824931983a425ee6d607a6",
+      "1f2773e3e80c4dd8b6b26e171bf33bc7",
+      "013f041c3e0b4e35bf2432fc345cb7bf",
+      "ef4e12f29f1e458f811a400faf21bdcc",
+      "f0e34f2bf626434fa73f0def26b3d1a5",
+      "1e6c02317171453cbd3d4d665879b0d4",
+      "5b6dbe662ca24834b7678638e101e1ff",
+      "39029f730ae140c7902fca6dac5361ad",
+      "723acefae33d448199fa5c1a9ec3f246",
+      "32a5c82c7a9845c09c11bb4e30c2f1aa",
+      "77273c2e4b4e4e4c8ee4b6b344749518",
+      "f0ac3b9b8f664479940c6ee18fc2f13e",
+      "393697738e724e9fad4d163de0a77840",
+      "e592db98c0c34c5e800f5d7b6d3c099e",
+      "568f11b4462f4b4e95f3ad5947bb275e",
+      "7fefe9e1121a43558d773500aef8935c"
+     ]
+    },
+    "id": "kKyLNWCvksOr",
+    "outputId": "7af3d914-4546-430c-c2f0-206b732e5131"
+   },
+   "outputs": [],
+   "source": [
+    "qa_dataset = processed.map(tokenize_align)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "8v5odGZBmGw0"
+   },
+   "outputs": [],
+   "source": [
+    "qa_dataset = qa_dataset.remove_columns(['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "yBHzbjffmJa8",
+    "outputId": "b0688636-fdec-4de0-c2d9-69372b1ddbac"
+   },
+   "outputs": [],
+   "source": [
+    "qa_dataset['train'][200]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "qw79BQfW4feu"
+   },
+   "source": [
+    "<font color='blue'><b>What you should remember:</b>\n",
+    "- The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question.\n",
+    "- Transformer models are often trained by tokenizers that split words into subwords.\n",
+    "  - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
+    "</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "rFfJozZvNZWG"
+   },
+   "source": [
+    "<a name='2'></a>\n",
+    "# 2 - Training \n",
+    "\n",
+    "Now that you have finished tokenizing and aligning your data, you can feed it into a pre-trained 🤗 Transformer model! You will use a DistilBERT model, which matches the tokenizer you used to preprocess your data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "8sdX5XY0Gwwc"
+   },
+   "outputs": [],
+   "source": [
+    "train_ds = qa_dataset['train']\n",
+    "test_ds = qa_dataset['test']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "Be5k3ilHsJ6q",
+    "outputId": "f2f7fea3-1394-4aaf-b159-994a38476994"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import TFDistilBertForQuestionAnswering\n",
+    "model = TFDistilBertForQuestionAnswering.from_pretrained(\"model/tensorflow\", return_dict=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "-aQVOG4ANcd2"
+   },
+   "source": [
+    "<a name='2-1'></a>\n",
+    "### 2.1 - TensorFlow implementation\n",
+    "For this assignment you will execute two implemenations, one in TensorFlow and one in PyTorch.\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "8pCRo_parYMc"
+   },
+   "source": [
+    "\n",
+    "#### Train and test datasets\n",
+    "\n",
+    "**Note:**\n",
+    "* In the TensorFlow implementation, you will have to set the data format type to tensors, which may create ragged tensors (tensors of different lengths). \n",
+    "* You will have to convert the ragged tensors to normal tensors using the `to_tensor()` method, which pads the tensors and sets the dimensions to `[None, tokenizer.model_max_length]` so you can feed different size tensors into your model based on the batch size.  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "FbpplBxNtanH"
+   },
+   "outputs": [],
+   "source": [
+    "import tensorflow as tf\n",
+    "\n",
+    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+    "\n",
+    "train_ds.set_format(type='tf', columns=columns_to_return)\n",
+    "\n",
+    "train_features = {x: train_ds[x] for x in ['input_ids', 'attention_mask']}\n",
+    "train_labels = {\"start_positions\": tf.reshape(train_ds['start_positions'], shape=[-1,1]),\n",
+    "                'end_positions': tf.reshape(train_ds['end_positions'], shape=[-1,1])}\n",
+    "\n",
+    "\n",
+    "train_tfdataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels)).batch(8)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "0_Jj8Av6rEuN"
+   },
+   "source": [
+    "#### Training \n",
+    "\n",
+    "It is finally time to start training your model! \n",
+    "\n",
+    "* Create a custom training function using [tf.GradientTape()](https://www.tensorflow.org/api_docs/python/tf/GradientTape)\n",
+    "* Target two loss functions, one for the start index and one for the end index. \n",
+    "* `tf.GradientTape()` records the operations performed during forward prop for automatic differentiation during backprop. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "PtZz249vQbLn",
+    "outputId": "24cdf861-af63-4581-a0ae-2de29d1880ed"
+   },
+   "outputs": [],
+   "source": [
+    "EPOCHS = 3\n",
+    "loss_fn1 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
+    "loss_fn2 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
+    "opt = tf.keras.optimizers.Adam(learning_rate=3e-5)\n",
+    "\n",
+    "losses = []\n",
+    "for epoch in range(EPOCHS):\n",
+    "    print(\"Starting epoch: %d\"% epoch )\n",
+    "    for step, (x_batch_train, y_batch_train) in enumerate(train_tfdataset):\n",
+    "        with tf.GradientTape() as tape:\n",
+    "            answer_start_scores, answer_end_scores = model(x_batch_train)\n",
+    "            loss_start = loss_fn1(y_batch_train['start_positions'], answer_start_scores)\n",
+    "            loss_end = loss_fn2(y_batch_train['end_positions'], answer_end_scores)\n",
+    "            loss = 0.5 * (loss_start + loss_end)\n",
+    "        losses.append(loss)\n",
+    "        grads = tape.gradient(loss, model.trainable_weights)\n",
+    "        opt.apply_gradients(zip(grads, model.trainable_weights))\n",
+    "\n",
+    "        if step % 20 == 0:\n",
+    "            print(\"Training loss (for one batch) at step %d: %.4f\"% (step, \n",
+    "                                                                   float(loss_start)))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "Q8ggB0JUWQuW"
+   },
+   "source": [
+    "Take a look at your losses and try playing around with some of the hyperparameters for better results!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 282
+    },
+    "id": "fK91EPvRYFcX",
+    "outputId": "6b7099dd-f918-4905-e3a3-fcce2880e506"
+   },
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "plt.plot(losses)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "64OtEmyUWUiM"
+   },
+   "source": [
+    "You have successfully trained your model to help automatically answer questions! Try asking it a question about a story."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "eFniMzpp1bpz",
+    "outputId": "0ce0e2a3-3d6a-4e6e-adff-d0c16b622c9a"
+   },
+   "outputs": [],
+   "source": [
+    "question, text = 'What is south of the bedroom?','The hallway is south of the garden. The garden is south of the bedroom.'\n",
+    "input_dict = tokenizer(text, question, return_tensors='tf')\n",
+    "outputs = model(input_dict)\n",
+    "start_logits = outputs[0]\n",
+    "end_logits = outputs[1]\n",
+    "\n",
+    "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
+    "answer = ' '.join(all_tokens[tf.math.argmax(start_logits, 1)[0] : tf.math.argmax(end_logits, 1)[0]+1])\n",
+    "print(question, answer.capitalize())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f07OtnCpuKFa"
+   },
+   "source": [
+    "Congratulations! You just implemented your first QA model in TensorFlow. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "9UaM5pY9u8EW"
+   },
+   "source": [
+    "<a name='2-1'></a>\n",
+    "## 2.2 PyTorch implementation\n",
+    "\n",
+    "[PyTorch](https://pytorch.org/) is an open source machine learning framework developed by Facebook's AI Research lab that can be used for computer vision and natural language processing. As you can imagine, it is quite compatible with the bAbI dataset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "nD9akXoXxMjd"
+   },
+   "source": [
+    "#### Train and test dataset\n",
+    "\n",
+    "Go ahead and try creating a train and test dataset by importing PyTorch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "JxMYWSG173ch"
+   },
+   "outputs": [],
+   "source": [
+    "from torch.utils.data import DataLoader\n",
+    "\n",
+    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+    "train_ds.set_format(type='pt', columns=columns_to_return)\n",
+    "test_ds.set_format(type='pt', columns=columns_to_return)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "OeuzZKlPHAAQ"
+   },
+   "source": [
+    "For the accuracy metrics for the PyTorch implementation, you will change things up a bit and use the [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) for start and end indicies over the entire test dataset as the loss functions. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "aD9tDpZfJsIB"
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.metrics import f1_score\n",
+    "\n",
+    "def compute_metrics(pred):\n",
+    "    start_labels = pred.label_ids[0]\n",
+    "    start_preds = pred.predictions[0].argmax(-1)\n",
+    "    end_labels = pred.label_ids[1]\n",
+    "    end_preds = pred.predictions[1].argmax(-1)\n",
+    "    \n",
+    "    f1_start = f1_score(start_labels, start_preds, average='macro')\n",
+    "    f1_end = f1_score(end_labels, end_preds, average='macro')\n",
+    "    \n",
+    "    return {\n",
+    "        'f1_start': f1_start,\n",
+    "        'f1_end': f1_end,\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "laX5cYQRHMXb"
+   },
+   "source": [
+    "#### Training\n",
+    "\n",
+    "Now it is time to load a pre-trained model. \n",
+    "\n",
+    "**Note:** You will be using the DistilBERT instead of TFDistilBERT for a PyTorch implementation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "del model # We delete the tensorflow model to avoid memory issues"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "YXFCsNcY79jx",
+    "outputId": "09af112f-e1e9-4a47-c988-37ee2a068df2"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import DistilBertForQuestionAnswering\n",
+    "\n",
+    "pytorch_model = DistilBertForQuestionAnswering.from_pretrained(\"model/pytorch\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "xCUdMmCxHP6_"
+   },
+   "source": [
+    "Instead of a custom training loop, you will use the [🤗 Trainer](https://huggingface.co/transformers/main_classes/trainer.html), which contains a basic training loop and is fairly easy to implement in PyTorch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 329
+    },
+    "id": "1htmS3TV-2Bk",
+    "outputId": "cc21bfbb-da09-47f9-ee16-7db0096d35e7"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import Trainer, TrainingArguments\n",
+    "\n",
+    "training_args = TrainingArguments(\n",
+    "    output_dir='results',          # output directory\n",
+    "    overwrite_output_dir=True,\n",
+    "    num_train_epochs=3,              # total number of training epochs\n",
+    "    per_device_train_batch_size=8,  # batch size per device during training\n",
+    "    per_device_eval_batch_size=8,   # batch size for evaluation\n",
+    "    warmup_steps=20,                # number of warmup steps for learning rate scheduler\n",
+    "    weight_decay=0.01,               # strength of weight decay\n",
+    "    logging_dir=None,            # directory for storing logs\n",
+    "    logging_steps=50\n",
+    ")\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=pytorch_model,                 # the instantiated 🤗 Transformers model to be trained\n",
+    "    args=training_args,                  # training arguments, defined above\n",
+    "    train_dataset=train_ds,         # training dataset\n",
+    "    eval_dataset=test_ds,\n",
+    "    compute_metrics=compute_metrics             # evaluation dataset\n",
+    ")\n",
+    "\n",
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 207
+    },
+    "id": "lDzbm7vzAiPJ",
+    "outputId": "7cd62f51-a04b-4583-bc0e-e459813d3103"
+   },
+   "outputs": [],
+   "source": [
+    "trainer.evaluate(test_ds)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "QAgrcs2pHvVu"
+   },
+   "source": [
+    "Now it is time to ask your PyTorch model a question! \n",
+    "* Before testing your model with a question, you can tell PyTorch to send your model and inputs to the GPU if your machine has one, or the CPU if it does not. \n",
+    "* You can then proceed to tokenize your input and create PyTorch tensors and send them to your device. \n",
+    "* The rest of the pipeline is relatively similar to the one you implemented for TensorFlow.   \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "yfBe9AFABqUr",
+    "outputId": "b5ca6039-8ce2-4e75-9161-1c96a0f39425"
+   },
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
+    "\n",
+    "pytorch_model.to(device)\n",
+    "\n",
+    "question, text = 'What is east of the hallway?','The kitchen is east of the hallway. The garden is south of the bedroom.'\n",
+    "\n",
+    "input_dict = tokenizer(text, question, return_tensors='pt')\n",
+    "\n",
+    "input_ids = input_dict['input_ids'].to(device)\n",
+    "attention_mask = input_dict['attention_mask'].to(device)\n",
+    "\n",
+    "outputs = pytorch_model(input_ids, attention_mask=attention_mask)\n",
+    "\n",
+    "start_logits = outputs[0]\n",
+    "end_logits = outputs[1]\n",
+    "\n",
+    "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
+    "answer = ' '.join(all_tokens[torch.argmax(start_logits, 1)[0] : torch.argmax(end_logits, 1)[0]+1])\n",
+    "\n",
+    "print(question, answer.capitalize())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "eGzuHkMZ4q9I"
+   },
+   "source": [
+    "### Congratulations!\n",
+    " \n",
+    "You've completed this notebook, and can now implement Transformer models for QA tasks!\n",
+    "\n",
+    "You are now able to:\n",
+    "* Perform extractive Question Answering \n",
+    "* Fine-tune a pre-trained transformer model to a custom dataset\n",
+    "* Implement a QA model in TensorFlow and PyTorch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "G8tAV-584vKE"
+   },
+   "source": [
+    "<font color='blue'><b>What you should remember</b>:\n",
+    "- Transformer models are often trained by tokenizers that split words into subwords.\n",
+    "  - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
+    "- PyTorch is a relatively light and easy to implement framework that can make rapid prototyping easier, while TensorFlow has advantages in scaling and is more widely used in production\n",
+    "  - `tf.GradientTape` allows you to build custom training loops in TensorFlow\n",
+    "  - The `Trainer` API in PyTorch gives you a basic training loop that is compatible with 🤗 models and datasets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%javascript\n",
+    "let element = document.getElementById('submit-notebook-button-group');\n",
+    "if (!element) {\n",
+    "    window._save_and_close = function(){\n",
+    "        IPython.notebook.save_checkpoint();\n",
+    "        IPython.notebook.session.delete();\n",
+    "        window.onbeforeunload = null\n",
+    "        setTimeout(function() {window.close();}, 1000)\n",
+    "    }\n",
+    "    let header = document.getElementById('maintoolbar-container');\n",
+    "    element = document.createElement(\"div\");\n",
+    "    element.setAttribute(\"class\", \"btn-group\");\n",
+    "    element.setAttribute(\"id\", \"submit-notebook-button-group\");\n",
+    "    element.setAttribute(\"align\", \"right\");\n",
+    "    element.setAttribute(\"style\", \"float:right\")\n",
+    "    element.innerHTML = '<button class=\"btn btn-default\" title=\"Save and close this notebook.\" style=\"background-color:rgb(42, 115, 204); color:white; padding:4px 8px\" onclick=window._save_and_close()>Save and close</button>'\n",
+    "    header.appendChild(element); \n",
+    "}                    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "collapsed_sections": [],
+   "name": "QA-dataset.ipynb",
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "013f041c3e0b4e35bf2432fc345cb7bf": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_1e6c02317171453cbd3d4d665879b0d4",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_f0e34f2bf626434fa73f0def26b3d1a5",
+      "value": 1000
+     }
+    },
+    "07aaa9b79a744856b19d723370d6e588": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_b39b85d8cb05418aa92e8476ad02f755",
+       "IPY_MODEL_0a8534ac52af4d48ad82b66463ad08c3"
+      ],
+      "layout": "IPY_MODEL_afedd2328cf141f78775e4cfa7758267"
+     }
+    },
+    "0a8534ac52af4d48ad82b66463ad08c3": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_8cfbd3b14b23417993270f851a2d8ff9",
+      "placeholder": "​",
+      "style": "IPY_MODEL_31fc08a1e7e04f6b9b3ea400ccfaea75",
+      "value": " 1000/1000 [01:40&lt;00:00,  9.90ex/s]"
+     }
+    },
+    "1e6c02317171453cbd3d4d665879b0d4": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "1f2773e3e80c4dd8b6b26e171bf33bc7": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "31fc08a1e7e04f6b9b3ea400ccfaea75": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "32a5c82c7a9845c09c11bb4e30c2f1aa": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "364ba960eb474c9084cc71851594d345": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "39029f730ae140c7902fca6dac5361ad": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "393697738e724e9fad4d163de0a77840": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "3abb36da57c841838867c56e2a3a325b": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "3dab28395f3f475d8242e4d4d45ed059": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_63b4ebafcead4c0784b5511219a6a198",
+      "placeholder": "​",
+      "style": "IPY_MODEL_58718e12f1b7459989ab5296846c4be6",
+      "value": " 1000/1000 [00:10&lt;00:00, 97.35ex/s]"
+     }
+    },
+    "44b7bea3e09d4e5684921c66dd4c7514": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_7e1325e57bf9417e93d7ef180794ab3c",
+       "IPY_MODEL_3dab28395f3f475d8242e4d4d45ed059"
+      ],
+      "layout": "IPY_MODEL_6af3ec5091d74bd1a95bf02a87dd240b"
+     }
+    },
+    "4d9152a30e824931983a425ee6d607a6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_013f041c3e0b4e35bf2432fc345cb7bf",
+       "IPY_MODEL_ef4e12f29f1e458f811a400faf21bdcc"
+      ],
+      "layout": "IPY_MODEL_1f2773e3e80c4dd8b6b26e171bf33bc7"
+     }
+    },
+    "4f5b06c3a5e44c6cade5bf83634d9f69": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "568f11b4462f4b4e95f3ad5947bb275e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "58718e12f1b7459989ab5296846c4be6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "5b6dbe662ca24834b7678638e101e1ff": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "63b4ebafcead4c0784b5511219a6a198": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "6af3ec5091d74bd1a95bf02a87dd240b": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "723acefae33d448199fa5c1a9ec3f246": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_77273c2e4b4e4e4c8ee4b6b344749518",
+       "IPY_MODEL_f0ac3b9b8f664479940c6ee18fc2f13e"
+      ],
+      "layout": "IPY_MODEL_32a5c82c7a9845c09c11bb4e30c2f1aa"
+     }
+    },
+    "77273c2e4b4e4e4c8ee4b6b344749518": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_e592db98c0c34c5e800f5d7b6d3c099e",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_393697738e724e9fad4d163de0a77840",
+      "value": 1000
+     }
+    },
+    "7e1325e57bf9417e93d7ef180794ab3c": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_7fb1118c0b4443b6b6dbb5803e9ec2e8",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_ca722dcd857c433c9058585e31a1673d",
+      "value": 1000
+     }
+    },
+    "7fb1118c0b4443b6b6dbb5803e9ec2e8": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "7fefe9e1121a43558d773500aef8935c": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "863c5ce96db84e3da162072c9a13c913": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "8968319cdaca476fb15c11a388dce39a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_a725734893004a45b61194f649f5f602",
+       "IPY_MODEL_c4a24656d67844e995d3b8e175c6c497"
+      ],
+      "layout": "IPY_MODEL_863c5ce96db84e3da162072c9a13c913"
+     }
+    },
+    "89fdda6e6688476495ca297bfe010bf8": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "8b961844b5004905922531bd805a9d57": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "8cfbd3b14b23417993270f851a2d8ff9": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "929946fdfaa04cf59d3b31cf92fc08d1": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_cda72c45821a4eb89f1a3ab5510b26d3",
+      "placeholder": "​",
+      "style": "IPY_MODEL_89fdda6e6688476495ca297bfe010bf8",
+      "value": " 1000/1000 [00:08&lt;00:00, 123.32ex/s]"
+     }
+    },
+    "a725734893004a45b61194f649f5f602": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_afc33fa78b5d440192c435bfca6f7914",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_4f5b06c3a5e44c6cade5bf83634d9f69",
+      "value": 1000
+     }
+    },
+    "aa5c0d374889482697fc0f7ce9c81afe": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "afc33fa78b5d440192c435bfca6f7914": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "afedd2328cf141f78775e4cfa7758267": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "b39b85d8cb05418aa92e8476ad02f755": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_8b961844b5004905922531bd805a9d57",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_3abb36da57c841838867c56e2a3a325b",
+      "value": 1000
+     }
+    },
+    "b4c6a18610734036a16a14a43174c52e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "c42644a4e6184a1cbdb2b453b5dbb7d6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_e8f1abd85f3e49f991d4c1312ffd416b",
+       "IPY_MODEL_929946fdfaa04cf59d3b31cf92fc08d1"
+      ],
+      "layout": "IPY_MODEL_364ba960eb474c9084cc71851594d345"
+     }
+    },
+    "c4a24656d67844e995d3b8e175c6c497": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_b4c6a18610734036a16a14a43174c52e",
+      "placeholder": "​",
+      "style": "IPY_MODEL_f37bd346f8614fec92d6c5b5e9b66d2f",
+      "value": " 1000/1000 [01:41&lt;00:00,  9.86ex/s]"
+     }
+    },
+    "ca722dcd857c433c9058585e31a1673d": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "cda72c45821a4eb89f1a3ab5510b26d3": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "e592db98c0c34c5e800f5d7b6d3c099e": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "e8f1abd85f3e49f991d4c1312ffd416b": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "100%",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_ff444b253e9a40e5bec755926d83740f",
+      "max": 1000,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_aa5c0d374889482697fc0f7ce9c81afe",
+      "value": 1000
+     }
+    },
+    "ef4e12f29f1e458f811a400faf21bdcc": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_39029f730ae140c7902fca6dac5361ad",
+      "placeholder": "​",
+      "style": "IPY_MODEL_5b6dbe662ca24834b7678638e101e1ff",
+      "value": " 1000/1000 [01:25&lt;00:00, 11.68ex/s]"
+     }
+    },
+    "f0ac3b9b8f664479940c6ee18fc2f13e": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_7fefe9e1121a43558d773500aef8935c",
+      "placeholder": "​",
+      "style": "IPY_MODEL_568f11b4462f4b4e95f3ad5947bb275e",
+      "value": " 1000/1000 [01:24&lt;00:00, 11.77ex/s]"
+     }
+    },
+    "f0e34f2bf626434fa73f0def26b3d1a5": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": "initial"
+     }
+    },
+    "f37bd346f8614fec92d6c5b5e9b66d2f": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "ff444b253e9a40e5bec755926d83740f": {
+     "model_module": "@jupyter-widgets/base",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

Transformer Mechanism/QA/tf/W4A3_UGL/data/dataset_dict.json

@@ -0,0 +1 @@
+{"splits": ["train", "test"]}

Transformer Mechanism/QA/tf/W4A3_UGL/data/test/._dataset.arrow

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a591d9521aff12eea1e7ee705de14a1c50ae25b9c5de477d9bcdd56c5986e83e
+size 212

Transformer Mechanism/QA/tf/W4A3_UGL/data/test/cache-26c237c56fc0b951.arrow

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36347fc2d623e02c4b5b1a365abadea94bb73d145a3ea91a3d0f02da01385d9e
+size 326328

Transformer Mechanism/QA/tf/W4A3_UGL/data/test/cache-6b23a7f03ef9fdb4.arrow

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e559136154bac6bc023887fc2b04d5a6ac67121e31f2c1969bfa88b19d7d895
+size 342632

Transformer Mechanism/QA/tf/W4A3_UGL/data/test/cache-c9959a793a67abd8.arrow

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5e1b69377781f5b617299b386b6b1185d60b4ae9c443dc12c4433dd7a98b8e2
+size 497544