Upload 2 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+text-classifier-model-sentiment-movie-reviews[[:space:]](1) filter=lfs diff=lfs merge=lfs -text

Lesson 4(NLP notes) (1).ipynb

@@ -0,0 +1,2614 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "5c9cf9c8-4458-42ec-b736-deb68e85be0b",
+   "metadata": {},
+   "source": [
+    "# NLP Deep Dive: RNNs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "01cfeab4-b755-4fac-9ddc-f99ce1f30234",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#hide\n",
+    "! [ -e /content ] && pip install -Uqq fastbook\n",
+    "import fastbook\n",
+    "fastbook.setup_book()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "600871ad-cb6c-47a1-afd1-56a39419599a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#hide\n",
+    "from fastbook import *\n",
+    "from IPython.display import display,HTML"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "424d5492-60a7-45c6-8796-e23e1315bef6",
+   "metadata": {},
+   "source": [
+    "Generally in NLP, the pretrained model is trained on different task.\n",
+    "\n",
+    "*Language model* is one that has been trained to predict the next word in a text having read the ones before and this kind of task is called *self-supervised learning*.\n",
+    "\n",
+    "Self-supervised learning is not used for model directly trained but is used for pretrained model used for transfer learning.\n",
+    "\n",
+    ">Self-supervised learning: Training a model using labels that are embedded in the independent variable instead of requiring external labels.\n",
+    "\n",
+    "**Why are we learning to train a language model in detail?**\n",
+    "- it will be helpful to understand the foundations of model you're using.\n",
+    "- practical reason is, we can fine-tune the model(i.e. language model) to get better results prior to fine-tuning classification model.\n",
+    "\n",
+    "The IMDb reviews data consists of reviews. We can use all of these reviews to fine-tune the language model that was trained on Wikipedia articles prior to transfer learning to a classification task to get better at predicting the next word of a movie review(ultimately getting better results). This is known as the *Universal Language Model Fine-tuning(ULMFit)* approach.\n",
+    "\n",
+    "3 stages for TL in NLP:\n",
+    "1. Language model (trained on Wikipedia articles)\n",
+    "2. Language model (fine-tuned previous model using IMDb reviews data)\n",
+    "3. Then train classification model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c03e511e-daf9-4ced-b28d-878a78c46470",
+   "metadata": {},
+   "source": [
+    "## Text Preprocessing\n",
+    "\n",
+    "Approach for a single categorical variable to be used for a NN as independent variable:(not understanding the steps 3&4)\n",
+    "1. Make a list of all possible levels of that categorical variable(this is vocab)\n",
+    "2. Replace each level with its index in the vocab.\n",
+    "3. Create an embedding matrix for this containing a row for each level(??)\n",
+    "4. Use the embedding matrix as the first layer of NN.\n",
+    "\n",
+    "The same thing can be done for text but\n",
+    "- first we concatenate the documents in our dataset to form a single long list\n",
+    "- then split it into words which gives a long list of words(or tokens)\n",
+    "Our independent variable is sequence of words starting from first word in long list and ending with second to last(does it mean start from first word and ends at the last word).\n",
+    "\n",
+    "Our *vocab* contains mix of common words already present in the vocab of pretrained model and new words specific to the movie reviews corpus.\n",
+    "\n",
+    "Our *embedding matrix* will be built accordingly: \n",
+    "1) For words already present in the pretrained vocabulary: When a word from our current task is also present in the vocabulary of pretrained model, we simply copy the corresponding row from the preatrained model's embedding matrix.\n",
+    "2) For new words: When we encounter a new word that wasn't in the pretrained vocabulary and therfore there will be no embedding for that word. So we initialise a new row in our embedding matrix with random vector."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "24b796e6-59df-4b9a-924e-6d9160a77359",
+   "metadata": {},
+   "source": [
+    "The steps necessary to create a language model has following jargon:\n",
+    "\n",
+    "- Tokenization: Convert text into list of words\n",
+    "- Numericalization: Make up a list of all unique words (i.e. vocab), convert them into numbers by looking up its index from vocab.\n",
+    "- Language model data loader creation: fastai provides an `LMDataLoader` class which automatically handles creating a dependent variable which is offset from independent variable by one token.(meaning the dependent and independent variables differ by one token since our task is to predict the next word in text). It also handles some important functions such as shuffling data in order to maintain proper structure of dependent and independent variables.\n",
+    "- Language model creation: we need a special kind of model that takes input lists(big or small). There are number of ways to do this but here we'll use RNNs(recurrent neural nets)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "559b77a4-3ae1-4830-bbb4-2bd353694ef7",
+   "metadata": {},
+   "source": [
+    "### Tokenization\n",
+    "\n",
+    "When we said \"convert the text into a list of words,\" we left out a lot of details. For instance, what do we do with punctuation? How do we deal with a word like \"don't\"? Is it one word, or two? What about long medical or chemical words? Should they be split into their separate pieces of meaning? How about hyphenated words? What about languages like German and Polish where we can create really long words from many, many pieces? What about languages like Japanese and Chinese that don't use bases at all, and don't really have a well-defined idea of word?\n",
+    "\n",
+    "Because there is no one correct answer to these questions, there is no one approach to tokenization.\n",
+    "\n",
+    "There are three main approaches-\n",
+    "1. word-based tokenization: split sentence on spaces and taking care of language specific rules to separate parts of meaning even when there is no space.\n",
+    "2. sub-word based tokenization: split words into smaller parts based on commonly occuring substrings.\n",
+    "3. character based tokenization: split sentece into individual characters.\n",
+    "\n",
+    ">token: One element of list created using tokenization process."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4385df82-61df-4e3d-be12-04ad48e1f38c",
+   "metadata": {},
+   "source": [
+    "### Word tokenization with fastai\n",
+    "\n",
+    "This relies on an assumption that spaces provide useful separation of components of meaning in sentence.\n",
+    "fastai provides a consistent interface to range of tokenizers instead of providing its own tokenizers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "6a0112bb-ced2-49c9-8e4a-ba0e820297b7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from fastai.text.all import *\n",
+    "path=untar_data(URLs.IMDB)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "32dd4c8b-9fa9-4b80-a12f-6542d790bc9d",
+   "metadata": {},
+   "source": [
+    "To try tokenizers, we need to get the text files. Similar to `get_image_files` fot CV tasks, we have `get_text_files` for NLP tasks to get all text files in a path."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "373acfea-ba21-414c-a4c0-a92fed27c3c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "files=get_text_files(path,folders=['train','test','unsup'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "84104613-1ab6-4a04-810f-06f5a91cf737",
+   "metadata": {},
+   "source": [
+    "The `folders` parameter is used to restrict the search to particular list of folders."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "83677b1e-d9a1-4811-ab66-83528d5cb68a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Path('/teamspace/studios/this_studio/.fastai/data/imdb/test/neg/0_2.txt')"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# grab the first file and view some part of it\n",
+    "files[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "976a1c13-1baa-452d-bec9-6b39ed51ad98",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<_io.TextIOWrapper name='/teamspace/studios/this_studio/.fastai/data/imdb/test/neg/0_2.txt' mode='r' encoding='UTF-8'>"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "files[0].open()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "97766ce1-68d9-4d59-862e-b14b2f629071",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "txt=files[0].open().read()[:100]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "abdc41d5-87c4-49c3-b2fa-2c5e05d56fac",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Once again Mr. Costner has dragged out a movie for far longer than necessary. Aside from the terrifi'"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87f46d64-f56c-497d-9201-9da74ed6ee51",
+   "metadata": {},
+   "source": [
+    "spaCy is the deafult English word tokenizer when the book was written and instead of using `SpacyTokenizer`, we'll use `WordTokenizer` by fastai which refers to the default tokenizer used now.\n",
+    "\n",
+    "fastai's `coll_repr()` is used to display results. Displays first n items of collection along with full size and it's what `L`(which is a list-like collection with added-functionality) uses by default for string representation.\n",
+    "\n",
+    "**Note**: fastai's tokenizer takes a list of text so we need to wrap txt in list."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "4d83ce26-8a1d-4e51-a540-31b35b7a0538",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(#19) ['Once','again','Mr.','Costner','has','dragged','out','a','movie','for','far','longer','than','necessary','.','Aside','from','the','terrifi']\n"
+     ]
+    }
+   ],
+   "source": [
+    "spacy=WordTokenizer()\n",
+    "toks=first(spacy([txt]))\n",
+    "print(coll_repr(toks,30))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9f45e3d9-3672-4b38-8cb3-e4e4f0e74a98",
+   "metadata": {},
+   "source": [
+    "spaCy handles the all little details in text."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6b71cf9b-b203-4bab-bdb6-4d4ed6689b65",
+   "metadata": {},
+   "source": [
+    "fastai adds some additional functionality to tokenization process with the `Tokenizer` class."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "276ffa86-741b-4c52-9ba7-37bcb87f2261",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(#25) ['xxbos','xxmaj','once','again','xxmaj','mr','.','xxmaj','costner','has','dragged','out','a','movie','for','far','longer','than','necessary','.','xxmaj','aside','from','the','terrifi']\n"
+     ]
+    }
+   ],
+   "source": [
+    "tkn=Tokenizer(spacy)\n",
+    "print(coll_repr(tkn(txt),30))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "19d9b3a3-91cd-4e18-b5bb-e0d3043b9258",
+   "metadata": {},
+   "source": [
+    "The tokens which start with characters 'xx' are called special tokens. By recognising the start token(xxbos), the model will be able to learn it needs to forget what was said previously and focus on upcoming words.\n",
+    "\n",
+    "These are not by deafult from spaCy, fastai adds them when processing text and desgined for model to understand important parts of sentence.\n",
+    "\n",
+    "Some main special tokens:\n",
+    "\n",
+    "- `xxbos`: indicates beginning of text\n",
+    "- `xxmaj`: indicates the next word begins with a capital letter(since everything is lowercased)\n",
+    "- `xxunk`: indicates word is unknown\n",
+    "\n",
+    "To see the rules that were used, we can check thedefault rules usinf the line below:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "54a10034-1447-4f0d-bb84-4ec686431884",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[<function fastai.text.core.fix_html(x)>,\n",
+       " <function fastai.text.core.replace_rep(t)>,\n",
+       " <function fastai.text.core.replace_wrep(t)>,\n",
+       " <function fastai.text.core.spec_add_spaces(t)>,\n",
+       " <function fastai.text.core.rm_useless_spaces(t)>,\n",
+       " <function fastai.text.core.replace_all_caps(t)>,\n",
+       " <function fastai.text.core.replace_maj(t)>,\n",
+       " <function fastai.text.core.lowercase(t, add_bos=True, add_eos=False)>]"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "defaults.text_proc_rules"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a286be44-bae6-4ee8-b5eb-abece30fded9",
+   "metadata": {},
+   "source": [
+    "Summary of what each does:\n",
+    "- `fix_html`: replaces special HTML characs with readable version\n",
+    "- `replace_rep`: replaces any character repeated more than three times with a special token(`xxrep`) followed by number of times and then the character\n",
+    "- `replace_wrep`: replaces any word repeated more than three times with a special token(`xxwrep`) followed by number of times and then the word.\n",
+    "- `spec_add_spaces`: adds spaces arong / and #\n",
+    "- `rm_useless_spaces`: removes all repition of space character\n",
+    "- `replace_maj`: lowercases a capitalised word and adds a specila token for capitalises(`xxmaj`) in front.\n",
+    "- `replace_all_caps`: lowercases a word written in all caps and adds a specialised token `xxup`\n",
+    "- `lowercase`: lowercases all text and adds special token at beginning(`xxbos`) and/or end(`xxeos`)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53bfd6ff-fb49-4df5-9ba5-cbfbed93e9c0",
+   "metadata": {},
+   "source": [
+    "### Subword tokenization \n",
+    "\n",
+    "This tokenization is best for languages such as chinese, hungarian etc where the words are not separated by space.\n",
+    "\n",
+    "To handle these, we use this tokenization which has the 2 steps:\n",
+    "1. Analyse corpus of documents to find the most *commonly occuring groups of letters* and this becomes the vocab.\n",
+    "2. Tokenise the corpus using this vocab of subword units."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "7f3f5ad6-2bca-489e-973d-cf33b1a9f80b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "txts=L(o.open().read() for o in files[:2000])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6e22e42f-b8b3-4e75-893a-16b5dc96add3",
+   "metadata": {},
+   "source": [
+    "We instantiate tokenizer and pass size of vocab and then train it, i.e. we want it to read our documnets and find out the common sequence of characters to create the vocab. This is done using `setup` which is automatically called in usual data processing pipelines."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "cbe29142-ed85-4984-9f4e-ca373de050cd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def subword(sz):\n",
+    "    sp=SubwordTokenizer(vocab_sz=sz)\n",
+    "    sp.setup(txts)\n",
+    "    return ' '.join(first(sp([txt])))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "15baf9db-8f29-477e-af19-fe25cf63f128",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "sentencepiece_trainer.cc(178) LOG(INFO) Running command: --input=tmp/texts.out --vocab_size=1000 --model_prefix=tmp/spm --character_coverage=0.99999 --model_type=unigram --unk_id=9 --pad_id=-1 --bos_id=-1 --eos_id=-1 --minloglevel=2 --user_defined_symbols=▁xxunk,▁xxpad,▁xxbos,▁xxeos,▁xxfld,▁xxrep,▁xxwrep,▁xxup,▁xxmaj --hard_vocab_limit=false\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'▁O n ce ▁again ▁M r . ▁Co st n er ▁has ▁ d ra g g ed ▁out ▁a ▁movie ▁for ▁far ▁long er ▁than ▁ ne ce s s ar y . ▁A side ▁from ▁the ▁ ter ri f i'"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "subword(1000)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "a1b95d33-836b-43d3-bcec-98637b586618",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'▁ O n ce ▁a g a in ▁ M r . ▁ C o s t n er ▁ h a s ▁ d ra g g ed ▁ o u t ▁a ▁movie ▁for ▁f ar ▁ l on g er ▁ th an ▁ n e ce s s ar y . ▁A s i d e ▁f ro m ▁the ▁ ter ri f i'"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "subword(200)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "c6cfbd46-d1b2-410d-95e3-aa526138e63b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'▁On ce ▁again ▁Mr . ▁Costner ▁has ▁dragged ▁out ▁a ▁movie ▁for ▁far ▁longer ▁than ▁necessary . ▁A side ▁from ▁the ▁ ter ri fi'"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "subword(10000)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e89434f5-0955-46c5-b02b-37f2ab9fe113",
+   "metadata": {},
+   "source": [
+    "When we use fastai's subword tokenizer, `_` character represents a space in actual text.\n",
+    "\n",
+    "- If we use a *smaller vocab*, then each token will represent *fewer characters* and it will take *more tokens to represent the sentence*.\n",
+    "- If we use a *larger vocab*, the most common English words will end up in the vocab themselves and we *will not need as many to represent a sentence*.\n",
+    "\n",
+    "Picking a subword vocab size represents a compromise:\n",
+    "\n",
+    "- a *larger vocab* means *fewer tokens per sentence*, which means faster training, lesser memory and less state for model to remember but it means *larger embedding matrix* and require more data to learn.\n",
+    "\n",
+    "Overall, subword tokenization provides a way to scale between character tokenizatio(i.e using small subword vocab) and word tokenizatio(i.e using large subword vocab) and handles every language without needing language specific models to be developed.\n",
+    "\n",
+    "Once our text is split into tokens, we need to convert them into numbers."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a7975c2-522a-4457-9581-65243487f469",
+   "metadata": {},
+   "source": [
+    "### Numerilization with fastai\n",
+    "\n",
+    "*Numericalization* is process of mapping tokens to integers. Steps are similar to those needed to create `Category` variable.\n",
+    "\n",
+    "1. Make a list of all possible levels of categorical variable(this is vocab)\n",
+    "2. Replace each level with its index value in vocab."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "98e65ad1-4b13-4929-b792-55ce1205b505",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(#25) ['xxbos','xxmaj','once','again','xxmaj','mr','.','xxmaj','costner','has'...] 31\n"
+     ]
+    }
+   ],
+   "source": [
+    "toks=tkn(txt)\n",
+    "print(coll_repr(tkn(txt)),31)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43f50521-23d2-4789-95bd-7852b1263a36",
+   "metadata": {},
+   "source": [
+    "Just like `SubwordTokenizer`, we need to call `setup` on `Numericalize`, in order to create vocab.\n",
+    "So first we need our tokenized corpus."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "d1d5f6d3-5c51-4e6d-b722-186adecb8e9b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(#207) ['xxbos','xxmaj','once','again','xxmaj','mr','.','xxmaj','costner','has'...]"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "toks200=txts[:200].map(tkn)\n",
+    "toks200[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "0c4a320b-ff04-4ce5-abb9-98c2f941b94f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"(#1968) ['xxunk','xxpad','xxbos','xxeos','xxfld','xxrep','xxwrep','xxup','xxmaj','the','.',',','a','and','of','to','is','it','i','in'...]\""
+      ]
+     },
+     "execution_count": 19,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# we then pass this to setup\n",
+    "num=Numericalize()\n",
+    "num.setup(toks200)\n",
+    "coll_repr(num.vocab,20)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97cdd2b1-cc1d-45bd-a3cb-6ef7c73b52ee",
+   "metadata": {},
+   "source": [
+    "Our special rules tokens appear first followed by every word appearing once in frequency order. The defaults of `Numericalize` are `min_freq=3, max_vocab=60000`. `max_vocab=60000` results in fastai replacing the words other than most common 60000 words with a special token `xxunk` means unknown word. This is useful to avoid haing large embedding matrix which can slow down traing. `min_freq=3` means any word woith freq<3 is replaced by `xxunk`.\n",
+    "\n",
+    "fastai can also numericalize our dataset using vocab that you provide, by passing a list of words as `vocab` parameter.\n",
+    "\n",
+    "Once we have created `Numericalize` object, we can use it as a function:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "fbafd886-1146-403b-8d08-2b655bdfad40",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "TensorText([   2,    8,  349,  183,    8, 1177,   10,    8, 1178,   60, 1455,   62,   12,   25,   28,  189,  957,   93,  958,   10])"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "nums=num(toks)[:20]\n",
+    "nums"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9c74d413-fb75-4e8a-a31e-b5539e429c59",
+   "metadata": {},
+   "source": [
+    "This time our tokens have been converted to a tensor of integers that our model can recieve. We can that they map back to original text as follows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "d5adb361-9088-4a03-8749-7b1067e8b953",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'xxbos xxmaj once again xxmaj mr . xxmaj costner has dragged out a movie for far longer than necessary .'"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "' '.join(num.vocab[o] for o in nums)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ec674dd1-1e69-4476-872d-d3c2d1a7e604",
+   "metadata": {},
+   "source": [
+    "Now that we have numbers, we need to put them into batches for our model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a64b6774-1b4e-49cf-987b-9bba45e6af68",
+   "metadata": {},
+   "source": [
+    "### Putting our texts into batches for a language model\n",
+    "\n",
+    "Our language model should read text in order such that each new batch should begin precisely where the previous one left off and this unlike images where we had to resize the images to get same height and width before grouping them into mini-batch so they can stack together efficiently in a single tensor.\n",
+    "\n",
+    "Suppose we have the following text:\n",
+    "\n",
+    "> : In this chapter, we will go back over the example of classifying movie reviews we studied in chapter 1 and dig deeper under the surface. First we will look at the processing steps necessary to convert text into numbers and how to customize it. By doing this, we'll have another example of the PreProcessor used in the data block API.\\nThen we will study how we build a language model and train it for a while.\n",
+    "\n",
+    "The tokenization process will add special tokens and deal with punctuation to return this text:\n",
+    "\n",
+    "> : xxbos xxmaj in this chapter , we will go back over the example of classifying movie reviews we studied in chapter 1 and dig deeper under the surface . xxmaj first we will look at the processing steps necessary to convert text into numbers and how to customize it . xxmaj by doing this , we 'll have another example of the preprocessor used in the data block xxup api . \\n xxmaj then we will study how we build a language model and train it for a while .\n",
+    "\n",
+    "We now have 90 tokens, separated by spaces. Let's say we want a batch size of 6. We need to break this text into 6 contiguous parts of length 15:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "aa7c6cc3-ec70-43c9-9b6e-57f67c2a7941",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>xxbos</td>\n",
+       "      <td>xxmaj</td>\n",
+       "      <td>in</td>\n",
+       "      <td>this</td>\n",
+       "      <td>chapter</td>\n",
+       "      <td>,</td>\n",
+       "      <td>we</td>\n",
+       "      <td>will</td>\n",
+       "      <td>go</td>\n",
+       "      <td>back</td>\n",
+       "      <td>over</td>\n",
+       "      <td>the</td>\n",
+       "      <td>example</td>\n",
+       "      <td>of</td>\n",
+       "      <td>classifying</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>movie</td>\n",
+       "      <td>reviews</td>\n",
+       "      <td>we</td>\n",
+       "      <td>studied</td>\n",
+       "      <td>in</td>\n",
+       "      <td>chapter</td>\n",
+       "      <td>1</td>\n",
+       "      <td>and</td>\n",
+       "      <td>dig</td>\n",
+       "      <td>deeper</td>\n",
+       "      <td>under</td>\n",
+       "      <td>the</td>\n",
+       "      <td>surface</td>\n",
+       "      <td>.</td>\n",
+       "      <td>xxmaj</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>first</td>\n",
+       "      <td>we</td>\n",
+       "      <td>will</td>\n",
+       "      <td>look</td>\n",
+       "      <td>at</td>\n",
+       "      <td>the</td>\n",
+       "      <td>processing</td>\n",
+       "      <td>steps</td>\n",
+       "      <td>necessary</td>\n",
+       "      <td>to</td>\n",
+       "      <td>convert</td>\n",
+       "      <td>text</td>\n",
+       "      <td>into</td>\n",
+       "      <td>numbers</td>\n",
+       "      <td>and</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>how</td>\n",
+       "      <td>to</td>\n",
+       "      <td>customize</td>\n",
+       "      <td>it</td>\n",
+       "      <td>.</td>\n",
+       "      <td>xxmaj</td>\n",
+       "      <td>by</td>\n",
+       "      <td>doing</td>\n",
+       "      <td>this</td>\n",
+       "      <td>,</td>\n",
+       "      <td>we</td>\n",
+       "      <td>'ll</td>\n",
+       "      <td>have</td>\n",
+       "      <td>another</td>\n",
+       "      <td>example</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>of</td>\n",
+       "      <td>the</td>\n",
+       "      <td>preprocessor</td>\n",
+       "      <td>used</td>\n",
+       "      <td>in</td>\n",
+       "      <td>the</td>\n",
+       "      <td>data</td>\n",
+       "      <td>block</td>\n",
+       "      <td>xxup</td>\n",
+       "      <td>api</td>\n",
+       "      <td>.</td>\n",
+       "      <td>\\n</td>\n",
+       "      <td>xxmaj</td>\n",
+       "      <td>then</td>\n",
+       "      <td>we</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>will</td>\n",
+       "      <td>study</td>\n",
+       "      <td>how</td>\n",
+       "      <td>we</td>\n",
+       "      <td>build</td>\n",
+       "      <td>a</td>\n",
+       "      <td>language</td>\n",
+       "      <td>model</td>\n",
+       "      <td>and</td>\n",
+       "      <td>train</td>\n",
+       "      <td>it</td>\n",
+       "      <td>for</td>\n",
+       "      <td>a</td>\n",
+       "      <td>while</td>\n",
+       "      <td>.</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "stream = \"In this chapter, we will go back over the example of classifying movie reviews we studied in chapter 1 and dig deeper under the surface. First we will look at the processing steps necessary to convert text into numbers and how to customize it. By doing this, we'll have another example of the PreProcessor used in the data block API.\\nThen we will study how we build a language model and train it for a while.\"\n",
+    "tokens=tkn(stream)\n",
+    "bs,seq_len=6,15\n",
+    "d_tokens=np.array([tokens[i*seq_len:(i+1)*seq_len] for i in range(bs)])\n",
+    "df=pd.DataFrame(d_tokens)\n",
+    "display(HTML(df.to_html(index=False,header=None)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0870a3e0-1d96-467b-bfd9-64087e9c5ff3",
+   "metadata": {},
+   "source": [
+    "We need to divide this array more finely into subarrays of a fixed sequence length. It is impostant to maintain order within and across these subarrays, because we will use a model that maintains a state so that it remembers what it read previously when predicting what comes next.\n",
+    "\n",
+    "**Next didn't understand why we need the two blocks of code**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "ebfea93b-eccd-44f8-9073-c037ad8cdeef",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>xxbos</td>\n",
+       "      <td>xxmaj</td>\n",
+       "      <td>in</td>\n",
+       "      <td>this</td>\n",
+       "      <td>chapter</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>movie</td>\n",
+       "      <td>reviews</td>\n",
+       "      <td>we</td>\n",
+       "      <td>studied</td>\n",
+       "      <td>in</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>first</td>\n",
+       "      <td>we</td>\n",
+       "      <td>will</td>\n",
+       "      <td>look</td>\n",
+       "      <td>at</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>how</td>\n",
+       "      <td>to</td>\n",
+       "      <td>customize</td>\n",
+       "      <td>it</td>\n",
+       "      <td>.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>of</td>\n",
+       "      <td>the</td>\n",
+       "      <td>preprocessor</td>\n",
+       "      <td>used</td>\n",
+       "      <td>in</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>will</td>\n",
+       "      <td>study</td>\n",
+       "      <td>how</td>\n",
+       "      <td>we</td>\n",
+       "      <td>build</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "bs,seq_len=6,5\n",
+    "d_tokens=np.array([tokens[i*15:i*15+seq_len] for i in range(bs)])\n",
+    "df = pd.DataFrame(d_tokens)\n",
+    "display(HTML(df.to_html(index=False,header=None)))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "c1c97fa8-b3fd-4dd9-a74f-5d975580784b",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>,</td>\n",
+       "      <td>we</td>\n",
+       "      <td>will</td>\n",
+       "      <td>go</td>\n",
+       "      <td>back</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>chapter</td>\n",
+       "      <td>1</td>\n",
+       "      <td>and</td>\n",
+       "      <td>dig</td>\n",
+       "      <td>deeper</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>the</td>\n",
+       "      <td>processing</td>\n",
+       "      <td>steps</td>\n",
+       "      <td>necessary</td>\n",
+       "      <td>to</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>xxmaj</td>\n",
+       "      <td>by</td>\n",
+       "      <td>doing</td>\n",
+       "      <td>this</td>\n",
+       "      <td>,</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>the</td>\n",
+       "      <td>data</td>\n",
+       "      <td>block</td>\n",
+       "      <td>xxup</td>\n",
+       "      <td>api</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>a</td>\n",
+       "      <td>language</td>\n",
+       "      <td>model</td>\n",
+       "      <td>and</td>\n",
+       "      <td>train</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# this one\n",
+    "bs,seq_len = 6,5\n",
+    "d_tokens = np.array([tokens[i*15+seq_len:i*15+2*seq_len] for i in range(bs)])\n",
+    "df = pd.DataFrame(d_tokens)\n",
+    "display(HTML(df.to_html(index=False,header=None)))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "4bd212b9-c832-4ba8-9526-089accbd729c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>over</td>\n",
+       "      <td>the</td>\n",
+       "      <td>example</td>\n",
+       "      <td>of</td>\n",
+       "      <td>classifying</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>under</td>\n",
+       "      <td>the</td>\n",
+       "      <td>surface</td>\n",
+       "      <td>.</td>\n",
+       "      <td>xxmaj</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>convert</td>\n",
+       "      <td>text</td>\n",
+       "      <td>into</td>\n",
+       "      <td>numbers</td>\n",
+       "      <td>and</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>we</td>\n",
+       "      <td>'ll</td>\n",
+       "      <td>have</td>\n",
+       "      <td>another</td>\n",
+       "      <td>example</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>.</td>\n",
+       "      <td>\\n</td>\n",
+       "      <td>xxmaj</td>\n",
+       "      <td>then</td>\n",
+       "      <td>we</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>it</td>\n",
+       "      <td>for</td>\n",
+       "      <td>a</td>\n",
+       "      <td>while</td>\n",
+       "      <td>.</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# and this one\n",
+    "bs,seq_len = 6,5\n",
+    "d_tokens = np.array([tokens[i*15+10:i*15+15] for i in range(bs)])\n",
+    "df = pd.DataFrame(d_tokens)\n",
+    "display(HTML(df.to_html(index=False,header=None)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0ba40f2-9df7-4190-ab7e-8a4e1627ac1a",
+   "metadata": {},
+   "source": [
+    "First step is to transform individual texts into stream by concatenating them together and at the beginning of each epoch, we will shuffle entries to make a new stream(we shuffle order of documents and not the order of words inside them).\n",
+    "\n",
+    "We then cut the stream into a certain number of batches(batch size). For examples, if stream has 50,000 tokens and we set batch size as 10, this gives us 10 min-streams of 5000 tokens. What is important is that we preserve the order of tokens(i.e first mini-batch from 1-5000 second from 5001-10000 and so on). An `xxbos` is added so that the model knows when a new stream is beginning when it reads it.\n",
+    "\n",
+    "So to recap:\n",
+    "\n",
+    "- First we shuffle collection of docs(such as articles,text,reviews,etc) at each epoch\n",
+    "- Concatenate them into a long stream\n",
+    "- Cut the stream into mini-streams\n",
+    "- Group mini-streams into batches\n",
+    "- Then the model will read mini-streams in order. It maintains an internal state allowing it to carry context from one mini-stream to next within a batch.\n",
+    "\n",
+    "This is all done behind the scenes by fastai library when we create an `LMDataLoader`. We do this by applying our `Numericalize` object to tokenised texts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "545ce1bb-25c6-4fa5-bd99-ad0a21e5077f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "nums200=toks200.map(num)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "id": "f9545e84-4691-4d78-8acd-e8e345f78afa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# then pass to LMDataLoader\n",
+    "dl=LMDataLoader(nums200)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "09c03299-39a6-4266-bb87-973d47a1e53f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(torch.Size([64, 72]), torch.Size([64, 72]))"
+      ]
+     },
+     "execution_count": 28,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# let's confirm it gives expected length by grabbing the first batch\n",
+    "x,y=first(dl)\n",
+    "x.shape,y.shape"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b932bddc-c7c1-430f-9672-50d187cecc2b",
+   "metadata": {},
+   "source": [
+    "then we look the independent variable which should be the start of first text."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "29108e71-1229-45fe-87ee-038c435e1e28",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'xxbos xxmaj once again xxmaj mr . xxmaj costner has dragged out a movie for far longer than necessary .'"
+      ]
+     },
+     "execution_count": 29,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "' '.join(num.vocab[o] for o in x[0][:20])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "799bfb8a-4175-4dd4-b5ee-87b99080cb7b",
+   "metadata": {},
+   "source": [
+    "The dependent variable is the same thing offset by one token"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "c570202c-0806-46aa-9f38-d8b9e0361f4a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'xxmaj once again xxmaj mr . xxmaj costner has dragged out a movie for far longer than necessary . xxmaj'"
+      ]
+     },
+     "execution_count": 30,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "' '.join(num.vocab[o] for o in y[0][:20])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b541d6e3-dbb9-4889-840c-17078d32a623",
+   "metadata": {},
+   "source": [
+    "This includes all the preprocessing steps we need to apply to our data and now we are ready to train our text classifier."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "640d5d9b-6bc5-49ee-8bdf-720656b452a2",
+   "metadata": {},
+   "source": [
+    "## Training a text classifier\n",
+    "\n",
+    "There are 2 steps to training a state-of-the-art text classifierusing transfer learning:\n",
+    "\n",
+    "1. fine-tune language model pretrained on Wikipedia to corpus of IMDb reviews\n",
+    "2. use the model to train a text classifier.\n",
+    "\n",
+    "Lets' start by assembling our data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c1bdab2f-fd9b-4623-b264-540bb2d1322a",
+   "metadata": {},
+   "source": [
+    "### Language model using DataBlock\n",
+    "\n",
+    "fastai handles tokenization and numericalization automatically when `TextBlock` is passed to `DataBlock`. All arguments that can be passed to `Tokenize` and `Numericalize` can alse be passed to `TextBlock`.\n",
+    "\n",
+    "In order to debug, we can run them manually on a subset of data as we have done before and also `DataBlock`'s `summary` method is very useful fo debugging data issues.\n",
+    "\n",
+    "Here's how we use `TextBlock` to create a language model using fastai's deafults\n",
+    "\n",
+    "```\n",
+    "get_imdb=partial(get_text_files,folders=['train','test','unsup'])\n",
+    "```\n",
+    "partial is used to create a new function with some of arguments of original function partially applied/prefilled.\n",
+    "\n",
+    "This line of code creates a new function get_imdb which always calls `get_text_files` with folders argument set to `['train', 'test', 'unsup']`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "59dde28c-7d57-4db7-9a14-21a91acab793",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "get_imdb=partial(get_text_files,folders=['train','test','unsup'])\n",
+    "dls_lm=DataBlock(\n",
+    "    blocks=TextBlock.from_folder(path,is_lm=True),\n",
+    "    get_items=get_imdb,splitter=RandomSplitter(0.1)\n",
+    ").dataloaders(path,path=path,bs=128,seq_len=80)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "92905fcf-2a17-4241-a1f7-3b5d26c4f8ce",
+   "metadata": {},
+   "source": [
+    "In this `DataBlock`, we are not using the `TextBlock` class directly but we are using the class method. We need to tell `TextBlock` how to access the texts, so that it can do initial preprocessing- that's what `from_folder` does."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "id": "a1a7d1b0-a411-4e14-a781-e78bfd18bf9f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>text</th>\n",
+       "      <th>text_</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>xxbos xxmaj africa xxmaj screams , one of the least seen of abbott&amp;costello 's films was an independent production that was released through xxmaj united xxmaj artists . xxmaj the thin plot has xxmaj hillary xxmaj brooke believing xxmaj costello has the map to a hidden territory that is rich with diamonds . xxmaj bud and xxmaj lou go to xxmaj africa at her behest with her two companions , the fighting xxmaj baer xxmaj brothers . xxmaj of course</td>\n",
+       "      <td>xxmaj africa xxmaj screams , one of the least seen of abbott&amp;costello 's films was an independent production that was released through xxmaj united xxmaj artists . xxmaj the thin plot has xxmaj hillary xxmaj brooke believing xxmaj costello has the map to a hidden territory that is rich with diamonds . xxmaj bud and xxmaj lou go to xxmaj africa at her behest with her two companions , the fighting xxmaj baer xxmaj brothers . xxmaj of course the</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>xxmaj xxunk and they got the balls to make the xxmaj christians out to be the intolerant , xenophobic and reactionary half - wits . \\n\\n xxmaj moral xxmaj orel is still an interesting watch ( as long as it comes between superior shows on xxmaj adult xxmaj swim ) because it is a satire . xxmaj however , xxmaj it is more a satire on the people that make it rather then the people it is depicting . \\n\\n</td>\n",
+       "      <td>xxunk and they got the balls to make the xxmaj christians out to be the intolerant , xenophobic and reactionary half - wits . \\n\\n xxmaj moral xxmaj orel is still an interesting watch ( as long as it comes between superior shows on xxmaj adult xxmaj swim ) because it is a satire . xxmaj however , xxmaj it is more a satire on the people that make it rather then the people it is depicting . \\n\\n xxmaj</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "dls_lm.show_batch(max_n=2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d8be9de0-3a29-4784-a45e-692f9cb8486f",
+   "metadata": {},
+   "source": [
+    "Now since our data is ready, we can fine-tune the pretrained language model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "539c7395-4e5d-48bb-9601-a5f4b5983b31",
+   "metadata": {},
+   "source": [
+    "### Fine-tuning Language model\n",
+    "\n",
+    "- To convert integer word indices into activations that we can use for our NN, we will use embeddings.\n",
+    "- Then we'll feed those embeddings into a RNN using an architecture called *AWD-LSTM*.\n",
+    "\n",
+    "The embeddings of pretrained model are merged with random embeddings added for words that were not in the preatrained vocab. This is handled automatically inside `language_model_learner`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "id": "6096ebdb-5bc7-4df3-a349-3f8a0790f20e",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    <div>\n",
+       "      <progress value='105070592' class='' max='105067061' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
+       "      100.00% [105070592/105067061 00:00&lt;00:00]\n",
+       "    </div>\n",
+       "    "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn=language_model_learner(\n",
+    "    dls_lm,AWD_LSTM, drop_mult=0.3,\n",
+    "    metrics=[accuracy,Perplexity()]).to_fp16()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2230d39c-e941-491e-bed7-8615e4168edc",
+   "metadata": {},
+   "source": [
+    "The loss function used is cross-entropy since this is a classification problem.\n",
+    "\n",
+    "Perplexity is a metric that is exponential of loss `torch.exp(cross_entropy)`.\n",
+    "\n",
+    "We'll use `fit_one_cycle` and save intermediate results during model training. Just like `vision_learner`, `language_model_learner` autimatically calls `freeze` when using a pretrained model, so this will only train the embeddings, the only part of model that contains randomly initialised weights i.e. embeddings of words that aren;'t in the pretrained vocabulary but are in the IMDb vocab.\n",
+    "\n",
+    "By choosing `fit_one_cycle` over `fine_tune`, you're opting for more manual control over the training process, which is often necessary when working with large models or datasets where you need to carefully manage resources and want to ensure you can resume training from checkpoints if needed.\n",
+    "Remember, you'll need to manually handle aspects like unfreezing layers if you switch from `fine_tune` to `fit_one_cycle`, as `fine_tune` automatically handles some of these steps for you."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "id": "c273b87a-6980-44bd-bf45-702afb6f4874",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: left;\">\n",
+       "      <th>epoch</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>accuracy</th>\n",
+       "      <th>perplexity</th>\n",
+       "      <th>time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>0</td>\n",
+       "      <td>4.004734</td>\n",
+       "      <td>3.904704</td>\n",
+       "      <td>0.300501</td>\n",
+       "      <td>49.635380</td>\n",
+       "      <td>20:57</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn.fit_one_cycle(1,2e-2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aa198661-09ee-4e26-a42a-28315582afd8",
+   "metadata": {},
+   "source": [
+    "### Saving and loading models\n",
+    "\n",
+    "We can easily save the staate of our model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "id": "46714e01-4f33-4966-9079-9c76c3babcf4",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Path('/teamspace/studios/this_studio/.fastai/data/imdb/models/1epoch.pth')"
+      ]
+     },
+     "execution_count": 35,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "learn.save('1epoch')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ab818ed4-79f8-4ad7-808b-e27d9700b2e0",
+   "metadata": {},
+   "source": [
+    "This will create a file in `learn.path/models/` named *1epoch.pth*. We can then load the contents of the loaded model to resume training or load it on another machine after creating our `Learner` as creatred here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "id": "89d0120f-0813-415e-b8d8-45f326b9b75e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "learn=learn.load('1epoch')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0d679c1c-a9c2-4602-919b-29ab209b8f47",
+   "metadata": {},
+   "source": [
+    "Once initial training has been completed, we continue fine-tuning model after unfreezing.\n",
+    "\n",
+    "`unfreeze()` is used to make all laeyrs of NN trainable.\n",
+    "\n",
+    "In context of TL,\n",
+    "\n",
+    "- we often start witha pre-trained model\n",
+    "- initially, we freeze the early layers and train only last few layers on our new data.\n",
+    "\n",
+    "What it does?\n",
+    "\n",
+    "- it sets `requires_grad_` to `True` for all parameters of the model, therefore during backpropagation, gradienst will be computed for all layers allowing them to be updated during training.\n",
+    "\n",
+    "When to Use unfreeze():\n",
+    "\n",
+    "- After initial training with frozen layers, when you want to fine-tune the entire model.\n",
+    "- When you have enough data and computational resources to train the entire network.\n",
+    "- When you want to allow the model to adapt more deeply to your specific task or dataset.\n",
+    "\n",
+    "Typical Workflow:\n",
+    "\n",
+    "- Start with a pre-trained model with most layers frozen.\n",
+    "- Train only the last few layers for some epochs.\n",
+    "- Call learn.unfreeze() to unfreeze all layers.\n",
+    "- Continue training with a lower learning rate."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "id": "9a9cfc05-c470-4728-85c9-2027343d7f39",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: left;\">\n",
+       "      <th>epoch</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>accuracy</th>\n",
+       "      <th>perplexity</th>\n",
+       "      <th>time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>0</td>\n",
+       "      <td>3.774635</td>\n",
+       "      <td>3.761657</td>\n",
+       "      <td>0.317040</td>\n",
+       "      <td>43.019672</td>\n",
+       "      <td>21:39</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>1</td>\n",
+       "      <td>3.712948</td>\n",
+       "      <td>3.701956</td>\n",
+       "      <td>0.323748</td>\n",
+       "      <td>40.526478</td>\n",
+       "      <td>21:18</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>2</td>\n",
+       "      <td>3.622169</td>\n",
+       "      <td>3.654953</td>\n",
+       "      <td>0.329183</td>\n",
+       "      <td>38.665722</td>\n",
+       "      <td>21:09</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>3</td>\n",
+       "      <td>3.570991</td>\n",
+       "      <td>3.620903</td>\n",
+       "      <td>0.333016</td>\n",
+       "      <td>37.371307</td>\n",
+       "      <td>21:28</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>4</td>\n",
+       "      <td>3.512937</td>\n",
+       "      <td>3.600293</td>\n",
+       "      <td>0.335536</td>\n",
+       "      <td>36.608948</td>\n",
+       "      <td>21:28</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>5</td>\n",
+       "      <td>3.410577</td>\n",
+       "      <td>3.586111</td>\n",
+       "      <td>0.337969</td>\n",
+       "      <td>36.093430</td>\n",
+       "      <td>21:14</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>6</td>\n",
+       "      <td>3.365716</td>\n",
+       "      <td>3.577701</td>\n",
+       "      <td>0.339490</td>\n",
+       "      <td>35.791176</td>\n",
+       "      <td>21:41</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>7</td>\n",
+       "      <td>3.294542</td>\n",
+       "      <td>3.575607</td>\n",
+       "      <td>0.340427</td>\n",
+       "      <td>35.716278</td>\n",
+       "      <td>21:36</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>8</td>\n",
+       "      <td>3.241257</td>\n",
+       "      <td>3.578088</td>\n",
+       "      <td>0.340611</td>\n",
+       "      <td>35.805027</td>\n",
+       "      <td>21:28</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>9</td>\n",
+       "      <td>3.224428</td>\n",
+       "      <td>3.582369</td>\n",
+       "      <td>0.340430</td>\n",
+       "      <td>35.958630</td>\n",
+       "      <td>21:28</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn.unfreeze()\n",
+    "learn.fit_one_cycle(10,2e-3)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aeb319ee-922f-4d97-a0af-54d971af5139",
+   "metadata": {},
+   "source": [
+    "Once this is completed, we save all of our model except the final layer that converts activatioins to probabilities of picking each token in our vocabulary.\n",
+    "\n",
+    "The model not including the final layer is called encoder and we save it using `save_encoder`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 38,
+   "id": "d9b44fd9-6310-4037-9e1d-5eea33455b40",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "learn.save_encoder('fine_tuned')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0fee79d2-497f-4ea0-9d90-f224a19c2e3d",
+   "metadata": {},
+   "source": [
+    ">encoder : The model not including the task specific final layer(s). This term means the same as body in CNNs but encoder is more used in NLP.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43b958af-33cf-41e0-8b92-070065319682",
+   "metadata": {},
+   "source": [
+    "This completes the second stage, fine-tuning the language model. We can now use it to fine-tune a classifier using IMDb senetiment labels."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "28c890dd-c93d-4158-9b90-c71e1bc7a9f2",
+   "metadata": {},
+   "source": [
+    "### Text generation\n",
+    "\n",
+    "Before we go to fine-tuning classifier, we will use the model to generate random reviews."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "id": "3046722c-b702-4131-8fed-4a7a980c0903",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "TEXT='I liked this movie because'\n",
+    "N_WORDS=40\n",
+    "N_SENTENCES=2\n",
+    "preds=[learn.predict(TEXT,N_WORDS,temperature=0.75)\n",
+    "       for _ in range(N_SENTENCES)]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "id": "4acc8091-132d-4130-a295-3491500e7f11",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "i liked this movie because it was a \" b \" movie . If you like some of the worst movies you 'll ever see , you 'll like this one . If you 're looking for a good horror flick , watch\n",
+      "i liked this movie because it actually turned out to be a real good movie . i thought it was predictable and you could see it coming a mile away . If you can plan on watching the movie you have to please make\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"\\n\".join(preds))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "619ee507-c3cd-4951-b8d5-852fa265fab2",
+   "metadata": {},
+   "source": [
+    "### Creating a classifier DataLoaders\n",
+    "\n",
+    "Recap - A language model predicts next word of a documnet and doesn't require external labels, whereas our classifier requires external label in this case the sentiment of document.\n",
+    "\n",
+    "This means the structure of `DataBlock` for NLP classification will look familiar to image classification datasets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "id": "ad44458f-4739-415a-b94c-8d29da0b773e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dls_clas=DataBlock(\n",
+    "    blocks=(TextBlock.from_folder(path,vocab=dls_lm.vocab),CategoryBlock),\n",
+    "    get_y=parent_label,\n",
+    "    get_items=partial(get_text_files,folders=['train','test']),\n",
+    "    splitter=GrandparentSplitter(valid_name='test')\n",
+    ").dataloaders(path,path=path,bs=128,seq_len=72)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d8bfd1d2-d513-4261-ace2-434d5318a6b1",
+   "metadata": {},
+   "source": [
+    "The first path is positional argument in dataloaders and specifies the root directory where data is located.\n",
+    "\n",
+    "The second path=path is a keywrod argument and in context of text classification it specifies where to save tokenised inputs.\n",
+    "\n",
+    "Why we use `GrandparentSplitter`?\n",
+    "\n",
+    "because the `GrandparentSplitter` looks at the grandparent folder name to determine the split. The structure of the IDMDb dataset is as follows\n",
+    "```\n",
+    "path/\n",
+    "  train/\n",
+    "    pos/\n",
+    "      review1.txt\n",
+    "      review2.txt\n",
+    "      ...\n",
+    "    neg/\n",
+    "      review1.txt\n",
+    "      review2.txt\n",
+    "      ...\n",
+    "  test/\n",
+    "    pos/\n",
+    "      review1.txt\n",
+    "      review2.txt\n",
+    "      ...\n",
+    "    neg/\n",
+    "      review1.txt\n",
+    "      review2.txt\n",
+    "      ...\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 42,
+   "id": "23e4c155-732b-4065-81d7-8e0692a46fc8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>text</th>\n",
+       "      <th>category</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules of the match , both opponents have to go through tables in order to get the win . xxmaj benoit and xxmaj guerrero heated up early on by taking turns hammering first xxmaj spike and then xxmaj bubba xxmaj ray . a xxmaj german xxunk by xxmaj benoit to xxmaj bubba took the wind out of the xxmaj dudley brother . xxmaj spike tried to help his brother , but the referee restrained him while xxmaj benoit and xxmaj guerrero</td>\n",
+       "      <td>pos</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>xxbos xxmaj some have praised _ xxunk _ as a xxmaj disney adventure for adults . i do n't think so -- at least not for thinking adults . \\n\\n xxmaj this script suggests a beginning as a live - action movie , that struck someone as the type of crap you can not sell to adults anymore . xxmaj the \" crack staff \" of many older adventure movies has been done well before , ( think _ the xxmaj dirty xxmaj dozen _ ) but _ atlantis _ represents one of the worse films in that motif . xxmaj the characters are weak . xxmaj even the background that each member trots out seems stock and awkward at best . xxmaj an xxup md / xxmaj medicine xxmaj man , a tomboy mechanic whose father always wanted sons , if we have not at least seen these before</td>\n",
+       "      <td>neg</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>xxbos xxmaj some have praised xxunk xxmaj lost xxmaj xxunk as a xxmaj disney adventure for adults . i do n't think so -- at least not for thinking adults . \\n\\n xxmaj this script suggests a beginning as a live - action movie , that struck someone as the type of crap you can not sell to adults anymore . xxmaj the \" crack staff \" of many older adventure movies has been done well before , ( think xxmaj the xxmaj dirty xxmaj dozen ) but xxunk represents one of the worse films in that motif . xxmaj the characters are weak . xxmaj even the background that each member trots out seems stock and awkward at best . xxmaj an xxup md / xxmaj medicine xxmaj man , a tomboy mechanic whose father always wanted sons , if we have not at least seen these before ,</td>\n",
+       "      <td>neg</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "dls_clas.show_batch(max_n=3)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1912546-c498-4e22-ac48-ba5dfdd9ec5c",
+   "metadata": {},
+   "source": [
+    "The DataBlock definition is similiar to previous definition of DataBlock except two chnages\n",
+    "\n",
+    "1. `TextBlock.from_folder` no longer has `is_lm=True`\n",
+    "2. We pass the `vocab` we created for the language model fine-tuning.\n",
+    "\n",
+    "The reason why we pass the vocab of language model is to make sure we use the same correspondance of token to index. Otherwise the embeddings we learned in our fine-tuned language model won't make any sense to this model and fine-tuning step won't be of any use.\n",
+    "\n",
+    "And by not passing `is_lm=True`, we tell `TextBlock` that we have regular labeled data rather than using next tokens as labels.\n",
+    "\n",
+    "However one challenge is collating multiple docs into a mini-batch. An example of creating a mini-batch containing first 10 docs but fiirst we need to numericalize them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "34136991-9528-47c0-bbfb-e4e9d4c6983e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "nums_samp=toks200[:10].map(num)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a210facd-8b11-4923-a718-156d71e7a9a8",
+   "metadata": {},
+   "source": [
+    "Now let's see how many tokens each of 10 reviews have"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "id": "07a9e9a5-92d3-46b4-a979-398bc86efe90",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(#10) [207,314,267,378,304,156,197,187,162,229]"
+      ]
+     },
+     "execution_count": 44,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "nums_samp.map(len)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f677a713-7e2f-47fe-8455-fe8f92df47de",
+   "metadata": {},
+   "source": [
+    "PyTorch `DataLoader`s need to collate/arrange all items in a batch into a single tensor and that has fixed shape.\n",
+    "\n",
+    "We can't crop the doc because we will lose some info. We can't also squish a doc and there's no data augmentation at resent for NLP. So that leaves padding.\n",
+    "\n",
+    "We expand the shortest texts to make them all the same size and to do this, we use a special padding token that is ignored by our model. Additionally, to avoid memory issues and improve performance, we will batch together texts that are roughly the same lengths( with some shuffling for training set). We do this by sortng docs by length prior to each epoch. \n",
+    "\n",
+    "The result of this is that docs collated into a single bath will tend to be of similar lengths. Also we won't pad every batch to same size, but will instead use the size of largest doc in each batch as target size.\n",
+    "\n",
+    "Sorting and padding are automatically done by data block API when we use `TextBlock` with `is_lm=True`.\n",
+    "\n",
+    "We now create a model to classify our texts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "id": "9c54eb3d-da09-4dfd-8e1c-07e0f3bf60cc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "learn=text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5,\n",
+    "                              metrics=accuracy).to_fp16()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c6f88ebe-80fe-43cc-95b2-c1c89939be13",
+   "metadata": {},
+   "source": [
+    "The final step prior to training classifier is to load the encoder from fine-tuned language model. We use `load_encoder` instead of `load` because we only have pretrained weights avaible for encoder."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "b6a2cb8f-87cb-4bb9-bea3-50302b9b99c4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "learn=learn.load_encoder('fine_tuned')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "858aef35-9fbb-44ff-885f-aa7b0c99c256",
+   "metadata": {},
+   "source": [
+    "### Fine-Tuning classifier\n",
+    "\n",
+    "The last step is to train with discriminative learning rates and gradual unfreezing. In CV, we often unfreeze the model at once, but for NLP classifiers, we find that unfreezing a few layers at a time makes a real difference."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "id": "ce455eb5-a459-4b73-a76f-8bb848b6e5ab",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: left;\">\n",
+       "      <th>epoch</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>accuracy</th>\n",
+       "      <th>time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>0</td>\n",
+       "      <td>0.245921</td>\n",
+       "      <td>0.180068</td>\n",
+       "      <td>0.931320</td>\n",
+       "      <td>00:45</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn.fit_one_cycle(1,2e-2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2c928b78-cd6b-4edf-8ae3-22a936e3411f",
+   "metadata": {},
+   "source": [
+    "\"Freezing\" a layer means we're telling the model not to update the weights of that layer during training.\n",
+    "\n",
+    "We can pass -2 to freeze_to to freeze all except the last two parameter groups/layers i.e the last two layers will only be trainable.\n",
+    "\n",
+    "Meaning train only last two layers.\n",
+    "\n",
+    "Negative indices count from end just like Python lists."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 48,
+   "id": "4d98aab5-1a76-40e4-a73f-2404660a13b6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: left;\">\n",
+       "      <th>epoch</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>accuracy</th>\n",
+       "      <th>time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>0</td>\n",
+       "      <td>0.221303</td>\n",
+       "      <td>0.164449</td>\n",
+       "      <td>0.938120</td>\n",
+       "      <td>00:51</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn.freeze_to(-2)\n",
+    "learn.fit_one_cycle(1,slice(1e-2/(2.6**4),1e-2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7372cdf7-f42e-438e-8886-899696aa2174",
+   "metadata": {},
+   "source": [
+    "Then unfreeze at bit more and continue training"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 49,
+   "id": "4c55c179-f3ec-4934-9313-a1d4c6b02305",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: left;\">\n",
+       "      <th>epoch</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>accuracy</th>\n",
+       "      <th>time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>0</td>\n",
+       "      <td>0.190569</td>\n",
+       "      <td>0.148871</td>\n",
+       "      <td>0.944920</td>\n",
+       "      <td>01:07</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn.freeze_to(-3)\n",
+    "learn.fit_one_cycle(1,slice(5e-3/(2.6**4),5e-3))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ca5e211c-adb7-406f-ab0c-2370fdbdb890",
+   "metadata": {},
+   "source": [
+    "Finally unfreeze the whole model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 50,
+   "id": "bff4c568-7a87-46e7-b2d1-7f05217774ca",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "<style>\n",
+       "    /* Turns off some styling */\n",
+       "    progress {\n",
+       "        /* gets rid of default border in Firefox and Opera. */\n",
+       "        border: none;\n",
+       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
+       "        background-size: auto;\n",
+       "    }\n",
+       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
+       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
+       "    }\n",
+       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
+       "        background: #F44336;\n",
+       "    }\n",
+       "</style>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: left;\">\n",
+       "      <th>epoch</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>accuracy</th>\n",
+       "      <th>time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>0</td>\n",
+       "      <td>0.161605</td>\n",
+       "      <td>0.145473</td>\n",
+       "      <td>0.946080</td>\n",
+       "      <td>01:25</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>1</td>\n",
+       "      <td>0.146006</td>\n",
+       "      <td>0.146676</td>\n",
+       "      <td>0.947240</td>\n",
+       "      <td>01:25</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "learn.unfreeze()\n",
+    "learn.fit_one_cycle(2,slice(1e-3/(2.6**4),1e-3))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 53,
+   "id": "12a6d889-c20a-4c01-a134-04f6e1c8fff3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "learn.export('text-classifier-model-sentiment-movie-reviews')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07ca849b-04c0-4b3f-a04b-37c4a416ce05",
+   "metadata": {},
+   "source": [
+    "Certainly! Let's break down the concept of **discriminative learning rates** and then explain the code you've provided.\n",
+    "\n",
+    "Discriminative Learning Rates:\n",
+    "\n",
+    "1. Basic Concept:\n",
+    "   - Instead of using a single learning rate for all layers of the network, we use different learning rates for different parts of the model.\n",
+    "\n",
+    "2. Rationale:\n",
+    "   - In transfer learning, earlier layers often capture more general features, while later layers are more task-specific.\n",
+    "   - We want to change the earlier layers less (lower learning rate) and the later layers more (higher learning rate).\n",
+    "\n",
+    "3. Implementation:\n",
+    "   - Usually, we set lower learning rates for earlier layers and higher rates for later layers.\n",
+    "   - This allows fine-tuning without destroying the useful features learned during pre-training.\n",
+    "\n",
+    "Now, let's look at the code:\n",
+    "\n",
+    "```python\n",
+    "learn.fit_one_cycle(1, slice(5e-3/(2.6**4), 5e-3))\n",
+    "```\n",
+    "\n",
+    "Breaking it down:\n",
+    "\n",
+    "1. `learn.fit_one_cycle(1, ...)`:\n",
+    "   - This runs the training for 1 epoch using the \"one cycle\" policy.\n",
+    "   - One cycle policy involves varying the learning rate and momentum during training for better performance.\n",
+    "\n",
+    "2. `slice(5e-3/(2.6**4), 5e-3)`:\n",
+    "   - This creates a range of learning rates.\n",
+    "   - The lower bound is `5e-3/(2.6**4)` ≈ 0.000137\n",
+    "   - The upper bound is `5e-3` = 0.005\n",
+    "\n",
+    "3. Discriminative Learning Rates in Action:\n",
+    "   - The `slice` object tells fastai to use a range of learning rates.\n",
+    "   - The earliest layers (those that were frozen) will use the lower learning rate (0.000137).\n",
+    "   - The latest layers will use the higher learning rate (0.005).\n",
+    "   - Layers in between will use learning rates interpolated between these values.\n",
+    "\n",
+    "4. The Specific Values:\n",
+    "   - `5e-3` (0.005) is chosen as a reasonable maximum learning rate.\n",
+    "   - Dividing by `2.6**4` for the minimum creates a spread where each layer group's learning rate is about 2.6 times larger than the previous one.\n",
+    "\n",
+    "5. Why These Values:\n",
+    "   - The factor of 2.6 and the use of 5e-3 are often empirically determined to work well for many transfer learning tasks.\n",
+    "   - The exact values might be adjusted based on specific model performance.\n",
+    "\n",
+    "In summary, this code sets up a training run with discriminative learning rates, allowing different parts of the model to learn at different speeds. This approach often leads to more effective fine-tuning in transfer learning scenarios."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1a58261a-38f1-4dcf-984b-3375e1bbebe9",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "We saw two models here\n",
+    "\n",
+    "1. language model\n",
+    "2. classifier\n",
+    "\n",
+    "To build a state-of-the-art classifier, \n",
+    "\n",
+    "- we used a  preatrained language model\n",
+    "- fine-tuned it on corpus of our task\n",
+    "- then used its body(encoder) with a new head to do the classification."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1c6290a-2802-4957-9dff-e4b0af7023b4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

text-classifier-model-sentiment-movie-reviews (1)

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c95910bba75be4619f5833ab6012ccfac45c448820241f4210c837aec94d7f59
+size 233520166