preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprossing\n",
    "\n",
    "Before fine-tuning our Large Language Model, we need to ensure that our dataset is of good quality. This is important to achieve a good model. Since our input is text, we are unable to normalize the data. We can only look at texts and remove those that do not help improve our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ippe\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset, Dataset\n",
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"imdb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Duplications\n",
    "First we check for duplications"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Amount of duplications in unsupervised: 493\n",
      "Amount of duplications in train: 96\n",
      "Amount of duplications in test: 199\n",
      "Amount of duplications combined: 418\n",
      "Amount of duplications between sets: 123\n"
     ]
    }
   ],
   "source": [
    "# Create dataframes\n",
    "df_unsupervised = pd.DataFrame(dataset[\"unsupervised\"][:]) \n",
    "df_train = pd.DataFrame(dataset[\"train\"][:])\n",
    "df_test = pd.DataFrame(dataset[\"test\"][:])  \n",
    "\n",
    "# Combine to check for duplication between sets (inspired by Pandas Docs)\n",
    "frames = [ df_test, df_train]\n",
    "df_total = pd.concat(frames)\n",
    "\n",
    "# Calculate amount of duplicates for each dataframe\n",
    "dups_unsupervised = df_unsupervised.duplicated().sum()\n",
    "dups_train = df_train.duplicated().sum()\n",
    "dups_test = df_test.duplicated().sum()\n",
    "dups_total = df_total.duplicated().sum()\n",
    "\n",
    "print(\"Amount of duplications in unsupervised: \" + str(dups_unsupervised)) \n",
    "print(\"Amount of duplications in train: \" + str(dups_train)) \n",
    "print(\"Amount of duplications in test: \" + str(dups_test )) \n",
    "print(\"Amount of duplications combined: \" + str(dups_total )) \n",
    "print(\"Amount of duplications between sets: \" + str(dups_total - (  dups_train + dups_test)))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-english characters\n",
    "Now, we would like to confirm that all the texts are in English. If not, it may decrease the model's accuracy. First we want to scan all the texts to see if there are any non-english sentences, such as Japanese or Chinese. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# print(dataset)\n",
    "# print(dataset[\"train\"][:5])\n",
    "\n",
    "# check for any non-english characters\n",
    "\n",
    "def is_not_english(text):       # returns true for unicode > U+0386, which are not used in English\n",
    "    return re.sub(r'[\\u0386-\\u2010\\u2E7F-\\uD7FC]', '',  text)     \n",
    "\n",
    "def print_non_english_texts(text_set):\n",
    "    for text in text_set:\n",
    "        if is_not_english(text):\n",
    "            print(text)\n",
    "            print('=' * 80)\n",
    "\n",
    "# print_non_english_texts(dataset[\"unsupervised\"][\"text\"])\n",
    "# print_non_english_texts(dataset[\"test\"][\"text\"])\n",
    "# print_non_english_texts(dataset[\"train\"][\"text\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this does print some sentences, it only prints sentences that contain some small words that are non-english."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language checking\n",
    "Now, there are some languages that share the same alphabet as English, such as Dutch. We are unable to check based on the characters to see if it is a different language. Therefore, we can use a use a library that can detect the language to see if all of them are in English. **Be aware that it takes a lot of time to run the code below.** \n",
    "\n",
    "Source used for implementing code: https://medium.com/@j.boldsen.ryan/detecting-languages-with-spacy-and-spacy-langdetect-0b733a2a06d2\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Library\n",
    "import spacy\n",
    "from spacy.language import Language  # For custom pipeline components\n",
    "from spacy_langdetect import LanguageDetector  # For language detection\n",
    "import pandas as pd  # For working with dataframes\n",
    "from tqdm import tqdm  # For progress bars\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'spacy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[12], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# load the English model\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m nlp \u001b[38;5;241m=\u001b[39m \u001b[43mspacy\u001b[49m\u001b[38;5;241m.\u001b[39mload(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124men_core_web_sm\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m      4\u001b[0m \u001b[38;5;66;03m# create pandas dataframe of text only\u001b[39;00m\n\u001b[0;32m      5\u001b[0m texts_unsupervised \u001b[38;5;241m=\u001b[39m df_unsupervised[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtext\u001b[39m\u001b[38;5;124m\"\u001b[39m]\u001b[38;5;241m.\u001b[39mtolist()\n",
      "\u001b[1;31mNameError\u001b[0m: name 'spacy' is not defined"
     ]
    }
   ],
   "source": [
    "# load the English model\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# create pandas dataframe of text only\n",
    "texts_unsupervised = df_unsupervised[\"text\"].tolist()\n",
    "texts_train = df_train[\"text\"].tolist()\n",
    "texts_test = df_test[\"text\"].tolist()\n",
    "texts_total = df_total[\"text\"].tolist()\n",
    "\n",
    "\n",
    "texts_unsupervised = [str(text) for text in texts_unsupervised]\n",
    "texts_train = [str(text) for text in texts_train]\n",
    "texts_test = [str(text) for text in texts_test]\n",
    "texts_total = [str(text) for text in texts_total]\n",
    "\n",
    "# Custom language detector factory function\n",
    "@Language.factory(\"language_detector\") \n",
    "def create_language_detector(nlp, name):\n",
    "    return LanguageDetector() # Create the detector component\n",
    "# added seed for reproducability\n",
    "\n",
    "# Add language detector to the spaCy pipeline\n",
    "nlp.add_pipe(\"language_detector\", last=True)  \n",
    "\n",
    "# Function to check if the text is in English\n",
    "def is_not_english(text):\n",
    "    doc = nlp(text)  # Process the text with the spaCy pipeline\n",
    "    detect_language = doc._.language  # Access language detection results\n",
    "    return detect_language['language'] != 'en'  # Check if detected language is English\n",
    "\n",
    "# Check if the texts are in English and store the result in a list\n",
    "english_checks_unsupervised = [is_not_english(text) for text in tqdm(texts_unsupervised, desc=\"Checking if texts are not in English\")] \n",
    "english_checks_train = [is_not_english(text) for text in tqdm(texts_train, desc=\"Checking if texts are not in English\")] \n",
    "english_checks_test = [is_not_english(text) for text in tqdm(texts_test, desc=\"Checking if texts are not in English\")] \n",
    "english_checks_total = [is_not_english(text) for text in tqdm(texts_test, desc=\"Checking if texts are not in English\")] \n",
    "\n",
    "# Print the amount of non-english texts:\n",
    "print(\"Amount of non-english texts in unsupervised: \" + str(sum(english_checks_unsupervised)))\n",
    "print(\"Amount of non-english texts in train: \" + str(sum(english_checks_train)))\n",
    "print(\"Amount of non-english texts in test: \" + str(sum(english_checks_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unsupervised:\n",
      "8132: Big joke! Not even bad enough to be interesting.\n",
      "47341: This movie is wacko.Is a story not so good. I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it.I didn't like it. Story line = 0 Music = 0 Martin Lawrence = 10 / = still awful\n",
      "Train:\n",
      "Test:\n",
      "21980: .....whoops - looks like it's gonna cost you a whopping £198.00 to buy a copy (either DVD or Video format)from ITV direct.<br /><br />Ouch.<br /><br />Sorry about this, but IMDB won't let me submit this comment unless it has at least 10 lines, so...........<br /><br />blahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblah blahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblah blahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblah blahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblah blahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblah blahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblahblah blahblah !!<br /><br />\n"
     ]
    }
   ],
   "source": [
    "def print_out_marked_texts(text_array, checks_array):\n",
    "    for i in range(0, len(text_array)):\n",
    "        if checks_array[i]:\n",
    "            print(str(i) + ': ' + text_array[i])\n",
    "\n",
    "print(\"Unsupervised:\")\n",
    "print_out_marked_texts(texts_unsupervised, english_checks_unsupervised)\n",
    "\n",
    "print(\"Train:\")\n",
    "print_out_marked_texts(texts_train, english_checks_train)\n",
    "\n",
    "print(\"Test:\")\n",
    "print_out_marked_texts(texts_test, english_checks_test)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTML elements\n",
    "\n",
    "Now, we would want to remove all the <**br**/> elements in the datasets. To make sure that there no other HTML present we will remove them all at once.\n",
    "\n",
    "Code inspired by: https://www.geeksforgeeks.org/how-to-remove-html-tags-from-string-in-python/ and by the \"map()\" documentation of huggingface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This is just a precious little diamond. The play, the script are excellent. I cant compare this movie with anything else, maybe except the movie \"Leon\" wonderfully played by Jean Reno and Natalie Portman. But... What can I say about this one? This is the best movie Anne Parillaud has ever played in (See please \"Frankie Starlight\", she\\'s speaking English there) to see what I mean. The story of young punk girl Nikita, taken into the depraved world of the secret government forces has been exceptionally over used by Americans. Never mind the \"Point of no return\" and especially the \"La femme Nikita\" TV series. They cannot compare the original believe me! Trash these videos. Buy this one, do not rent it, BUY it. BTW beware of the subtitles of the LA company which \"translate\" the US release. What a disgrace! If you cant understand French, get a dubbed version. But you\\'ll regret later :)', 'When I say this is my favourite film of all time, that comment is not to be taken lightly. I probably watch far too many films than is healthy for me, and have loved quite a few of them. I first saw \"La Femme Nikita\" nearly ten years ago, and it still manages to be my absolute favourite. Why?<br /><br />This is more than an incredibly stylish and sexy thriller. Luc Besson\\'s great flair for impeccable direction, fashion, and appropriate usage of music makes this a very watchable film. But it is Anne Parillaud\\'s perfect rendering of a complex character who transforms from a heartless killer into a compassionate, vibrant young woman that makes this film beautiful. I can\\'t keep my eyes off of her when she is on screen.<br /><br />I have seen several of Luc Besson\\'s films including \"Subway\", \"The Professional\", and the irritating \"Fifth Element\", and \"Nikita\" is without a doubt, far superior to any of these. Although this film has tragic elements, it is ultimately extremely hopeful. It is the story of a person who is cruel and merciless, who ultimately comes to realize her own humanity and her own personal power. That, to me is extremely inspiring. If there is hope for Nikita, there is hope for all of us.', 'I saw this movie because I am a huge fan of the TV series of the same name starring Roy Dupuis and Pet Wilson. The movie was really good and I saw how the TV show is based on the movie. A few episodes of the TV series came directly from the movie and their similarity was amazing. To keep things short, any fan of the movie has to watch the series and any fan of the series must see the original Nikita.', \"Being that the only foreign films I usually like star a Japanese person in a rubber suit who crushes little tiny buildings and tanks, I had high hopes for this movie. I thought that this was a movie that wouldn't put me to sleep. WRONG! Starts off with a bang, okay, now she's in training, alright, she's an assassin, I'm still with you, oh, now she's having this moral dilemma and she can't decide if she loves her boyfriend or her controller, zzzzz.... Oh well, back to Gamera!\", \"After seeing Point of No Return (a great movie) and being told that the original was better, I was certainly thrilled to see that one of the indie film channels was running La Femme Nikita. Then I saw the movie. Ouch! This was a major let-down.<br /><br />Nikita herself reminds me of Jar Jar Binks more than any other character I've seen recently. She comes across entirely as comic relief. The movie simply has nothing to recommend it besides the core concept of an evil, inhuman character paradoxically learning to be human while training as an assassin, and that concept failed miserably in Nikita due to the poor writing of the title role.\", 'Stylish thriller? Hardly! This French action film from director Luc Besson is more confused and unsure about itself than anything else I have seen recently.<br /><br />The opening action sequence shows some promise, and the following action sequences are good. However they are sadly too few and far between. The drama is dull and erratic, and the characters are generally unlikeable. Anne Parillaud and the support cast are unable to involve the audience emotionally.<br /><br />\"La Femme Nikita\" ends up more like a try hard, B-grade action thriller. Very average!<br /><br />Saturday, April 28, 1991 - Balwyn Cinema', \"A Luc Besson (Léon, The Fith Element, The Messenger: The Story of Joan of Arc. Angel-A) film with Anne Parillaud (Innocent Blood), Tchéky Karyo(The Messenger: The Story of Joan of Arc, The Patriot, Kiss of the Dragon) and Jean Reno (Léon , The Crimson Rivers, Godzilla). It doesn't get better than this.<br /><br />The photography by Thierry Arbogast was perfect, especially the blue lighting at the pharmacy scene. It is easy to see why Besson uses him for his films. He is a master.<br /><br />Parillaud was absolutely magnificent in a character that was a strong woman, and a little girl at the same time. She was a profession, but she was also human. To see this instead of the comic book characters we are so familiar with is refreshing.<br /><br />Reno was a perfect cleaner. Hard and professional. Get the mission done.<br /><br />Jean-Hugues Anglade added just the right touch of sensitivity. Outstanding! There are just not enough superlatives to use in describing the acting and action in the film.\", 'This isn\\'t my typical movie genre. I am more interested in character drama than action movies. This is able to span both genres in a way that Hollywood hasn\\'t managed to accomplish. The plot was interesting and the acting by Anne Parillaud was exceptional.<br /><br />I had seen the American remake, called \"Point of No Return\" and found it to be an OK movie, but I wanted to see the original. I am glad that I did. The technical aspects of direction and editing are very good. The plot was interesting - revolving around the timeless theme of redemption. The acting made this an exceptional movie.<br /><br />For fans of the action genre, they will be quite happy. For those like myself who are more interested in character and plot-driven movies may find this to be a pleasant surprise as well.', \"La Femme Nikita (1990) was a great movie from genre director Luc Besson. The story is about a drugged out young woman (Anne Parillund) who's hung out with the wrong crowd one too many times. During a botched drug store robbery and subsequent shoot out with the police, NIkita is the only survivor. She's tried, sentenced and executed. But, she wakes up in a strange room and greeted by a quiet and subdued individual (Tcheky Kayro) he tells her that she's been given a second chance at life. But will she do what needs to be done in order for her to receive. Will Nikita take this opportunity at a second life or will she throw it all away? Watch LA FEMME NIKITA to find out.<br /><br />The always imitated by never properly duplicated chic thriller has become a fixture in pop culture. Not only was this film remade several times but it also spawned an unrelated television series. Anne Parillund is super hot in her role as Nikita. Watch out for Luc Besson regular Jean Reno as Victor the Cleaner.<br /><br />Highly recommended.\", 'Luc Besson has been behind some of the most provocative and intriguing action films of the last 15 years, and his introduction to mainstream American cinema-goers began with 1991\\'s \"La Femme Nikita.\" Beginning with a drug store shootout that leaves three junkie punkers and a few policemen dead, this film is rapidly engaging and shows off Besson\\'s violent chic. In the fray, only young Nikita (Anne Parillaud) survives, after offing a policeman who had let his guard down at a crucial moment. Facing certain death in France\\'s court of law, she is given two choices: lethal injection, or have her execution faked, and then assume a new life as a female super-assassin/spy. A wild-raised feral child of sorts, Nikita had picked up certain skills on the street that eventually prove to be quite valuable to her government handlers, as she makes things difficult for them and especially her chief mentor Bob (Tcheky Karyo). No secret is made of his affections for her, but he tries to mold Nikita into a valuable person, and he\\'s able to do it with the help of another agent, played by actress Jeanne Moreau, who teaches her the finer ways of womanhood. By the end of her training, Nikita has been modeled into a fine, sexy female assassin. Besson makes no attempts to sacrifice his film in the ways of Hollywood convention, and he proves that the days of female-driven action cinema are not yet dead. Parillaud is compelling, as at first, she almost seems unaware of her identity as a woman. When she isn\\'t taking out high-powered political officials, she\\'s romancing a kind and gentle shop clerk, who falls madly in love with her as she does with him. Lastly, \"La Femme Nikita\" has some of the most stylized and violent action scenes of any movie I\\'ve seen in a while. Luc Besson, with this film and much of his subsequent work, has proved he is one of cinema\\'s most valuable personalities on the European, Asian, and American action film circuits. Also, be aware of a cameo by future Besson \"Leon\" (1994) star Jean Reno as Victor the Cleaner, in what is pretty much an early prototype of his role from that star-making breakthrough film.<br /><br />8/10']\n"
     ]
    }
   ],
   "source": [
    "print(dataset[\"unsupervised\"][\"text\"][:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 50000/50000 [00:03<00:00, 16119.02 examples/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This is just a precious little diamond. The play, the script are excellent. I cant compare this movie with anything else, maybe except the movie \"Leon\" wonderfully played by Jean Reno and Natalie Portman. But... What can I say about this one? This is the best movie Anne Parillaud has ever played in (See please \"Frankie Starlight\", she\\'s speaking English there) to see what I mean. The story of young punk girl Nikita, taken into the depraved world of the secret government forces has been exceptionally over used by Americans. Never mind the \"Point of no return\" and especially the \"La femme Nikita\" TV series. They cannot compare the original believe me! Trash these videos. Buy this one, do not rent it, BUY it. BTW beware of the subtitles of the LA company which \"translate\" the US release. What a disgrace! If you cant understand French, get a dubbed version. But you\\'ll regret later :)', 'When I say this is my favourite film of all time, that comment is not to be taken lightly. I probably watch far too many films than is healthy for me, and have loved quite a few of them. I first saw \"La Femme Nikita\" nearly ten years ago, and it still manages to be my absolute favourite. Why?This is more than an incredibly stylish and sexy thriller. Luc Besson\\'s great flair for impeccable direction, fashion, and appropriate usage of music makes this a very watchable film. But it is Anne Parillaud\\'s perfect rendering of a complex character who transforms from a heartless killer into a compassionate, vibrant young woman that makes this film beautiful. I can\\'t keep my eyes off of her when she is on screen.I have seen several of Luc Besson\\'s films including \"Subway\", \"The Professional\", and the irritating \"Fifth Element\", and \"Nikita\" is without a doubt, far superior to any of these. Although this film has tragic elements, it is ultimately extremely hopeful. It is the story of a person who is cruel and merciless, who ultimately comes to realize her own humanity and her own personal power. That, to me is extremely inspiring. If there is hope for Nikita, there is hope for all of us.', 'I saw this movie because I am a huge fan of the TV series of the same name starring Roy Dupuis and Pet Wilson. The movie was really good and I saw how the TV show is based on the movie. A few episodes of the TV series came directly from the movie and their similarity was amazing. To keep things short, any fan of the movie has to watch the series and any fan of the series must see the original Nikita.', \"Being that the only foreign films I usually like star a Japanese person in a rubber suit who crushes little tiny buildings and tanks, I had high hopes for this movie. I thought that this was a movie that wouldn't put me to sleep. WRONG! Starts off with a bang, okay, now she's in training, alright, she's an assassin, I'm still with you, oh, now she's having this moral dilemma and she can't decide if she loves her boyfriend or her controller, zzzzz.... Oh well, back to Gamera!\", \"After seeing Point of No Return (a great movie) and being told that the original was better, I was certainly thrilled to see that one of the indie film channels was running La Femme Nikita. Then I saw the movie. Ouch! This was a major let-down.Nikita herself reminds me of Jar Jar Binks more than any other character I've seen recently. She comes across entirely as comic relief. The movie simply has nothing to recommend it besides the core concept of an evil, inhuman character paradoxically learning to be human while training as an assassin, and that concept failed miserably in Nikita due to the poor writing of the title role.\", 'Stylish thriller? Hardly! This French action film from director Luc Besson is more confused and unsure about itself than anything else I have seen recently.The opening action sequence shows some promise, and the following action sequences are good. However they are sadly too few and far between. The drama is dull and erratic, and the characters are generally unlikeable. Anne Parillaud and the support cast are unable to involve the audience emotionally.\"La Femme Nikita\" ends up more like a try hard, B-grade action thriller. Very average!Saturday, April 28, 1991 - Balwyn Cinema', \"A Luc Besson (Léon, The Fith Element, The Messenger: The Story of Joan of Arc. Angel-A) film with Anne Parillaud (Innocent Blood), Tchéky Karyo(The Messenger: The Story of Joan of Arc, The Patriot, Kiss of the Dragon) and Jean Reno (Léon , The Crimson Rivers, Godzilla). It doesn't get better than this.The photography by Thierry Arbogast was perfect, especially the blue lighting at the pharmacy scene. It is easy to see why Besson uses him for his films. He is a master.Parillaud was absolutely magnificent in a character that was a strong woman, and a little girl at the same time. She was a profession, but she was also human. To see this instead of the comic book characters we are so familiar with is refreshing.Reno was a perfect cleaner. Hard and professional. Get the mission done.Jean-Hugues Anglade added just the right touch of sensitivity. Outstanding! There are just not enough superlatives to use in describing the acting and action in the film.\", 'This isn\\'t my typical movie genre. I am more interested in character drama than action movies. This is able to span both genres in a way that Hollywood hasn\\'t managed to accomplish. The plot was interesting and the acting by Anne Parillaud was exceptional.I had seen the American remake, called \"Point of No Return\" and found it to be an OK movie, but I wanted to see the original. I am glad that I did. The technical aspects of direction and editing are very good. The plot was interesting - revolving around the timeless theme of redemption. The acting made this an exceptional movie.For fans of the action genre, they will be quite happy. For those like myself who are more interested in character and plot-driven movies may find this to be a pleasant surprise as well.', \"La Femme Nikita (1990) was a great movie from genre director Luc Besson. The story is about a drugged out young woman (Anne Parillund) who's hung out with the wrong crowd one too many times. During a botched drug store robbery and subsequent shoot out with the police, NIkita is the only survivor. She's tried, sentenced and executed. But, she wakes up in a strange room and greeted by a quiet and subdued individual (Tcheky Kayro) he tells her that she's been given a second chance at life. But will she do what needs to be done in order for her to receive. Will Nikita take this opportunity at a second life or will she throw it all away? Watch LA FEMME NIKITA to find out.The always imitated by never properly duplicated chic thriller has become a fixture in pop culture. Not only was this film remade several times but it also spawned an unrelated television series. Anne Parillund is super hot in her role as Nikita. Watch out for Luc Besson regular Jean Reno as Victor the Cleaner.Highly recommended.\", 'Luc Besson has been behind some of the most provocative and intriguing action films of the last 15 years, and his introduction to mainstream American cinema-goers began with 1991\\'s \"La Femme Nikita.\" Beginning with a drug store shootout that leaves three junkie punkers and a few policemen dead, this film is rapidly engaging and shows off Besson\\'s violent chic. In the fray, only young Nikita (Anne Parillaud) survives, after offing a policeman who had let his guard down at a crucial moment. Facing certain death in France\\'s court of law, she is given two choices: lethal injection, or have her execution faked, and then assume a new life as a female super-assassin/spy. A wild-raised feral child of sorts, Nikita had picked up certain skills on the street that eventually prove to be quite valuable to her government handlers, as she makes things difficult for them and especially her chief mentor Bob (Tcheky Karyo). No secret is made of his affections for her, but he tries to mold Nikita into a valuable person, and he\\'s able to do it with the help of another agent, played by actress Jeanne Moreau, who teaches her the finer ways of womanhood. By the end of her training, Nikita has been modeled into a fine, sexy female assassin. Besson makes no attempts to sacrifice his film in the ways of Hollywood convention, and he proves that the days of female-driven action cinema are not yet dead. Parillaud is compelling, as at first, she almost seems unaware of her identity as a woman. When she isn\\'t taking out high-powered political officials, she\\'s romancing a kind and gentle shop clerk, who falls madly in love with her as she does with him. Lastly, \"La Femme Nikita\" has some of the most stylized and violent action scenes of any movie I\\'ve seen in a while. Luc Besson, with this film and much of his subsequent work, has proved he is one of cinema\\'s most valuable personalities on the European, Asian, and American action film circuits. Also, be aware of a cameo by future Besson \"Leon\" (1994) star Jean Reno as Victor the Cleaner, in what is pretty much an early prototype of his role from that star-making breakthrough film.8/10']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Remove html in text and return it\n",
    "def remove_html_element(text):\n",
    "    return re.sub(r\"<.*?>\", \"\", text)\n",
    "\n",
    "def remove_html_elements_dataset(dataset):\n",
    "    dataset[\"text\"] = remove_html_element(dataset[\"text\"])\n",
    "    return dataset\n",
    "\n",
    "dataset[\"unsupervised\"] = dataset[\"unsupervised\"].map(remove_html_elements_dataset)\n",
    "print(dataset[\"unsupervised\"][\"text\"][:10])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lowercasing\n",
    "\n",
    "When reviewing other NLP applications, we often see that all the inputs are put into lowercase. The reason why this is the case, is explained in the report. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 50000/50000 [00:02<00:00, 17272.17 examples/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['this is just a precious little diamond. the play, the script are excellent. i cant compare this movie with anything else, maybe except the movie \"leon\" wonderfully played by jean reno and natalie portman. but... what can i say about this one? this is the best movie anne parillaud has ever played in (see please \"frankie starlight\", she\\'s speaking english there) to see what i mean. the story of young punk girl nikita, taken into the depraved world of the secret government forces has been exceptionally over used by americans. never mind the \"point of no return\" and especially the \"la femme nikita\" tv series. they cannot compare the original believe me! trash these videos. buy this one, do not rent it, buy it. btw beware of the subtitles of the la company which \"translate\" the us release. what a disgrace! if you cant understand french, get a dubbed version. but you\\'ll regret later :)', 'when i say this is my favourite film of all time, that comment is not to be taken lightly. i probably watch far too many films than is healthy for me, and have loved quite a few of them. i first saw \"la femme nikita\" nearly ten years ago, and it still manages to be my absolute favourite. why?this is more than an incredibly stylish and sexy thriller. luc besson\\'s great flair for impeccable direction, fashion, and appropriate usage of music makes this a very watchable film. but it is anne parillaud\\'s perfect rendering of a complex character who transforms from a heartless killer into a compassionate, vibrant young woman that makes this film beautiful. i can\\'t keep my eyes off of her when she is on screen.i have seen several of luc besson\\'s films including \"subway\", \"the professional\", and the irritating \"fifth element\", and \"nikita\" is without a doubt, far superior to any of these. although this film has tragic elements, it is ultimately extremely hopeful. it is the story of a person who is cruel and merciless, who ultimately comes to realize her own humanity and her own personal power. that, to me is extremely inspiring. if there is hope for nikita, there is hope for all of us.', 'i saw this movie because i am a huge fan of the tv series of the same name starring roy dupuis and pet wilson. the movie was really good and i saw how the tv show is based on the movie. a few episodes of the tv series came directly from the movie and their similarity was amazing. to keep things short, any fan of the movie has to watch the series and any fan of the series must see the original nikita.', \"being that the only foreign films i usually like star a japanese person in a rubber suit who crushes little tiny buildings and tanks, i had high hopes for this movie. i thought that this was a movie that wouldn't put me to sleep. wrong! starts off with a bang, okay, now she's in training, alright, she's an assassin, i'm still with you, oh, now she's having this moral dilemma and she can't decide if she loves her boyfriend or her controller, zzzzz.... oh well, back to gamera!\", \"after seeing point of no return (a great movie) and being told that the original was better, i was certainly thrilled to see that one of the indie film channels was running la femme nikita. then i saw the movie. ouch! this was a major let-down.nikita herself reminds me of jar jar binks more than any other character i've seen recently. she comes across entirely as comic relief. the movie simply has nothing to recommend it besides the core concept of an evil, inhuman character paradoxically learning to be human while training as an assassin, and that concept failed miserably in nikita due to the poor writing of the title role.\", 'stylish thriller? hardly! this french action film from director luc besson is more confused and unsure about itself than anything else i have seen recently.the opening action sequence shows some promise, and the following action sequences are good. however they are sadly too few and far between. the drama is dull and erratic, and the characters are generally unlikeable. anne parillaud and the support cast are unable to involve the audience emotionally.\"la femme nikita\" ends up more like a try hard, b-grade action thriller. very average!saturday, april 28, 1991 - balwyn cinema', \"a luc besson (léon, the fith element, the messenger: the story of joan of arc. angel-a) film with anne parillaud (innocent blood), tchéky karyo(the messenger: the story of joan of arc, the patriot, kiss of the dragon) and jean reno (léon , the crimson rivers, godzilla). it doesn't get better than this.the photography by thierry arbogast was perfect, especially the blue lighting at the pharmacy scene. it is easy to see why besson uses him for his films. he is a master.parillaud was absolutely magnificent in a character that was a strong woman, and a little girl at the same time. she was a profession, but she was also human. to see this instead of the comic book characters we are so familiar with is refreshing.reno was a perfect cleaner. hard and professional. get the mission done.jean-hugues anglade added just the right touch of sensitivity. outstanding! there are just not enough superlatives to use in describing the acting and action in the film.\", 'this isn\\'t my typical movie genre. i am more interested in character drama than action movies. this is able to span both genres in a way that hollywood hasn\\'t managed to accomplish. the plot was interesting and the acting by anne parillaud was exceptional.i had seen the american remake, called \"point of no return\" and found it to be an ok movie, but i wanted to see the original. i am glad that i did. the technical aspects of direction and editing are very good. the plot was interesting - revolving around the timeless theme of redemption. the acting made this an exceptional movie.for fans of the action genre, they will be quite happy. for those like myself who are more interested in character and plot-driven movies may find this to be a pleasant surprise as well.', \"la femme nikita (1990) was a great movie from genre director luc besson. the story is about a drugged out young woman (anne parillund) who's hung out with the wrong crowd one too many times. during a botched drug store robbery and subsequent shoot out with the police, nikita is the only survivor. she's tried, sentenced and executed. but, she wakes up in a strange room and greeted by a quiet and subdued individual (tcheky kayro) he tells her that she's been given a second chance at life. but will she do what needs to be done in order for her to receive. will nikita take this opportunity at a second life or will she throw it all away? watch la femme nikita to find out.the always imitated by never properly duplicated chic thriller has become a fixture in pop culture. not only was this film remade several times but it also spawned an unrelated television series. anne parillund is super hot in her role as nikita. watch out for luc besson regular jean reno as victor the cleaner.highly recommended.\", 'luc besson has been behind some of the most provocative and intriguing action films of the last 15 years, and his introduction to mainstream american cinema-goers began with 1991\\'s \"la femme nikita.\" beginning with a drug store shootout that leaves three junkie punkers and a few policemen dead, this film is rapidly engaging and shows off besson\\'s violent chic. in the fray, only young nikita (anne parillaud) survives, after offing a policeman who had let his guard down at a crucial moment. facing certain death in france\\'s court of law, she is given two choices: lethal injection, or have her execution faked, and then assume a new life as a female super-assassin/spy. a wild-raised feral child of sorts, nikita had picked up certain skills on the street that eventually prove to be quite valuable to her government handlers, as she makes things difficult for them and especially her chief mentor bob (tcheky karyo). no secret is made of his affections for her, but he tries to mold nikita into a valuable person, and he\\'s able to do it with the help of another agent, played by actress jeanne moreau, who teaches her the finer ways of womanhood. by the end of her training, nikita has been modeled into a fine, sexy female assassin. besson makes no attempts to sacrifice his film in the ways of hollywood convention, and he proves that the days of female-driven action cinema are not yet dead. parillaud is compelling, as at first, she almost seems unaware of her identity as a woman. when she isn\\'t taking out high-powered political officials, she\\'s romancing a kind and gentle shop clerk, who falls madly in love with her as she does with him. lastly, \"la femme nikita\" has some of the most stylized and violent action scenes of any movie i\\'ve seen in a while. luc besson, with this film and much of his subsequent work, has proved he is one of cinema\\'s most valuable personalities on the european, asian, and american action film circuits. also, be aware of a cameo by future besson \"leon\" (1994) star jean reno as victor the cleaner, in what is pretty much an early prototype of his role from that star-making breakthrough film.8/10']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Lowercase text and return it\n",
    "def lowercase_text(text):\n",
    "    return text.lower()\n",
    "\n",
    "def lower_texts_dataset(dataset):\n",
    "    dataset[\"text\"] = lowercase_text(dataset[\"text\"])\n",
    "    return dataset\n",
    "\n",
    "dataset[\"unsupervised\"] = dataset[\"unsupervised\"].map(lower_texts_dataset)\n",
    "print(dataset[\"unsupervised\"][\"text\"][:10])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finalizing dataset\n",
    "Now that we know all the functions are working correctly we can apply them to the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New amount of duplicates in dataframe: 0\n",
      "              label\n",
      "count  49580.000000\n",
      "mean       0.501876\n",
      "std        0.500002\n",
      "min        0.000000\n",
      "25%        0.000000\n",
      "50%        1.000000\n",
      "75%        1.000000\n",
      "max        1.000000\n",
      "Dataset({\n",
      "    features: ['text', 'label'],\n",
      "    num_rows: 49580\n",
      "})\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving the dataset (1/1 shards): 100%|██████████| 49580/49580 [00:00<00:00, 668070.33 examples/s]\n"
     ]
    }
   ],
   "source": [
    "# Remove duplicates\n",
    "df_total = df_total.drop_duplicates()\n",
    "\n",
    "print(\"New amount of duplicates in dataframe: \" + str(df_total.duplicated().sum()))\n",
    "\n",
    "# remove non-english characters\n",
    "df_total[\"text\"] = df_total[\"text\"].apply(is_not_english)\n",
    "\n",
    "# Remove HTML elements\n",
    "df_total[\"text\"] = df_total[\"text\"].apply(remove_html_element)\n",
    "\n",
    "# Remove the row that contains the many amounts of \"blah\", since applying the Spacy Libary takes a lot of time, we have to filter out the text manualy. Inspired by: https://saturncloud.io/blog/how-to-remove-rows-from-pandas-data-frame-that-contains-any-string-in-a-particular-column/#:~:text=To%20remove%20rows%20from%20a%20pandas%20data%20frame%20that%20contains,a%20specified%20substring%20or%20not.\n",
    "substring = 'blahblahblahblahblahblah'\n",
    "filter = df_total['text'].str.contains(substring)\n",
    "df_total = df_total[~filter]\n",
    "\n",
    "\n",
    "# Set all texts to lowercase\n",
    "df_total[\"text\"] = df_total[\"text\"].apply(lowercase_text)\n",
    "\n",
    "print(df_total.describe())\n",
    "\n",
    "df_total = df_total.reset_index(drop=True)\n",
    "new_dataset = Dataset.from_pandas(df_total)\n",
    "\n",
    "print(new_dataset)\n",
    "\n",
    "# save dataset to disk, such that it can be used in training\n",
    "new_dataset.save_to_disk(\"preprocessed_dataset\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}