ProjectInstructions.md

Overview: You’ve been asked by a Frankenstein superfan to finetune a custom model that can write original Frankenstein fanfiction ahead of this year’s Halloween.

You have two options in accomplishing this task. You can either perform QLoRA on the Mistral 7B language model, or perform a full finetune on Distil GPT-2.

You will perform these tasks in finetuning the model:

Import and perform EDA on a dataset made of snippets from Frankenstein Convert data from Pandas into Hugging Face Datasets and tokenize the data If you’re doing QLoRA: Shrink a generative language model with 4bit quantization Configure LoRA to train a small subset of the quantized model’s parameters Configure and train the finetuned model Evaluate and compare the base model and the finetuned model using both perplexity and more informal methods

Setup: Each step on the kanban board to the right will be flagged with a comment in the project notebook (i.e., Step 1 will be marked with # Step 1 so you can find it.)

You have some choice in this project as to how you’ll perform this finetuning. The choice you make will depend on whether or not you have access to a Graphics Processing Unit (GPU), whether on your local device or via a cloud provider.

If all options are available to you, we recommend doing this project on Google Colab.

Option 1: QLoRA with Mistral7B

Download Mistral 7B GPU notebook using the following link: https://static-assets.codecademy.com/Courses/finetuning-transformer-models/FrankensteinGPUStarter.zip

When you should choose option 1

You can choose Mistral 7B, a cutting-edge 7 billion parameter language model, if you meet one of the following conditions:

You’re allowed to use Google Colab (https://colab.research.google.com) where you are You need to be logged into Google to use Colab, so make sure that’s OK too You are comfortable using Paperspace Gradient (https://paperspace.com/notebooks), Kaggle (https://kaggle.com), or similar cloud notebook providers You have a personal computer with an NVIDIA GPU Setup Guide

Let’s assume you’re using Colab.

First, head to https://colab.research.google.com and select the “Upload” option on the sidebar of the “Open notebook” screen. Then navigate to wherever you downloaded and unzipped the project folder and select the notebook.

Now make sure you’re using the GPU by navigating to “Runtime > Change runtime type” and selecting the T4 GPU.

Next, click on the folder icon on the left-hand side of Colab. Ensure you’re connected to the runtime and select the upload file icon (furthest left). Track down frankenstein_chunks.csv and select it. Now you’re ready to follow along with the project.

If you’re using Paperspace or another cloud notebook provider, the process is mostly the same, though you’ll have to find all the relevant options in the UI yourself. And if you’ve got a big enough GPU to perform this project locally, we’ll assume you know how to get yourself started.

Option 2: DistilGPT-2 on the CPU

Download DistilGPT2 CPU notebook with the following link: https://static-assets.codecademy.com/Courses/finetuning-transformer-models/FrankensteinCPUStarter.zip

When you should choose option 2

If none of those options are available to you, no problem. You can still use DistilGPT-2 with this lesson and you’ll perform a full finetune. This is a good option if you both:

Are at work and aren’t allowed to log into Google for security reasons Don’t have a powerful enough graphics card to finetune Mistral7B Setup Guide

Open the unzipped project folder in your IDE of choice and get started!

Resources: This project will only challenge you on concepts and syntax we’ve already covered in this course. You may want to open the exercises from the previous lesson in another tab to easily look up the relevant notebooks.

If you get stuck, consult the following solutions notebooks:

GPU Solution Notebook: https://static-assets.codecademy.com/Courses/finetuning-transformer-models/FrankensteinGPU.ipynb Reminder: This notebook will only work if you have access to a GPU, either locally or via a cloud notebook provider like Colab.

CPU Solution Notebook https://static-assets.codecademy.com/Courses/finetuning-transformer-models/FrankensteinMPSCPU.ipynb If you’d like to learn more about the models we’re using, you can find them by their Hugging Face model cards, for starters:

https://huggingface.co/mistralai/Mistral-7B-v0.1 https://huggingface.co/distilbert/distilgpt2 The original text of Shelley’s Frankenstein was obtained via Project Gutenberg: https://www.gutenberg.org/ebooks/84

Steps: Step 1 Execute the installation and config cells under “Setup”. Then, according to the option you selected in the Setup tab of this project, make sure the correct model is being assigned to model_name. If you’re able to use a GPU, whether locally or via a cloud provider like Colab, this should be mistral7b. If you’re only able to use a CPU, this should say gpt2. Consult the Setup tab on the left if you have any questions. These are already filled in for you. Just double check to make sure you’re using the right notebook for your use case. If you’re using a CPU, it should say gpt2. If you’re using a GPU, it should say mistral7b.

Step 2 First, execute the first three cells of the “EDA” section. Inspect their outputs to learn more about the dataset.

Then, in the lines under # STEP 2. in the fourth cell of EDA, convert train_df and test_df to Hugging Face Datasets and assign them to train_dataset and test_dataset. The method to convert Pandas DataFrames to Hugging Face Datasets is Dataset.from_pandas(your_dataframe).

Step 3 If you’re using Mistral 7B: Scroll to the “Model Import and Tokenization” section. We’ll now define the configuration for the quantization. We’ll once again use 4bit quantization, and continue using bitsandbytes‘s double quantization method. Use the normalized float 4bit data type abbreviated "nf4" for the bnb_4bit_quant_type and specify compute_dtype to be torch.bfloat16 to temporarily dequantize the model weights to Brain Float 16 precision (torch.bfloat16) when performing computations.

Next, load in the model, passing in the predefined model name and the quantization_config with the values you just defined. Then check what device the model is running on.

Be advised: the model will take a few minutes to download.

If you’re using GPT-2: Next, load in the model, passing in the predefined model name in the first cell of “Model Import.” Next you’ll see some code that checks if you’re using a Macbook M1 chip, in which case we’ll set device as MPS, a specialized platform for using the M1. Finally, check what device the model is running on in the final line of the “Model Import” cell. If you’re using Mistral 7B: Set load_in_4bit to be True, and bnb_4bit_use_double_quant also to True. Since we want normalized floats as the quantized weights, set bnb_4bit_quant_type to "nf4" and finally set bnb_4bit_compute_dtype to Brain Float 16-bit or torch.bfloat16 to use that precision during computation.

Use the .from_pretrained() method of AutoModelForCausalLM to load in the model_name, and add a second, named argument of quantization_config set to the variable you used to store the BitsAndBytesConfig in the previous cell.

To see what device you’re using, look at the attribute .device of model.

If you’re using GPT-2: Use the .from_pretrained() method of AutoModelForCausalLM to load in the model_name. To see what device you’re using, look at the attribute .device of model.

Step 4 For Mistral 7B: Prepare the model for QLoRA by passing it through the required peft function in the last cell of the “Model import” section on the line immediately under the comment announcing # STEP 4.

Now we’ll configure LoRA for the finetuning run. Decent starting values for this notebook have been a rank of 32, an alpha value of 64, and the dropout set to 0.05. However, you’ll gain valuable experience if you experiment with tweaking these values after completing the project.

In the lessons, our task_type was sentiment classification, but in this project we’re generating text. Language models that generate text are commonly called “causal” models. Pass "CAUSAL_LM" to task_type.

For both Mistral7B and GPT-2: Tokenize the train and test sets. Remember that the first argument should be a function that takes in a string and outputs a sequence of tokens. The second, named argument should be batched=True.

In the exercises, we defined the function above the place where we tokenized our data, but you can also pass a lambda function as this argument, for instance: lambda examples: tokenizer(examples['text']) for both training and test datasets. For Mistral 7B: The function we need to pass model through is prepare_model_for_kbit_training().

Pass each of the hyperparameter settings we specified into their respective named arguments of LoraConfig().

For both Mistral 7B and GPT-2: Here’s an example snippet you could use to tokenize the training dataset:

tokenized_train_dataset = train_dataset.map(lambda examples: tokenizer(examples["text"], padding="longest", truncation=True), batched=True).

Do the same thing for the test data and you’re good to go.

Step 5 Generate a completion with the base model for informal evaluation. Try writing the first sentence of a Frankenstein-inspired fanfiction and see how it continues your work. Got writer’s block? Pass "I'm afraid I've created a " as the prompt to generate_text(). Pass a string with your prompt to the generate_text() function in the second cell of “Base model evaluation.”

Step 6 Complete the function to calculate perplexity. Most of this function is already written for you, but we’ve left the final step in the perplexity equation for you to complete. Remember: perplexity is defined as the exponentiated cross-entropy loss of a sequence of predictions. Emphasis on exponentiated. To exponentiate a value (in our case, loss), use torch.exp().

Step 7 Configure the training arguments. Use the values specified below to start–we’ve discovered they produce good results after trial and error. It will behoove you to experiment with these values when we finish the project, though. A large component of ML engineering is tweaking and experimenting with these settings.

Some decent initial values:

2 examples per batch (per_device_train_batch_size) 2 epochs a learning rate of 2e-5 For the optimizer, if you’re using the GPU try "paged_adamw_8bit". The AdamW optimizer adjusts its learning rate intelligently during training. It’s designed to be memory efficient (the paged part) and to work with quantized models (the 8bit part.) If you’re using the CPU, try "adamw_hf" Set per_device_train_batch_size to 2, num_train_epochs to 2, learning_rate at 2e-5 and the optimizer optim as "paged_adamw_8bit" if you’re using the GPU, else "adamw_hf".

Step 8 Train the model using the Trainer API’s method for training on the last line of the first cell in the “Training” section.

If you selected Mistral 7B for your model, go take a break and leave your computer on. You should see the ETA printed out at the top of the output of the “Training” cell. Could take anywhere from 10m to 40m depending on how you tweaked the hyperparameters. (Lower the num_epochs to 1 and raise the batch size if you don’t want it to take so long, though be advised that performance might suffer.)

If you’re using Distil GPT-2, this shouldn’t take quite as long, though you should still be prepared to wait. All you need to do is call a method on the trainer you’ve just instantiated. It’s .train().

Step 9 Generate a completion with the finetuned model using the same prompt you passed to the base model. Then compare it to the base model’s output in the first cell of “Evaluating the finetuned model.” Call the generate_text() function again on the same string you fed the base model.

Step 10 Calculate the finetuned model’s perplexity and compare it to the base model’s. Pass the model through the calc_perplexity function in the second cell of “Evaluating the finetuned model.”

solutions are also added under FrankensteinGPU.ipynb and FrankensteinMPSCPU.ipynb