marianeft commited on
Commit
030c7b9
·
verified ·
1 Parent(s): 230bb4c

Update fine_tune_gpt2_medquad.py

Browse files
Files changed (1) hide show
  1. fine_tune_gpt2_medquad.py +2 -12
fine_tune_gpt2_medquad.py CHANGED
@@ -1,13 +1,8 @@
1
- It looks like the `marianeft/MedQuAD` model you're trying to load does not have the required files (`pytorch_model.bin`, `model.safetensors`, `tf_model.h5`, `model.ckpt` or `flax_model.msgpack`). To address this, let's use a different approach.
2
-
3
- Instead of saving the fine-tuned model to the `marianeft/MedQuAD` directory, let's save it to a new directory. Here’s the modified code to handle this issue:
4
-
5
- ```python
6
  import pandas as pd
7
  from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
8
  from datasets import load_dataset
9
 
10
- # Load MedQuAD dataset properly
11
  dataset = load_dataset("marianeft/MedQuAD", split="train")
12
 
13
  # Load the GPT-2 model and tokenizer
@@ -49,9 +44,4 @@ trainer.train()
49
 
50
  # Save the model to a new directory
51
  model.save_pretrained("fine_tuned_medquad")
52
- tokenizer.save_pretrained("fine_tuned_medquad")
53
- ```
54
-
55
- This code will save your fine-tuned model and tokenizer to the "fine_tuned_medquad" directory instead of "marianeft/MedQuAD". This way, you avoid trying to save the model to the original model directory which caused the issue.
56
-
57
- Give this a try and let me know if it works for you!
 
 
 
 
 
 
1
  import pandas as pd
2
  from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
3
  from datasets import load_dataset
4
 
5
+ # Load MedQuAD dataset
6
  dataset = load_dataset("marianeft/MedQuAD", split="train")
7
 
8
  # Load the GPT-2 model and tokenizer
 
44
 
45
  # Save the model to a new directory
46
  model.save_pretrained("fine_tuned_medquad")
47
+ tokenizer.save_pretrained("fine_tuned_medquad")