Update fine_tune_gpt2_medquad.py
Browse files- fine_tune_gpt2_medquad.py +2 -12
fine_tune_gpt2_medquad.py
CHANGED
|
@@ -1,13 +1,8 @@
|
|
| 1 |
-
It looks like the `marianeft/MedQuAD` model you're trying to load does not have the required files (`pytorch_model.bin`, `model.safetensors`, `tf_model.h5`, `model.ckpt` or `flax_model.msgpack`). To address this, let's use a different approach.
|
| 2 |
-
|
| 3 |
-
Instead of saving the fine-tuned model to the `marianeft/MedQuAD` directory, let's save it to a new directory. Here’s the modified code to handle this issue:
|
| 4 |
-
|
| 5 |
-
```python
|
| 6 |
import pandas as pd
|
| 7 |
from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
|
| 8 |
from datasets import load_dataset
|
| 9 |
|
| 10 |
-
# Load MedQuAD dataset
|
| 11 |
dataset = load_dataset("marianeft/MedQuAD", split="train")
|
| 12 |
|
| 13 |
# Load the GPT-2 model and tokenizer
|
|
@@ -49,9 +44,4 @@ trainer.train()
|
|
| 49 |
|
| 50 |
# Save the model to a new directory
|
| 51 |
model.save_pretrained("fine_tuned_medquad")
|
| 52 |
-
tokenizer.save_pretrained("fine_tuned_medquad")
|
| 53 |
-
```
|
| 54 |
-
|
| 55 |
-
This code will save your fine-tuned model and tokenizer to the "fine_tuned_medquad" directory instead of "marianeft/MedQuAD". This way, you avoid trying to save the model to the original model directory which caused the issue.
|
| 56 |
-
|
| 57 |
-
Give this a try and let me know if it works for you!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
import pandas as pd
|
| 2 |
from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
|
| 3 |
from datasets import load_dataset
|
| 4 |
|
| 5 |
+
# Load MedQuAD dataset
|
| 6 |
dataset = load_dataset("marianeft/MedQuAD", split="train")
|
| 7 |
|
| 8 |
# Load the GPT-2 model and tokenizer
|
|
|
|
| 44 |
|
| 45 |
# Save the model to a new directory
|
| 46 |
model.save_pretrained("fine_tuned_medquad")
|
| 47 |
+
tokenizer.save_pretrained("fine_tuned_medquad")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|