Update fine_tune_gpt2_medquad.py
Browse files- fine_tune_gpt2_medquad.py +13 -3
fine_tune_gpt2_medquad.py
CHANGED
|
@@ -1,3 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
import pandas as pd
|
| 2 |
from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
|
| 3 |
from datasets import load_dataset
|
|
@@ -42,6 +47,11 @@ trainer = Trainer(
|
|
| 42 |
# Fine-tune the model
|
| 43 |
trainer.train()
|
| 44 |
|
| 45 |
-
# Save the model
|
| 46 |
-
model.save_pretrained("
|
| 47 |
-
tokenizer.save_pretrained("
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
It looks like the `marianeft/MedQuAD` model you're trying to load does not have the required files (`pytorch_model.bin`, `model.safetensors`, `tf_model.h5`, `model.ckpt` or `flax_model.msgpack`). To address this, let's use a different approach.
|
| 2 |
+
|
| 3 |
+
Instead of saving the fine-tuned model to the `marianeft/MedQuAD` directory, let's save it to a new directory. Here’s the modified code to handle this issue:
|
| 4 |
+
|
| 5 |
+
```python
|
| 6 |
import pandas as pd
|
| 7 |
from transformers import GPT2LMHeadModel, GPT2Tokenizer, Trainer, TrainingArguments
|
| 8 |
from datasets import load_dataset
|
|
|
|
| 47 |
# Fine-tune the model
|
| 48 |
trainer.train()
|
| 49 |
|
| 50 |
+
# Save the model to a new directory
|
| 51 |
+
model.save_pretrained("fine_tuned_medquad")
|
| 52 |
+
tokenizer.save_pretrained("fine_tuned_medquad")
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
This code will save your fine-tuned model and tokenizer to the "fine_tuned_medquad" directory instead of "marianeft/MedQuAD". This way, you avoid trying to save the model to the original model directory which caused the issue.
|
| 56 |
+
|
| 57 |
+
Give this a try and let me know if it works for you!
|