|
|
--- |
|
|
tags: |
|
|
- generated_from_trainer |
|
|
model-index: |
|
|
- name: multi-label-class-classification-on-github-issues |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
|
|
# multi-label-class-classification-on-github-issues |
|
|
|
|
|
This model is a fine-tuned version of [neuralmagic/oBERT-12-upstream-pruned-unstructured-97](https://huggingface.co/neuralmagic/oBERT-12-upstream-pruned-unstructured-97) on the None dataset. |
|
|
It achieves the following results on the evaluation set: |
|
|
- Loss: 0.1301 |
|
|
- Micro f1: 0.5159 |
|
|
- Macro f1: 0.0352 |
|
|
|
|
|
## Model description |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Training and evaluation data |
|
|
|
|
|
More information needed |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
### Training hyperparameters |
|
|
|
|
|
The following hyperparameters were used during training: |
|
|
- learning_rate: 3e-05 |
|
|
- train_batch_size: 64 |
|
|
- eval_batch_size: 8 |
|
|
- seed: 42 |
|
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
|
- lr_scheduler_type: linear |
|
|
- num_epochs: 15 |
|
|
- mixed_precision_training: Native AMP |
|
|
|
|
|
### Training results |
|
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Micro f1 | Macro f1 | |
|
|
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:| |
|
|
| No log | 1.0 | 25 | 0.3994 | 0.3783 | 0.0172 | |
|
|
| No log | 2.0 | 50 | 0.2846 | 0.3791 | 0.0172 | |
|
|
| No log | 3.0 | 75 | 0.2159 | 0.3791 | 0.0172 | |
|
|
| No log | 4.0 | 100 | 0.1802 | 0.3791 | 0.0172 | |
|
|
| No log | 5.0 | 125 | 0.1618 | 0.3791 | 0.0172 | |
|
|
| No log | 6.0 | 150 | 0.1515 | 0.3791 | 0.0172 | |
|
|
| No log | 7.0 | 175 | 0.1452 | 0.3791 | 0.0172 | |
|
|
| No log | 8.0 | 200 | 0.1411 | 0.3931 | 0.0202 | |
|
|
| No log | 9.0 | 225 | 0.1379 | 0.4413 | 0.0277 | |
|
|
| No log | 10.0 | 250 | 0.1350 | 0.4694 | 0.0309 | |
|
|
| No log | 11.0 | 275 | 0.1327 | 0.4993 | 0.0336 | |
|
|
| No log | 12.0 | 300 | 0.1309 | 0.5084 | 0.0344 | |
|
|
| No log | 13.0 | 325 | 0.1297 | 0.5147 | 0.0349 | |
|
|
| No log | 14.0 | 350 | 0.1291 | 0.5060 | 0.0343 | |
|
|
| No log | 15.0 | 375 | 0.1287 | 0.5107 | 0.0346 | |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.25.1 |
|
|
- Pytorch 1.13.0+cu116 |
|
|
- Datasets 2.8.0 |
|
|
- Tokenizers 0.13.2 |
|
|
# Day 1 |
|
|
|
|
|
1. Tried to use the Neural Magic Model "neuralmagic/oBERT-12-upstream-pruned-unstructured-97". The macro and micro f1 scores were much smaller at the |
|
|
beginning of the model; the initial step did not increase much. However, it did outperform in the same epoch by .159 difference in the f1 score. |
|
|
2. Modification of the code was more significant was able to add errors in my program to move to the CPU if there was an error in my program |
|
|
``` Python |
|
|
import gc |
|
|
''' |
|
|
Try and Catch the block when training the model using more memory than the GPU, it will produce an error. |
|
|
1. Check the Amount of GPU memory used |
|
|
2. Move the model to the CPU |
|
|
3. Call the garbage collector |
|
|
4. Free the GPU memory in the cache |
|
|
5. Check the amount of GPU memory used to see if it is freed |
|
|
''' |
|
|
def check_gpu_memory(): |
|
|
print(torch.cuda.memory_allocated()/1e9) |
|
|
return torch.cuda.memory_allocated()/1e9 |
|
|
try: |
|
|
trainer.train() |
|
|
except RuntimeError as e: |
|
|
if "CUDA out of memory" in str(e): |
|
|
print("CUDA out of memory") |
|
|
print("Let's free some GPU memory and re-allocate") |
|
|
check_gpu_memory() |
|
|
## Move the model to CPU |
|
|
model.to("cpu") |
|
|
gc.collect() |
|
|
## Free the GPU memory |
|
|
torch.cuda.empty_cache() |
|
|
check_gpu_memory() |
|
|
else: |
|
|
raise e |
|
|
``` |
|
|
4. Able to check if there was a number of support my model can support in my model |
|
|
``` Python |
|
|
from transformers import Trainer, TrainingArguments |
|
|
def is_on_colab(): |
|
|
if 'google.colab' in sys.modules: |
|
|
return True |
|
|
return False |
|
|
|
|
|
training_args_fine_tune = TrainingArguments( |
|
|
output_dir = "./multi-label-class-classification-on-github-issues" , |
|
|
num_train_epochs = 15, |
|
|
learning_rate = 3e-5, |
|
|
per_device_train_batch_size = 64 , |
|
|
evaluation_strategy = "epoch" , |
|
|
save_strategy="epoch" , |
|
|
load_best_model_at_end=True, |
|
|
metric_for_best_model='micro f1', |
|
|
save_total_limit=1, |
|
|
log_level='error', |
|
|
push_to_hub = True if is_on_colab else False , |
|
|
) |
|
|
if torch.cuda.is_available(): |
|
|
## check if the Cuda GPU can bfloat16 |
|
|
if torch.cuda.is_bf16_supported(): |
|
|
print("Cuda GPU can support bfloat16") |
|
|
training_args_fine_tune.fp16 = True |
|
|
else: |
|
|
print("Cuda GPU cannot support bfloat16 so instead we will use float16 ") |
|
|
training_args_fine_tune.fp16 = True |
|
|
``` |
