GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Downloading and preparing dataset json/default to /home/ace14459tv/t5maru/cache/json/default-508436ce1b237537/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...

Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 8371.86it/s]

Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 407.97it/s]

Generating train split: 0 examples [00:00, ? examples/s]
                                                        
Dataset json downloaded and prepared to /home/ace14459tv/t5maru/cache/json/default-508436ce1b237537/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.

Map (num_proc=4):   0%|          | 0/2880 [00:00<?, ? examples/s]
Map (num_proc=4):   1%|▏         | 38/2880 [00:00<00:43, 64.61 examples/s]
Map (num_proc=4):   7%|▋         | 215/2880 [00:00<00:07, 361.81 examples/s]
Map (num_proc=4):  14%|█▍        | 396/2880 [00:00<00:04, 593.37 examples/s]
Map (num_proc=4):  18%|█▊        | 511/2880 [00:00<00:03, 707.90 examples/s]
Map (num_proc=4):  26%|██▌       | 744/2880 [00:01<00:02, 1067.25 examples/s]
Map (num_proc=4):  33%|███▎      | 950/2880 [00:01<00:01, 1230.14 examples/s]
Map (num_proc=4):  39%|███▉      | 1128/2880 [00:01<00:01, 1302.62 examples/s]
Map (num_proc=4):  45%|████▌     | 1308/2880 [00:01<00:01, 1265.23 examples/s]
Map (num_proc=4):  53%|█████▎    | 1536/2880 [00:01<00:00, 1508.47 examples/s]
Map (num_proc=4):  60%|██████    | 1731/2880 [00:01<00:00, 1575.00 examples/s]
Map (num_proc=4):  66%|██████▋   | 1912/2880 [00:01<00:00, 1463.51 examples/s]
Map (num_proc=4):  74%|███████▍  | 2145/2880 [00:01<00:00, 1540.43 examples/s]
Map (num_proc=4):  83%|████████▎ | 2386/2880 [00:02<00:00, 1751.55 examples/s]
Map (num_proc=4):  90%|█████████ | 2601/2880 [00:02<00:00, 1469.03 examples/s]
Map (num_proc=4):  97%|█████████▋| 2785/2880 [00:02<00:00, 1390.78 examples/s]
                                                                              
Downloading and preparing dataset json/default to /home/ace14459tv/t5maru/cache/json/default-85f4bdc1e5b3a604/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...

Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 9962.72it/s]

Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 483.05it/s]

Generating train split: 0 examples [00:00, ? examples/s]
                                                        
Dataset json downloaded and prepared to /home/ace14459tv/t5maru/cache/json/default-85f4bdc1e5b3a604/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.

Map (num_proc=4):   0%|          | 0/618 [00:00<?, ? examples/s]
Map (num_proc=4):   8%|▊         | 49/618 [00:00<00:06, 85.68 examples/s]
Map (num_proc=4):  25%|██▌       | 155/618 [00:00<00:01, 277.60 examples/s]
Map (num_proc=4):  42%|████▏     | 262/618 [00:01<00:01, 274.08 examples/s]
Map (num_proc=4):  58%|█████▊    | 359/618 [00:01<00:00, 261.13 examples/s]
Map (num_proc=4):  75%|███████▌  | 464/618 [00:01<00:00, 356.96 examples/s]
Map (num_proc=4):  92%|█████████▏| 571/618 [00:01<00:00, 329.51 examples/s]
                                                                           
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name  | Type            | Params
------------------------------------------
0 | model | OptimizedModule | 222 M 
------------------------------------------
222 M     Trainable params
0         Non-trainable params
222 M     Total params
891.528   Total estimated model params size (MB)
[2023-12-21 03:07:16,808] torch._inductor.utils: [WARNING] using triton random, expect difference from eager
Metric val_loss improved. New best score: 0.060
Metric val_loss improved by 0.034 >= min_delta = 0.0. New best score: 0.027
Metric val_loss improved by 0.004 >= min_delta = 0.0. New best score: 0.022
Metric val_loss improved by 0.002 >= min_delta = 0.0. New best score: 0.020
Metric val_loss improved by 0.001 >= min_delta = 0.0. New best score: 0.019
Monitored metric val_loss did not improve in the last 3 records. Best score: 0.019. Signaling Trainer to stop.
{"log": "trained", "date": "2023-12-21T03:06:16", "elapsed": "00:06:28", "model": "Salesforce/codet5-base", "max_length": 324, "target_max_length": 324, "batch_size": 32, "gradient_accumulation_steps": 1, "train_steps": 2700, "accelerator": "gpu", "devices": "auto", "precision": 32, "strategy": "auto", "gradient_clip_val": 1.0, "compile": true, "solver": "adamw", "lr": 0.0003, "warmup_steps": 1, "training_steps": 100000, "adam_epsilon": 1e-08, "weight_decay": 0.0, "epoch": 9, "step": 810, "saved": "error_codet5_base_324"}
Traceback (most recent call last):
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 259, in hf_raise_for_status
    response.raise_for_status()
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/error_codet5_base_512/resolve/main/tokenizer_config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/utils/hub.py", line 409, in cached_file
    resolved_file = hf_hub_download(
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download
    metadata = get_hf_file_metadata(
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1541, in get_hf_file_metadata
    hf_raise_for_status(r)
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 291, in hf_raise_for_status
    raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-65832ea4-32e585824a1465906a86bc9c)

Repository Not Found for url: https://huggingface.co/error_codet5_base_512/resolve/main/tokenizer_config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ace14459tv/cuda117/bin/t5test", line 8, in <module>
    sys.exit(main_test())
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/t5maru/t5tune.py", line 612, in main_test
    model = T5Model(
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/t5maru/t5tune.py", line 227, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 642, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 486, in get_tokenizer_config
    resolved_config_file = cached_file(
  File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/utils/hub.py", line 424, in cached_file
    raise EnvironmentError(
OSError: error_codet5_base_512 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.