| GPU available: True (cuda), used: True |
| TPU available: False, using: 0 TPU cores |
| IPU available: False, using: 0 IPUs |
| HPU available: False, using: 0 HPUs |
| Downloading and preparing dataset json/default to /home/ace14459tv/t5maru/cache/json/default-508436ce1b237537/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4... |
|
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 8371.86it/s] |
|
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 407.97it/s] |
|
Generating train split: 0 examples [00:00, ? examples/s]
Dataset json downloaded and prepared to /home/ace14459tv/t5maru/cache/json/default-508436ce1b237537/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data. |
|
Map (num_proc=4): 0%| | 0/2880 [00:00<?, ? examples/s]
Map (num_proc=4): 1%|▏ | 38/2880 [00:00<00:43, 64.61 examples/s]
Map (num_proc=4): 7%|▋ | 215/2880 [00:00<00:07, 361.81 examples/s]
Map (num_proc=4): 14%|█▍ | 396/2880 [00:00<00:04, 593.37 examples/s]
Map (num_proc=4): 18%|█▊ | 511/2880 [00:00<00:03, 707.90 examples/s]
Map (num_proc=4): 26%|██▌ | 744/2880 [00:01<00:02, 1067.25 examples/s]
Map (num_proc=4): 33%|███▎ | 950/2880 [00:01<00:01, 1230.14 examples/s]
Map (num_proc=4): 39%|███▉ | 1128/2880 [00:01<00:01, 1302.62 examples/s]
Map (num_proc=4): 45%|████▌ | 1308/2880 [00:01<00:01, 1265.23 examples/s]
Map (num_proc=4): 53%|█████▎ | 1536/2880 [00:01<00:00, 1508.47 examples/s]
Map (num_proc=4): 60%|██████ | 1731/2880 [00:01<00:00, 1575.00 examples/s]
Map (num_proc=4): 66%|██████▋ | 1912/2880 [00:01<00:00, 1463.51 examples/s]
Map (num_proc=4): 74%|███████▍ | 2145/2880 [00:01<00:00, 1540.43 examples/s]
Map (num_proc=4): 83%|████████▎ | 2386/2880 [00:02<00:00, 1751.55 examples/s]
Map (num_proc=4): 90%|█████████ | 2601/2880 [00:02<00:00, 1469.03 examples/s]
Map (num_proc=4): 97%|█████████▋| 2785/2880 [00:02<00:00, 1390.78 examples/s]
Downloading and preparing dataset json/default to /home/ace14459tv/t5maru/cache/json/default-85f4bdc1e5b3a604/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4... |
|
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 9962.72it/s] |
|
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 483.05it/s] |
|
Generating train split: 0 examples [00:00, ? examples/s]
Dataset json downloaded and prepared to /home/ace14459tv/t5maru/cache/json/default-85f4bdc1e5b3a604/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data. |
|
Map (num_proc=4): 0%| | 0/618 [00:00<?, ? examples/s]
Map (num_proc=4): 8%|▊ | 49/618 [00:00<00:06, 85.68 examples/s]
Map (num_proc=4): 25%|██▌ | 155/618 [00:00<00:01, 277.60 examples/s]
Map (num_proc=4): 42%|████▏ | 262/618 [00:01<00:01, 274.08 examples/s]
Map (num_proc=4): 58%|█████▊ | 359/618 [00:01<00:00, 261.13 examples/s]
Map (num_proc=4): 75%|███████▌ | 464/618 [00:01<00:00, 356.96 examples/s]
Map (num_proc=4): 92%|█████████▏| 571/618 [00:01<00:00, 329.51 examples/s]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
|
|
| | Name | Type | Params |
| ------------------------------------------ |
| 0 | model | OptimizedModule | 222 M |
| ------------------------------------------ |
| 222 M Trainable params |
| 0 Non-trainable params |
| 222 M Total params |
| 891.528 Total estimated model params size (MB) |
| [2023-12-21 03:07:16,808] torch._inductor.utils: [WARNING] using triton random, expect difference from eager |
| Metric val_loss improved. New best score: 0.060 |
| Metric val_loss improved by 0.034 >= min_delta = 0.0. New best score: 0.027 |
| Metric val_loss improved by 0.004 >= min_delta = 0.0. New best score: 0.022 |
| Metric val_loss improved by 0.002 >= min_delta = 0.0. New best score: 0.020 |
| Metric val_loss improved by 0.001 >= min_delta = 0.0. New best score: 0.019 |
| Monitored metric val_loss did not improve in the last 3 records. Best score: 0.019. Signaling Trainer to stop. |
| {"log": "trained", "date": "2023-12-21T03:06:16", "elapsed": "00:06:28", "model": "Salesforce/codet5-base", "max_length": 324, "target_max_length": 324, "batch_size": 32, "gradient_accumulation_steps": 1, "train_steps": 2700, "accelerator": "gpu", "devices": "auto", "precision": 32, "strategy": "auto", "gradient_clip_val": 1.0, "compile": true, "solver": "adamw", "lr": 0.0003, "warmup_steps": 1, "training_steps": 100000, "adam_epsilon": 1e-08, "weight_decay": 0.0, "epoch": 9, "step": 810, "saved": "error_codet5_base_324"} |
| Traceback (most recent call last): |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 259, in hf_raise_for_status |
| response.raise_for_status() |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status |
| raise HTTPError(http_error_msg, response=self) |
| requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/error_codet5_base_512/resolve/main/tokenizer_config.json |
| |
| The above exception was the direct cause of the following exception: |
| |
| Traceback (most recent call last): |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/utils/hub.py", line 409, in cached_file |
| resolved_file = hf_hub_download( |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn |
| return fn(*args, **kwargs) |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download |
| metadata = get_hf_file_metadata( |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn |
| return fn(*args, **kwargs) |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1541, in get_hf_file_metadata |
| hf_raise_for_status(r) |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 291, in hf_raise_for_status |
| raise RepositoryNotFoundError(message, response) from e |
| huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-65832ea4-32e585824a1465906a86bc9c) |
| |
| Repository Not Found for url: https://huggingface.co/error_codet5_base_512/resolve/main/tokenizer_config.json. |
| Please make sure you specified the correct `repo_id` and `repo_type`. |
| If you are trying to access a private or gated repo, make sure you are authenticated. |
| |
| During handling of the above exception, another exception occurred: |
| |
| Traceback (most recent call last): |
| File "/home/ace14459tv/cuda117/bin/t5test", line 8, in <module> |
| sys.exit(main_test()) |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/t5maru/t5tune.py", line 612, in main_test |
| model = T5Model( |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/t5maru/t5tune.py", line 227, in __init__ |
| self.tokenizer = AutoTokenizer.from_pretrained( |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 642, in from_pretrained |
| tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs) |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 486, in get_tokenizer_config |
| resolved_config_file = cached_file( |
| File "/home/ace14459tv/cuda117/lib/python3.10/site-packages/transformers/utils/hub.py", line 424, in cached_file |
| raise EnvironmentError( |
| OSError: error_codet5_base_512 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' |
| If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`. |
| |
