Model save

README.md

@@ -0,0 +1,82 @@
+---
+license: apache-2.0
+base_model: openai/whisper-tiny
+tags:
+- generated_from_trainer
+datasets:
+- common_voice_16_0
+metrics:
+- wer
+model-index:
+- name: breeze-dsw-tiny-id
+  results:
+  - task:
+      name: Automatic Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: common_voice_16_0
+      type: common_voice_16_0
+      config: id
+      split: test
+      args: id
+    metrics:
+    - name: Wer
+      type: wer
+      value: 45.243352654338025
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# breeze-dsw-tiny-id
+
+This model is a fine-tuned version of [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny) on the common_voice_16_0 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.7109
+- Wer: 45.2434
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 32
+- eval_batch_size: 16
+- seed: 42
+- distributed_type: multi-GPU
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 500
+- training_steps: 2000
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Wer     |
+|:-------------:|:-----:|:----:|:---------------:|:-------:|
+| 0.99          | 0.1   | 100  | 0.8486          | 54.1724 |
+| 0.7896        | 1.04  | 200  | 0.7578          | 48.3393 |
+| 0.4164        | 1.14  | 300  | 0.7388          | 49.2594 |
+| 0.5456        | 2.09  | 400  | 0.7178          | 46.1266 |
+| 0.476         | 3.03  | 500  | 0.7109          | 45.2434 |
+
+
+### Framework versions
+
+- Transformers 4.37.0.dev0
+- Pytorch 2.1.2+cu121
+- Datasets 2.16.2.dev0
+- Tokenizers 0.15.0

breeze-dsw-tiny-id.log

@@ -1,17 +1,17 @@
-[2024-01-12 16:30:04,760] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
-[2024-01-12 16:30:05,517] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
-[2024-01-12 16:30:05,517] [INFO] [runner.py:571:main] cmd = /home/sp-operator/hf_env/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/run_speech_recognition_seq2seq_streaming.py --deepspeed=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/ds_config.json --model_name_or_path=openai/whisper-tiny --dataset_name=mozilla-foundation/common_voice_16_0 --dataset_config_name=id --language=Indonesian --train_split_name=train --eval_split_name=test --model_index_name=Breeze DSW Indonesian - tiny --max_steps=1000 --output_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id --per_device_train_batch_size=32 --per_device_eval_batch_size=16 --logging_steps=25 --learning_rate=1e-5 --warmup_steps=500 --evaluation_strategy=steps --eval_steps=100 --save_strategy=steps --save_steps=100 --generation_max_length=225 --length_column_name=input_length --max_duration_in_seconds=30 --text_column_name=sentence --freeze_feature_encoder=False --report_to=tensorboard --metric_for_best_model=wer --greater_is_better=False --load_best_model_at_end --gradient_checkpointing --fp16 --overwrite_output_dir --do_train --do_eval --predict_with_generate --do_normalize_eval --streaming --use_auth_token --push_to_hub
-[2024-01-12 16:30:07,172] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
-[2024-01-12 16:30:07,952] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0]}
-[2024-01-12 16:30:07,952] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
-[2024-01-12 16:30:07,952] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
-[2024-01-12 16:30:07,952] [INFO] [launch.py:163:main] dist_world_size=1
-[2024-01-12 16:30:07,952] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0
-[2024-01-12 16:30:10,956] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
-[2024-01-12 16:30:11,158] [INFO] [comm.py:637:init_distributed] cdb=None
-[2024-01-12 16:30:11,158] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
-01/12/2024 16:30:11 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: True
-01/12/2024 16:30:11 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
+[2024-01-12 23:20:02,304] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
+[2024-01-12 23:20:03,384] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
+[2024-01-12 23:20:03,384] [INFO] [runner.py:571:main] cmd = /home/sp-operator/hf_env/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/run_speech_recognition_seq2seq_streaming.py --deepspeed=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/ds_config.json --model_name_or_path=openai/whisper-tiny --dataset_name=mozilla-foundation/common_voice_16_0 --dataset_config_name=id --language=Indonesian --train_split_name=train --eval_split_name=test --model_index_name=Breeze DSW Indonesian - tiny --max_steps=2000 --output_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id --per_device_train_batch_size=32 --per_device_eval_batch_size=16 --logging_steps=25 --learning_rate=1e-5 --warmup_steps=500 --evaluation_strategy=steps --eval_steps=200 --save_strategy=steps --save_steps=200 --generation_max_length=225 --length_column_name=input_length --max_duration_in_seconds=30 --text_column_name=sentence --freeze_feature_encoder=False --report_to=tensorboard --metric_for_best_model=wer --greater_is_better=False --load_best_model_at_end --gradient_checkpointing --fp16 --do_train --do_eval --predict_with_generate --do_normalize_eval --streaming --use_auth_token --push_to_hub
+[2024-01-12 23:20:05,084] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
+[2024-01-12 23:20:05,852] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0]}
+[2024-01-12 23:20:05,852] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
+[2024-01-12 23:20:05,852] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
+[2024-01-12 23:20:05,852] [INFO] [launch.py:163:main] dist_world_size=1
+[2024-01-12 23:20:05,852] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0
+[2024-01-12 23:20:08,880] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
+[2024-01-12 23:20:09,073] [INFO] [comm.py:637:init_distributed] cdb=None
+[2024-01-12 23:20:09,073] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
+01/12/2024 23:20:09 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: True
+01/12/2024 23:20:09 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
 _n_gpu=1,
 adafactor=False,
 adam_beta1=0.9,
@@ -39,7 +39,7 @@ do_predict=False,
 do_train=True,
 eval_accumulation_steps=None,
 eval_delay=0,
-eval_steps=100,
+eval_steps=200,
 evaluation_strategy=steps,
 fp16=True,
 fp16_backend=auto,
@@ -78,7 +78,7 @@ local_rank=0,
 log_level=passive,
 log_level_replica=warning,
 log_on_each_node=True,
-logging_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/runs/Jan12_16-30-10_knight,
+logging_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/runs/Jan12_23-20-08_knight,
 logging_first_step=False,
 logging_nan_inf_filter=True,
 logging_steps=25,
@@ -86,7 +86,7 @@ logging_strategy=steps,
 lr_scheduler_kwargs={},
 lr_scheduler_type=linear,
 max_grad_norm=1.0,
-max_steps=1000,
+max_steps=2000,
 metric_for_best_model=wer,
 mp_parameters=,
 neftune_noise_alpha=None,
@@ -95,7 +95,7 @@ num_train_epochs=3.0,
 optim=adamw_torch,
 optim_args=None,
 output_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id,
-overwrite_output_dir=True,
+overwrite_output_dir=False,
 past_index=-1,
 per_device_eval_batch_size=16,
 per_device_train_batch_size=32,
@@ -113,7 +113,7 @@ run_name=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../bree
 save_on_each_node=False,
 save_only_model=False,
 save_safetensors=True,
-save_steps=100,
+save_steps=200,
 save_strategy=steps,
 save_total_limit=None,
 seed=42,
@@ -135,7 +135,7 @@ warmup_ratio=0.0,
 warmup_steps=500,
 weight_decay=0.0,
 )
-01/12/2024 16:30:11 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
+01/12/2024 23:20:09 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
 _n_gpu=1,
 adafactor=False,
 adam_beta1=0.9,
@@ -163,7 +163,7 @@ do_predict=False,
 do_train=True,
 eval_accumulation_steps=None,
 eval_delay=0,
-eval_steps=100,
+eval_steps=200,
 evaluation_strategy=steps,
 fp16=True,
 fp16_backend=auto,
@@ -202,7 +202,7 @@ local_rank=0,
 log_level=passive,
 log_level_replica=warning,
 log_on_each_node=True,
-logging_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/runs/Jan12_16-30-10_knight,
+logging_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/runs/Jan12_23-20-08_knight,
 logging_first_step=False,
 logging_nan_inf_filter=True,
 logging_steps=25,
@@ -210,7 +210,7 @@ logging_strategy=steps,
 lr_scheduler_kwargs={},
 lr_scheduler_type=linear,
 max_grad_norm=1.0,
-max_steps=1000,
+max_steps=2000,
 metric_for_best_model=wer,
 mp_parameters=,
 neftune_noise_alpha=None,
@@ -219,7 +219,7 @@ num_train_epochs=3.0,
 optim=adamw_torch,
 optim_args=None,
 output_dir=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id,
-overwrite_output_dir=True,
+overwrite_output_dir=False,
 past_index=-1,
 per_device_eval_batch_size=16,
 per_device_train_batch_size=32,
@@ -237,7 +237,7 @@ run_name=/cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../bree
 save_on_each_node=False,
 save_only_model=False,
 save_safetensors=True,
-save_steps=100,
+save_steps=200,
 save_strategy=steps,
 save_total_limit=None,
 seed=42,
@@ -259,41 +259,42 @@ warmup_ratio=0.0,
 warmup_steps=500,
 weight_decay=0.0,
 )
-[2024-01-12 16:30:24,445] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.12.6, git-hash=unknown, git-branch=unknown
-[2024-01-12 16:30:25,203] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
+01/12/2024 23:20:09 - INFO - __main__ - Checkpoint detected, resuming training at /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500. To avoid this behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch.
+[2024-01-12 23:20:24,568] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.12.6, git-hash=unknown, git-branch=unknown
+[2024-01-12 23:20:25,411] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
 Installed CUDA version 12.3 does not match the version torch was compiled with 12.1 but since the APIs are compatible, accepting this combination
 [1/4] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/sp-operator/hf_env/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/TH -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_61,code=compute_61 -c /home/sp-operator/hf_env/lib/python3.10/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o 
 [2/4] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/sp-operator/hf_env/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/TH -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/local/cuda/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -c /home/sp-operator/hf_env/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o 
 [3/4] c++ -MMD -MF cpu_adam_impl.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/sp-operator/hf_env/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/TH -isystem /home/sp-operator/hf_env/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/local/cuda/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -c /home/sp-operator/hf_env/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam_impl.cpp -o cpu_adam_impl.o 
 [4/4] c++ cpu_adam.o cpu_adam_impl.o custom_cuda_kernel.cuda.o -shared -lcurand -L/home/sp-operator/hf_env/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o cpu_adam.so
-Time to load cpu_adam op: 27.245380878448486 seconds
+Time to load cpu_adam op: 27.198795795440674 seconds
 Adam Optimizer #0 is created with AVX2 arithmetic capability.
 Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
-[2024-01-12 16:30:54,187] [INFO] [logging.py:96:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
-[2024-01-12 16:30:54,187] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
-[2024-01-12 16:30:54,191] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
-[2024-01-12 16:30:54,191] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
-[2024-01-12 16:30:54,191] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.float16 ZeRO stage 2 optimizer
-[2024-01-12 16:30:54,192] [INFO] [stage_1_and_2.py:148:__init__] Reduce bucket size 200000000
-[2024-01-12 16:30:54,192] [INFO] [stage_1_and_2.py:149:__init__] Allgather bucket size 200000000
-[2024-01-12 16:30:54,192] [INFO] [stage_1_and_2.py:150:__init__] CPU Offload: True
-[2024-01-12 16:30:54,192] [INFO] [stage_1_and_2.py:151:__init__] Round robin gradient partitioning: False
-[2024-01-12 16:30:54,469] [INFO] [utils.py:791:see_memory_usage] Before initializing optimizer states
-[2024-01-12 16:30:54,470] [INFO] [utils.py:792:see_memory_usage] MA 0.11 GB         Max_MA 0.11 GB         CA 0.13 GB         Max_CA 0 GB 
-[2024-01-12 16:30:54,470] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 3.57 GB, percent = 5.7%
-[2024-01-12 16:30:54,736] [INFO] [utils.py:791:see_memory_usage] After initializing optimizer states
-[2024-01-12 16:30:54,737] [INFO] [utils.py:792:see_memory_usage] MA 0.11 GB         Max_MA 0.11 GB         CA 0.13 GB         Max_CA 0 GB 
-[2024-01-12 16:30:54,737] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 4.1 GB, percent = 6.5%
-[2024-01-12 16:30:54,737] [INFO] [stage_1_and_2.py:516:__init__] optimizer state initialized
-[2024-01-12 16:30:54,814] [INFO] [utils.py:791:see_memory_usage] After initializing ZeRO optimizer
-[2024-01-12 16:30:54,815] [INFO] [utils.py:792:see_memory_usage] MA 0.11 GB         Max_MA 0.11 GB         CA 0.13 GB         Max_CA 0 GB 
-[2024-01-12 16:30:54,815] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 4.1 GB, percent = 6.5%
-[2024-01-12 16:30:54,819] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
-[2024-01-12 16:30:54,819] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupDecayLR
-[2024-01-12 16:30:54,819] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupDecayLR object at 0x7fbfbe6be260>
-[2024-01-12 16:30:54,819] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[[0.9, 0.999]]
-[2024-01-12 16:30:54,820] [INFO] [config.py:984:print] DeepSpeedEngine configuration:
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   activation_checkpointing_config  {
+[2024-01-12 23:20:54,305] [INFO] [logging.py:96:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
+[2024-01-12 23:20:54,305] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
+[2024-01-12 23:20:54,310] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
+[2024-01-12 23:20:54,310] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
+[2024-01-12 23:20:54,310] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.float16 ZeRO stage 2 optimizer
+[2024-01-12 23:20:54,310] [INFO] [stage_1_and_2.py:148:__init__] Reduce bucket size 200000000
+[2024-01-12 23:20:54,310] [INFO] [stage_1_and_2.py:149:__init__] Allgather bucket size 200000000
+[2024-01-12 23:20:54,310] [INFO] [stage_1_and_2.py:150:__init__] CPU Offload: True
+[2024-01-12 23:20:54,311] [INFO] [stage_1_and_2.py:151:__init__] Round robin gradient partitioning: False
+[2024-01-12 23:20:54,579] [INFO] [utils.py:791:see_memory_usage] Before initializing optimizer states
+[2024-01-12 23:20:54,580] [INFO] [utils.py:792:see_memory_usage] MA 0.11 GB         Max_MA 0.11 GB         CA 0.13 GB         Max_CA 0 GB 
+[2024-01-12 23:20:54,580] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 3.56 GB, percent = 5.7%
+[2024-01-12 23:20:54,841] [INFO] [utils.py:791:see_memory_usage] After initializing optimizer states
+[2024-01-12 23:20:54,841] [INFO] [utils.py:792:see_memory_usage] MA 0.11 GB         Max_MA 0.11 GB         CA 0.13 GB         Max_CA 0 GB 
+[2024-01-12 23:20:54,841] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 4.09 GB, percent = 6.5%
+[2024-01-12 23:20:54,841] [INFO] [stage_1_and_2.py:516:__init__] optimizer state initialized
+[2024-01-12 23:20:54,919] [INFO] [utils.py:791:see_memory_usage] After initializing ZeRO optimizer
+[2024-01-12 23:20:54,920] [INFO] [utils.py:792:see_memory_usage] MA 0.11 GB         Max_MA 0.11 GB         CA 0.13 GB         Max_CA 0 GB 
+[2024-01-12 23:20:54,920] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 4.09 GB, percent = 6.5%
+[2024-01-12 23:20:54,924] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
+[2024-01-12 23:20:54,924] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupDecayLR
+[2024-01-12 23:20:54,924] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupDecayLR object at 0x7f0d093ad6f0>
+[2024-01-12 23:20:54,924] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[[0.9, 0.999]]
+[2024-01-12 23:20:54,925] [INFO] [config.py:984:print] DeepSpeedEngine configuration:
+[2024-01-12 23:20:54,925] [INFO] [config.py:988:print]   activation_checkpointing_config  {
     "partition_activations": false, 
     "contiguous_memory_optimization": false, 
     "cpu_checkpointing": false, 
@@ -301,10 +302,10 @@ Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_
     "synchronize_checkpoint_boundary": false, 
     "profile": false
 }
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   amp_enabled .................. False
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   amp_params ................... False
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   autotuning_config ............ {
+[2024-01-12 23:20:54,925] [INFO] [config.py:988:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
+[2024-01-12 23:20:54,925] [INFO] [config.py:988:print]   amp_enabled .................. False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   amp_params ................... False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   autotuning_config ............ {
     "enabled": false, 
     "start_step": null, 
     "end_step": null, 
@@ -329,31 +330,31 @@ Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_
     "min_train_micro_batch_size_per_gpu": 1, 
     "num_tuning_micro_batch_sizes": 3
 }
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   bfloat16_enabled ............. False
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   checkpoint_parallel_write_pipeline  False
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   checkpoint_tag_validation_enabled  True
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   checkpoint_tag_validation_fail  False
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fbfe623d510>
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   communication_data_type ...... None
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
-[2024-01-12 16:30:54,820] [INFO] [config.py:988:print]   curriculum_enabled_legacy .... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   curriculum_params_legacy ..... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   data_efficiency_enabled ...... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   dataloader_drop_last ......... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   disable_allgather ............ False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   dump_state ................... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   dynamic_loss_scale_args ...... {'init_scale': 65536, 'scale_window': 1000, 'delayed_shift': 2, 'consecutive_hysteresis': False, 'min_scale': 1}
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_enabled ........... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_gas_boundary_resolution  1
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_layer_name ........ bert.encoder.layer
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_layer_num ......... 0
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_max_iter .......... 100
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_stability ......... 1e-06
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_tol ............... 0.01
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   eigenvalue_verbose ........... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   elasticity_enabled ........... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   flops_profiler_config ........ {
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   bfloat16_enabled ............. False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   checkpoint_parallel_write_pipeline  False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   checkpoint_tag_validation_enabled  True
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   checkpoint_tag_validation_fail  False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f0d08915ea0>
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   communication_data_type ...... None
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   curriculum_enabled_legacy .... False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   curriculum_params_legacy ..... False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   data_efficiency_enabled ...... False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   dataloader_drop_last ......... False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   disable_allgather ............ False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   dump_state ................... False
+[2024-01-12 23:20:54,926] [INFO] [config.py:988:print]   dynamic_loss_scale_args ...... {'init_scale': 65536, 'scale_window': 1000, 'delayed_shift': 2, 'consecutive_hysteresis': False, 'min_scale': 1}
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_enabled ........... False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_gas_boundary_resolution  1
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_layer_name ........ bert.encoder.layer
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_layer_num ......... 0
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_max_iter .......... 100
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_stability ......... 1e-06
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_tol ............... 0.01
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   eigenvalue_verbose ........... False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   elasticity_enabled ........... False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   flops_profiler_config ........ {
     "enabled": false, 
     "recompute_fwd_factor": 0.0, 
     "profile_step": 1, 
@@ -362,24 +363,24 @@ Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_
     "detailed": true, 
     "output_file": null
 }
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   fp16_auto_cast ............... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   fp16_enabled ................. True
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   fp16_master_weights_and_gradients  False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   global_rank .................. 0
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   grad_accum_dtype ............. None
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   gradient_accumulation_steps .. 1
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   gradient_clipping ............ 1.0
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   gradient_predivide_factor .... 1.0
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   graph_harvesting ............. False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   initial_dynamic_scale ........ 65536
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   load_universal_checkpoint .... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   loss_scale ................... 0
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   memory_breakdown ............. False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   mics_hierarchial_params_gather  False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   mics_shard_size .............. -1
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   nebula_config ................ {
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   fp16_auto_cast ............... False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   fp16_enabled ................. True
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   fp16_master_weights_and_gradients  False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   global_rank .................. 0
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   grad_accum_dtype ............. None
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   gradient_accumulation_steps .. 1
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   gradient_clipping ............ 1.0
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   gradient_predivide_factor .... 1.0
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   graph_harvesting ............. False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   initial_dynamic_scale ........ 65536
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   load_universal_checkpoint .... False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   loss_scale ................... 0
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   memory_breakdown ............. False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   mics_hierarchial_params_gather  False
+[2024-01-12 23:20:54,927] [INFO] [config.py:988:print]   mics_shard_size .............. -1
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   nebula_config ................ {
     "enabled": false, 
     "persistent_storage_path": null, 
     "persistent_time_interval": 100, 
@@ -387,32 +388,32 @@ Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_
     "enable_nebula_load": true, 
     "load_path": null
 }
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   optimizer_legacy_fusion ...... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   optimizer_name ............... adamw
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   optimizer_params ............. {'lr': 1e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   pld_enabled .................. False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   pld_params ................... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   prescale_gradients ........... False
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   scheduler_name ............... WarmupDecayLR
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   scheduler_params ............. {'last_batch_iteration': -1, 'total_num_steps': 1000, 'warmup_min_lr': 0, 'warmup_max_lr': 1e-05, 'warmup_num_steps': 500}
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   seq_parallel_communication_data_type  torch.float32
-[2024-01-12 16:30:54,821] [INFO] [config.py:988:print]   sparse_attention ............. None
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   sparse_gradients_enabled ..... False
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   steps_per_print .............. inf
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   train_batch_size ............. 32
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   train_micro_batch_size_per_gpu  32
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   use_data_before_expert_parallel_  False
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   use_node_local_storage ....... False
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   wall_clock_breakdown ......... False
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   weight_quantization_config ... None
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   world_size ................... 1
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   zero_allow_untested_optimizer  False
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   zero_enabled ................. True
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   zero_force_ds_cpu_optimizer .. True
-[2024-01-12 16:30:54,822] [INFO] [config.py:988:print]   zero_optimization_stage ...... 2
-[2024-01-12 16:30:54,822] [INFO] [config.py:974:print_user_config]   json = {
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   optimizer_legacy_fusion ...... False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   optimizer_name ............... adamw
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   optimizer_params ............. {'lr': 1e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   pld_enabled .................. False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   pld_params ................... False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   prescale_gradients ........... False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   scheduler_name ............... WarmupDecayLR
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   scheduler_params ............. {'last_batch_iteration': -1, 'total_num_steps': 2000, 'warmup_min_lr': 0, 'warmup_max_lr': 1e-05, 'warmup_num_steps': 500}
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   seq_parallel_communication_data_type  torch.float32
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   sparse_attention ............. None
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   sparse_gradients_enabled ..... False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   steps_per_print .............. inf
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   train_batch_size ............. 32
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   train_micro_batch_size_per_gpu  32
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   use_data_before_expert_parallel_  False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   use_node_local_storage ....... False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   wall_clock_breakdown ......... False
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   weight_quantization_config ... None
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   world_size ................... 1
+[2024-01-12 23:20:54,928] [INFO] [config.py:988:print]   zero_allow_untested_optimizer  False
+[2024-01-12 23:20:54,929] [INFO] [config.py:988:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
+[2024-01-12 23:20:54,929] [INFO] [config.py:988:print]   zero_enabled ................. True
+[2024-01-12 23:20:54,929] [INFO] [config.py:988:print]   zero_force_ds_cpu_optimizer .. True
+[2024-01-12 23:20:54,929] [INFO] [config.py:988:print]   zero_optimization_stage ...... 2
+[2024-01-12 23:20:54,929] [INFO] [config.py:974:print_user_config]   json = {
     "fp16": {
         "enabled": true, 
         "loss_scale": 0, 
@@ -434,7 +435,7 @@ Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_
         "type": "WarmupDecayLR", 
         "params": {
             "last_batch_iteration": -1, 
-            "total_num_steps": 1000, 
+            "total_num_steps": 2.000000e+03, 
             "warmup_min_lr": 0, 
             "warmup_max_lr": 1e-05, 
             "warmup_num_steps": 500
@@ -462,72 +463,20 @@ Config: alpha=0.000010, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_
         "enabled": false
     }
 }
-[2024-01-12 16:31:18,456] [INFO] [loss_scaler.py:190:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 2. Reducing hysteresis to 1
-[2024-01-12 16:31:30,969] [INFO] [loss_scaler.py:183:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 32768
-[2024-01-12 16:31:43,512] [INFO] [loss_scaler.py:183:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 32768, reducing to 16384
-[2024-01-12 16:31:56,986] [INFO] [loss_scaler.py:183:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 16384, reducing to 8192
-{'loss': 1.7847, 'learning_rate': 4.898977360288234e-06, 'epoch': 0.03}
-{'loss': 1.3566, 'learning_rate': 6.160712527409633e-06, 'epoch': 0.05}
-{'loss': 1.1407, 'learning_rate': 6.85912902234906e-06, 'epoch': 0.07}
-{'loss': 0.99, 'learning_rate': 7.344547104469332e-06, 'epoch': 0.1}
-{'eval_loss': 0.8486328125, 'eval_wer': 54.17241696568221, 'eval_runtime': 1294.216, 'eval_samples_per_second': 2.814, 'eval_steps_per_second': 0.176, 'epoch': 0.1}
-[2024-01-12 17:14:22,872] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step100 is about to be saved!
-[2024-01-12 17:14:22,879] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-100/global_step100/mp_rank_00_model_states.pt
-[2024-01-12 17:14:22,879] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-100/global_step100/mp_rank_00_model_states.pt...
-[2024-01-12 17:14:23,774] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-100/global_step100/mp_rank_00_model_states.pt.
-[2024-01-12 17:14:23,782] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-100/global_step100/zero_pp_rank_0_mp_rank_00_optim_states.pt...
-[2024-01-12 17:14:28,538] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-100/global_step100/zero_pp_rank_0_mp_rank_00_optim_states.pt.
-[2024-01-12 17:14:28,695] [INFO] [engine.py:3431:_save_zero_checkpoint] zero checkpoint saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-100/global_step100/zero_pp_rank_0_mp_rank_00_optim_states.pt
-[2024-01-12 17:14:28,695] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step100 is ready now!
-{'loss': 0.765, 'learning_rate': 7.716963756434345e-06, 'epoch': 0.12}
-{'loss': 0.5849, 'learning_rate': 8.019180844200955e-06, 'epoch': 0.15}
-{'loss': 0.7834, 'learning_rate': 8.27351214279797e-06, 'epoch': 1.02}
-{'loss': 0.7896, 'learning_rate': 8.49307723936858e-06, 'epoch': 1.04}
-{'eval_loss': 0.7578125, 'eval_wer': 48.339313644309506, 'eval_runtime': 1161.7981, 'eval_samples_per_second': 3.135, 'eval_steps_per_second': 0.196, 'epoch': 1.04}
-[2024-01-12 17:55:30,694] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step200 is about to be saved!
-[2024-01-12 17:55:30,700] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-200/global_step200/mp_rank_00_model_states.pt
-[2024-01-12 17:55:30,701] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-200/global_step200/mp_rank_00_model_states.pt...
-[2024-01-12 17:55:31,582] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-200/global_step200/mp_rank_00_model_states.pt.
-[2024-01-12 17:55:31,591] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-200/global_step200/zero_pp_rank_0_mp_rank_00_optim_states.pt...
-[2024-01-12 17:55:36,176] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-200/global_step200/zero_pp_rank_0_mp_rank_00_optim_states.pt.
-[2024-01-12 17:55:36,323] [INFO] [engine.py:3431:_save_zero_checkpoint] zero checkpoint saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-200/global_step200/zero_pp_rank_0_mp_rank_00_optim_states.pt
-[2024-01-12 17:55:36,323] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step200 is ready now!
-{'loss': 0.8413, 'learning_rate': 8.686247975778677e-06, 'epoch': 1.07}
-{'loss': 0.68, 'learning_rate': 8.858694625217149e-06, 'epoch': 1.09}
-{'loss': 0.5851, 'learning_rate': 9.014436199608479e-06, 'epoch': 1.12}
-{'loss': 0.4164, 'learning_rate': 9.156425255148058e-06, 'epoch': 1.14}
-{'eval_loss': 0.73876953125, 'eval_wer': 49.25936148679732, 'eval_runtime': 1191.6636, 'eval_samples_per_second': 3.056, 'eval_steps_per_second': 0.191, 'epoch': 1.14}
-[2024-01-12 18:37:19,231] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step300 is about to be saved!
-[2024-01-12 18:37:19,238] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-300/global_step300/mp_rank_00_model_states.pt
-[2024-01-12 18:37:19,238] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-300/global_step300/mp_rank_00_model_states.pt...
-[2024-01-12 18:37:20,104] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-300/global_step300/mp_rank_00_model_states.pt.
-[2024-01-12 18:37:20,112] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_optim_states.pt...
-[2024-01-12 18:37:24,766] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_optim_states.pt.
-[2024-01-12 18:37:24,902] [INFO] [engine.py:3431:_save_zero_checkpoint] zero checkpoint saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-300/global_step300/zero_pp_rank_0_mp_rank_00_optim_states.pt
-[2024-01-12 18:37:24,903] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step300 is ready now!
-{'loss': 0.5102, 'learning_rate': 9.28689473531776e-06, 'epoch': 2.01}
-{'loss': 0.602, 'learning_rate': 9.407574351377137e-06, 'epoch': 2.04}
-{'loss': 0.6399, 'learning_rate': 9.519831289296397e-06, 'epoch': 2.06}
-{'loss': 0.5456, 'learning_rate': 9.624764935335318e-06, 'epoch': 2.09}
-{'eval_loss': 0.7177734375, 'eval_wer': 46.126598583126324, 'eval_runtime': 1169.8926, 'eval_samples_per_second': 3.113, 'eval_steps_per_second': 0.195, 'epoch': 2.09}
-[2024-01-12 19:18:32,645] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step400 is about to be saved!
-[2024-01-12 19:18:32,655] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-400/global_step400/mp_rank_00_model_states.pt
-[2024-01-12 19:18:32,655] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-400/global_step400/mp_rank_00_model_states.pt...
-[2024-01-12 19:18:33,633] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-400/global_step400/mp_rank_00_model_states.pt.
-[2024-01-12 19:18:33,641] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-400/global_step400/zero_pp_rank_0_mp_rank_00_optim_states.pt...
-[2024-01-12 19:18:38,426] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-400/global_step400/zero_pp_rank_0_mp_rank_00_optim_states.pt.
-[2024-01-12 19:18:38,469] [INFO] [engine.py:3431:_save_zero_checkpoint] zero checkpoint saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-400/global_step400/zero_pp_rank_0_mp_rank_00_optim_states.pt
-[2024-01-12 19:18:38,470] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step400 is ready now!
-{'loss': 0.48, 'learning_rate': 9.723272550712454e-06, 'epoch': 2.11}
-{'loss': 0.3325, 'learning_rate': 9.816095971633122e-06, 'epoch': 2.14}
-{'loss': 0.3389, 'learning_rate': 9.90385555539545e-06, 'epoch': 3.01}
-{'loss': 0.476, 'learning_rate': 9.987075336738768e-06, 'epoch': 3.03}
-{'eval_loss': 0.7109375, 'eval_wer': 45.243352654338025, 'eval_runtime': 1126.9534, 'eval_samples_per_second': 3.232, 'eval_steps_per_second': 0.202, 'epoch': 3.03}
-[2024-01-12 20:00:05,991] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step500 is about to be saved!
-[2024-01-12 20:00:06,003] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-500/global_step500/mp_rank_00_model_states.pt
-[2024-01-12 20:00:06,003] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-500/global_step500/mp_rank_00_model_states.pt...
-[2024-01-12 20:00:06,891] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-500/global_step500/mp_rank_00_model_states.pt.
-[2024-01-12 20:00:06,900] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt...
-[2024-01-12 20:00:11,513] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt.
-[2024-01-12 20:00:11,652] [INFO] [engine.py:3431:_save_zero_checkpoint] zero checkpoint saved /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/tmp-checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt
-[2024-01-12 20:00:11,653] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step500 is ready now!
+[2024-01-12 23:20:54,947] [INFO] [torch_checkpoint_engine.py:27:load] [Torch] Loading checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt...
+[2024-01-12 23:20:56,196] [INFO] [torch_checkpoint_engine.py:29:load] [Torch] Loaded checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt.
+[2024-01-12 23:20:56,219] [INFO] [torch_checkpoint_engine.py:27:load] [Torch] Loading checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt...
+[2024-01-12 23:20:56,262] [INFO] [torch_checkpoint_engine.py:29:load] [Torch] Loaded checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt.
+[2024-01-12 23:20:56,313] [INFO] [torch_checkpoint_engine.py:27:load] [Torch] Loading checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2024-01-12 23:21:01,178] [INFO] [torch_checkpoint_engine.py:29:load] [Torch] Loaded checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2024-01-12 23:21:01,178] [INFO] [engine.py:2988:_get_all_zero_checkpoint_state_dicts] successfully read 1 ZeRO state_dicts for rank 0
+[2024-01-12 23:21:01,212] [INFO] [engine.py:2920:_load_zero_checkpoint] loading 1 zero partition checkpoints for rank 0
+[2024-01-12 23:22:01,846] [INFO] [torch_checkpoint_engine.py:27:load] [Torch] Loading checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt...
+[2024-01-12 23:22:02,649] [INFO] [torch_checkpoint_engine.py:29:load] [Torch] Loaded checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt.
+[2024-01-12 23:22:02,658] [INFO] [torch_checkpoint_engine.py:27:load] [Torch] Loading checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt...
+[2024-01-12 23:22:02,684] [INFO] [torch_checkpoint_engine.py:29:load] [Torch] Loaded checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/mp_rank_00_model_states.pt.
+[2024-01-12 23:22:02,762] [INFO] [torch_checkpoint_engine.py:27:load] [Torch] Loading checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2024-01-12 23:22:07,278] [INFO] [torch_checkpoint_engine.py:29:load] [Torch] Loaded checkpoint from /cosmos/home/sp-operator/ai/training/models/huggingface/scripts/../breeze-dsw-tiny-id/checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2024-01-12 23:22:07,279] [INFO] [engine.py:2988:_get_all_zero_checkpoint_state_dicts] successfully read 1 ZeRO state_dicts for rank 0
+[2024-01-12 23:22:07,313] [INFO] [engine.py:2920:_load_zero_checkpoint] loading 1 zero partition checkpoints for rank 0
+{'train_runtime': 66.1004, 'train_samples_per_second': 968.225, 'train_steps_per_second': 30.257, 'train_loss': 0.0, 'epoch': 3.03}

generation_config.json

@@ -0,0 +1,247 @@
+{
+  "alignment_heads": [
+    [
+      2,
+      2
+    ],
+    [
+      3,
+      0
+    ],
+    [
+      3,
+      2
+    ],
+    [
+      3,
+      3
+    ],
+    [
+      3,
+      4
+    ],
+    [
+      3,
+      5
+    ]
+  ],
+  "begin_suppress_tokens": [
+    220,
+    50257
+  ],
+  "bos_token_id": 50257,
+  "decoder_start_token_id": 50258,
+  "eos_token_id": 50257,
+  "forced_decoder_ids": [
+    [
+      1,
+      null
+    ],
+    [
+      2,
+      50359
+    ]
+  ],
+  "is_multilingual": true,
+  "lang_to_id": {
+    "<|af|>": 50327,
+    "<|am|>": 50334,
+    "<|ar|>": 50272,
+    "<|as|>": 50350,
+    "<|az|>": 50304,
+    "<|ba|>": 50355,
+    "<|be|>": 50330,
+    "<|bg|>": 50292,
+    "<|bn|>": 50302,
+    "<|bo|>": 50347,
+    "<|br|>": 50309,
+    "<|bs|>": 50315,
+    "<|ca|>": 50270,
+    "<|cs|>": 50283,
+    "<|cy|>": 50297,
+    "<|da|>": 50285,
+    "<|de|>": 50261,
+    "<|el|>": 50281,
+    "<|en|>": 50259,
+    "<|es|>": 50262,
+    "<|et|>": 50307,
+    "<|eu|>": 50310,
+    "<|fa|>": 50300,
+    "<|fi|>": 50277,
+    "<|fo|>": 50338,
+    "<|fr|>": 50265,
+    "<|gl|>": 50319,
+    "<|gu|>": 50333,
+    "<|haw|>": 50352,
+    "<|ha|>": 50354,
+    "<|he|>": 50279,
+    "<|hi|>": 50276,
+    "<|hr|>": 50291,
+    "<|ht|>": 50339,
+    "<|hu|>": 50286,
+    "<|hy|>": 50312,
+    "<|id|>": 50275,
+    "<|is|>": 50311,
+    "<|it|>": 50274,
+    "<|ja|>": 50266,
+    "<|jw|>": 50356,
+    "<|ka|>": 50329,
+    "<|kk|>": 50316,
+    "<|km|>": 50323,
+    "<|kn|>": 50306,
+    "<|ko|>": 50264,
+    "<|la|>": 50294,
+    "<|lb|>": 50345,
+    "<|ln|>": 50353,
+    "<|lo|>": 50336,
+    "<|lt|>": 50293,
+    "<|lv|>": 50301,
+    "<|mg|>": 50349,
+    "<|mi|>": 50295,
+    "<|mk|>": 50308,
+    "<|ml|>": 50296,
+    "<|mn|>": 50314,
+    "<|mr|>": 50320,
+    "<|ms|>": 50282,
+    "<|mt|>": 50343,
+    "<|my|>": 50346,
+    "<|ne|>": 50313,
+    "<|nl|>": 50271,
+    "<|nn|>": 50342,
+    "<|no|>": 50288,
+    "<|oc|>": 50328,
+    "<|pa|>": 50321,
+    "<|pl|>": 50269,
+    "<|ps|>": 50340,
+    "<|pt|>": 50267,
+    "<|ro|>": 50284,
+    "<|ru|>": 50263,
+    "<|sa|>": 50344,
+    "<|sd|>": 50332,
+    "<|si|>": 50322,
+    "<|sk|>": 50298,
+    "<|sl|>": 50305,
+    "<|sn|>": 50324,
+    "<|so|>": 50326,
+    "<|sq|>": 50317,
+    "<|sr|>": 50303,
+    "<|su|>": 50357,
+    "<|sv|>": 50273,
+    "<|sw|>": 50318,
+    "<|ta|>": 50287,
+    "<|te|>": 50299,
+    "<|tg|>": 50331,
+    "<|th|>": 50289,
+    "<|tk|>": 50341,
+    "<|tl|>": 50348,
+    "<|tr|>": 50268,
+    "<|tt|>": 50351,
+    "<|uk|>": 50280,
+    "<|ur|>": 50290,
+    "<|uz|>": 50337,
+    "<|vi|>": 50278,
+    "<|yi|>": 50335,
+    "<|yo|>": 50325,
+    "<|zh|>": 50260
+  },
+  "max_initial_timestamp_index": 1,
+  "max_length": 448,
+  "no_timestamps_token_id": 50363,
+  "pad_token_id": 50257,
+  "return_timestamps": false,
+  "suppress_tokens": [
+    1,
+    2,
+    7,
+    8,
+    9,
+    10,
+    14,
+    25,
+    26,
+    27,
+    28,
+    29,
+    31,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63,
+    90,
+    91,
+    92,
+    93,
+    359,
+    503,
+    522,
+    542,
+    873,
+    893,
+    902,
+    918,
+    922,
+    931,
+    1350,
+    1853,
+    1982,
+    2460,
+    2627,
+    3246,
+    3253,
+    3268,
+    3536,
+    3846,
+    3961,
+    4183,
+    4667,
+    6585,
+    6647,
+    7273,
+    9061,
+    9383,
+    10428,
+    10929,
+    11938,
+    12033,
+    12331,
+    12562,
+    13793,
+    14157,
+    14635,
+    15265,
+    15618,
+    16553,
+    16604,
+    18362,
+    18956,
+    20075,
+    21675,
+    22520,
+    26130,
+    26161,
+    26435,
+    28279,
+    29464,
+    31650,
+    32302,
+    32470,
+    36865,
+    42863,
+    47425,
+    49870,
+    50254,
+    50258,
+    50358,
+    50359,
+    50360,
+    50361,
+    50362
+  ],
+  "task_to_id": {
+    "transcribe": 50359,
+    "translate": 50358
+  },
+  "transformers_version": "4.37.0.dev0"
+}

runs/Jan12_16-30-10_knight/events.out.tfevents.1705073454.knight.2969.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:47a10f2b390f8fcf2d8432d9a833e6e89e5082aeb6216492ad977e96b4cbd615
-size 10146
+oid sha256:2ba9e233371b3e102c4903c003d3b8439a3a019599bfacc858d99690184902e0
+size 10774

runs/Jan12_23-20-08_knight/events.out.tfevents.1705098061.knight.5818.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ebe12d70f80901f82ef94e5b29b2a2326f06d8ff421d77d2bbcef91d8b4f272
+size 5792

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3fd618dd6c112a73e5aab8aeb42b07a9a09d43debf0d38a038d883287d3dc201
+oid sha256:46f603b46ca246f27404536a46a7e47f47604598223fce33b450057908051757
 size 6648