[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,925] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:52,926] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,014] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,014] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,014] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,014] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,014] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,014] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,015] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,015] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,175] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,176] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,489] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,490] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:53,555] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible

 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,565] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:54,566] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,000] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,173] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,173] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,173] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,173] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,173] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,173] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,174] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,174] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-21 10:19:55,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,407] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,407] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,436] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,558] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,603] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,603] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,603] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,603] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,604] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,604] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,604] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,604] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,611] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,612] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,712] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,713] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,715] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,717] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,716] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,717] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-21 10:19:55,822] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH

 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH

 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,026] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,104] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:56,105] [INFO] [comm.py:637:init_distributed] cdb=None
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3

 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH

 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.

 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING]  async_io: please install the libaio-dev package with apt

 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-21 10:19:57,234] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,235] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,237] [INFO] [comm.py:637:init_distributed] cdb=None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:57,725] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,725] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,726] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,726] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,726] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,726] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,726] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,726] [INFO] [comm.py:637:init_distributed] cdb=None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:57,877] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,878] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,879] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,879] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,879] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,879] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,879] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:57,880] [INFO] [comm.py:637:init_distributed] cdb=None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,116] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,117] [INFO] [comm.py:637:init_distributed] cdb=None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,123] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,124] [INFO] [comm.py:637:init_distributed] cdb=None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:58,233] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,234] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,235] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,235] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,236] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,236] [INFO] [comm.py:637:init_distributed] cdb=None

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.02it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.62it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.65it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.24it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,310] [INFO] [comm.py:637:init_distributed] cdb=None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-21 10:19:58,350] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,351] [INFO] [comm.py:637:init_distributed] cdb=None

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s][2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,445] [INFO] [comm.py:637:init_distributed] cdb=None

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s][2024-08-21 10:19:58,454] [INFO] [comm.py:637:init_distributed] cdb=None

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.05it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.04it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.54it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.31it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.43it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.67it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.32it/s][2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-21 10:19:58,529] [INFO] [comm.py:637:init_distributed] cdb=None

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.63it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.20it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.34it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.63it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.72it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.04it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.25it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.45it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.32it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.23it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.30it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.80it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.73it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.32it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.97it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.42it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.16it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.01it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.73it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.52it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.03it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.56it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.49it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.23it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.97it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.02it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.46it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.39it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.92it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.46it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.53it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.02it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.04it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.09it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.83it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.13it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.13it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.47it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.19it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.33it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.30it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.13it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.60it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.84it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.42it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.16it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.21it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.67it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.92it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.86it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.10it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  3.61it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.82it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.60it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  3.58it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  3.56it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.54it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.05it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.60it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.39it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.18it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.48it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.20it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.00it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.75it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.40it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.28it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.45it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  3.49it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.27it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.95it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.48it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.40it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.21it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.36it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.12it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.08it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.04it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.15it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.38it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.01it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.54it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.12it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.22it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.55it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.29it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.73it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.10it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.11it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.16it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.35it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.09it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.69it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.38it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.34it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.45it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.10it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.22it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.07it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.06it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.17it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.81it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.75it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.64it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.44it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.06it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.69it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.82it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.71it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.73it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.62it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.28it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  4.28it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  3.90it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.78it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.50it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.58it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.60it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.60it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.39it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.70it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.34it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.00it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.79it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.52it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.75it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.36it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.40it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.83it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.37it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.22it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.12it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  4.09it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.68it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.08it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  3.98it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.32it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.48it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.76it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.80it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.63it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.52it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.39it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.18it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.00it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.79it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.90it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.81it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.34it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.07it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.31it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.38it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.45it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.27it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.02it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.52it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.38it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.38it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.01it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.48it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.10it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.64it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.67it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.48it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.03it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.77it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.16it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.85it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.57it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.17it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.90it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.35it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.22it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.47it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.28it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.91it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.63it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.46it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.07it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.45it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.46it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.31it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.22it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.73it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.88it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.77it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.72it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.06it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.30it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.41it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.71it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.25it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.64it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.18it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.20it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.64it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.42it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.37it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.28it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.19it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.67it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.53it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.34it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.76it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.56it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.38it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.16it/s]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.62it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.51it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.36it/s]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.17it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.29it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
**************************************** Experiment setup ****************************************:
llm_pretrained: meta-llama/Meta-Llama-3.1-8B-Instruct
vision_pretrained: google/siglip-so400m-patch14-384
vision_hidden_size: 1152
img_resolution: None
use_img_cls_token: True
img_patch_token_size: 3
img_patch_token_layer: -2
img_sep_token: 
max_seq_len: 4096
padding_side: right
attn_implementation: flash_attention_2
w2t_logit_weight: 1.0
use_binary_decision_head: False
binary_loss_weight: 0.1
binary_decision_head_type: linear
output_dir: /fsx_0/user/imzyc/proact_exps/20240821-L4096-I10-ep4-NOSEP-nr0.1-klgmix-1s-lora-bs512
overwrite_output_dir: False
do_train: False
do_eval: True
do_predict: False
eval_strategy: steps
prediction_loss_only: False
per_device_train_batch_size: 4
per_device_eval_batch_size: 4
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
eval_delay: 0
torch_empty_cache_steps: None
learning_rate: 0.0002
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4.0
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 100
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_dir: /fsx_0/user/imzyc/proact_exps/20240821-L4096-I10-ep4-NOSEP-nr0.1-klgmix-1s-lora-bs512/runs/Aug21_10-19-54_h100-st-p548xlarge-136
logging_strategy: steps
logging_first_step: False
logging_steps: 10
logging_nan_inf_filter: True
save_strategy: epoch
save_steps: 500
save_total_limit: 1
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
eval_steps: 400
dataloader_num_workers: 2
dataloader_prefetch_factor: 2
past_index: -1
run_name: 20240821-L4096-I10-ep4-NOSEP-nr0.1-klgmix-1s-lora-bs512
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
metric_for_best_model: None
greater_is_better: None
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: deepspeed/zero2.json
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
report_to: ['tensorboard', 'wandb']
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: None
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_token: <HUB_TOKEN>
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: True
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
evaluation_strategy: steps
push_to_hub_model_id: None
push_to_hub_organization: None
push_to_hub_token: <PUSH_TO_HUB_TOKEN>
mp_parameters: 
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
eval_use_gather_object: False
data_root_dir: /fsx_0/user/imzyc/processed_data/
train_datasets: ego4d/narration_train_L4096_I10,sthsthv2/narration_train_L4096_I10@10,llava/caption_train_L4096_I10@2,egoobjects/detection_train_L4096_I10@20,ego4d/summary_train_L4096_I10@2,holoassist/summary_train_L4096_I10@2,epickitchens/summary_train_L4096_I10@2,egoexolearn/summary_train_L4096_I10@2,wtag/summary_train_L4096_I10@2,assembly101/summary_train_L4096_I10@2,ego4d/dialog-klg-sum_train_L4096_I10@2,holoassist/dialog-klg-sum_train_L4096_I10@2,epickitchens/dialog-klg-sum_train_L4096_I10@2,egoexolearn/dialog-klg-sum_train_L4096_I10@2,wtag/dialog-klg-sum_train_L4096_I10@5,assembly101/dialog-klg-sum_train_L4096_I10@2
eval_datasets: sthsthv2/narration_val_L4096_I10,ego4d/dialog-klg-sum_val_L4096_I10,holoassist/dialog-klg-sum_val_L4096_I10,ego4d/narration_val_L4096_I10,ego4d/summary_val_L4096_I10
neg_frame_sampling_rate: 0.1
use_pose: False
llm_train_mode: lora
lora_modules: model.*(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)|lm_head$
lora_r: 128
lora_alpha: 256
finetune_modules: mm_projector
is_debug: False
****************************************************************************************************
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.18it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.31it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.35it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.88it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.29it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.76it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.39it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.34it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.03it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.88it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.65it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.03it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.13it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.02it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.16it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.05it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.05it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.31it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.73it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.34it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.38it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.30it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.20it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.61it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.11it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.21it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.62it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.46it/s]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.04it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.99it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.65it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.46it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.67it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.48it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.63it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.39it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.54it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.39it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.98it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.63it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.93it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.53it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.40it/s]
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.18it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.25it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.37it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.43it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.35it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.82it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.66it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.43it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.52it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.76it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.22it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.29it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.22it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.04it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.86it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.28it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.53it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.19it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.11it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.25it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.08it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.23it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.69it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.73it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.14it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.93it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  3.85it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.45it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.35it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.28it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.98it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.64it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.15it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.93it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.52it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.22it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.53it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.41it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.61it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.73it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.39it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.64it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.46it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.40it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.51it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.01it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.32it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.98it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.42it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.59it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.65it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.66it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.44it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.52it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.37it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.30it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.47it/s]You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  4.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  3.96it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.41it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.27it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.11it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.42it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.20it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.24it/s]

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.35it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.16it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.77it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.90it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.76it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.80it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.69it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.21it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.21it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.13it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.28it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.80it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.42it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.16it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.04it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.90it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.61it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.92it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.38it/s]

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.32it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.84it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.94it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.56it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.10it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.61it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.64it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.34it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.58it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.48it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.15it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.74it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.77it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.66it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.46it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.96it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.17it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.09it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.85it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.61it/s]

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.81it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.25it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.66it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.72it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.70it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.40it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.97it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.49it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.32it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.43it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.67it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.44it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.05it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.41it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.20it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.84it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.67it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.59it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.33it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.60it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.46it/s]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.18it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.40it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.85it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.60it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.43it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.13it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.36it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.99it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.40it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.23it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.24it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.27it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.97it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.24it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.67it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.17it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.27it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.91it/s]

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.44it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.99it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.82it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.27it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.93it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.27it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.02it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.33it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.58it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.29it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.41it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.02it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.42it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.22it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.43it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.73it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.85it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.32it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.52it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.48it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.16it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.37it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.30it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.42it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.01it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.69it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.30it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.68it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.49it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.15it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.79it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.70it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.41it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.89it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.00it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.33it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.88it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.78it/s]

Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.61it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  3.86it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.40it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.27it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.83it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.67it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.56it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.71it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.81it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.17it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.91it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.79it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.24it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.21it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.20it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.45it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.76it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.62it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.38it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.92it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.97it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.71it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.44it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.56it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.74it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.16it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.43it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.39it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.08it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.17it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.52it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  5.71it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.71it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.60it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.40it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.24it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.75it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.67it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.38it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.30it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.04it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.37it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.39it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.19it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.01it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.45it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.96it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.60it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.34it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.48it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  6.02it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.40it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.01it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.59it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.34it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.49it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.56it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.36it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.46it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.34it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.38it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.27it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.22it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.03it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.53it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.77it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.57it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.05it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.76it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.98it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.13it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.45it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.29it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.69it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.44it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.78it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.60it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.10it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.68it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.78it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.53it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.60it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  4.07it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00,  3.96it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.70it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.02it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.46it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.26it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  5.23it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.89it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.63it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.65it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.14it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.18it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.47it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.79it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.69it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.23it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.84it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.51it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.57it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.24it/s]

Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  6.07it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.73it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.62it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.40it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.20it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.00it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.44it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.15it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.33it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.55it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.15it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.02it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.83it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.77it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.44it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.35it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.98it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.08it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.10it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.86it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.51it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.91it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.72it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.50it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.32it/s]

Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  6.10it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.18it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.89it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.70it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.87it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.62it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.36it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.20it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.98it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.72it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.46it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.33it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.26it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  6.19it/s]
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386

trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
trainable params: 373,989,376 || all params: 8,425,754,624 || trainable%: 4.4386
Training datasets:
* ego4d/narration_train_L4096_I10@1 | num samples: 101672
* sthsthv2/narration_train_L4096_I10@10 | num samples: 39590
* llava/caption_train_L4096_I10@2 | num samples: 15680
* egoobjects/detection_train_L4096_I10@20 | num samples: 11040
* ego4d/summary_train_L4096_I10@2 | num samples: 12740
* holoassist/summary_train_L4096_I10@2 | num samples: 18558
* epickitchens/summary_train_L4096_I10@2 | num samples: 5506
* egoexolearn/summary_train_L4096_I10@2 | num samples: 10136
* wtag/summary_train_L4096_I10@2 | num samples: 948
* assembly101/summary_train_L4096_I10@2 | num samples: 7336
* ego4d/dialog-klg-sum_train_L4096_I10@2 | num samples: 41436
* holoassist/dialog-klg-sum_train_L4096_I10@2 | num samples: 32232
* epickitchens/dialog-klg-sum_train_L4096_I10@2 | num samples: 38640
* egoexolearn/dialog-klg-sum_train_L4096_I10@2 | num samples: 37454
* wtag/dialog-klg-sum_train_L4096_I10@5 | num samples: 17685
* assembly101/dialog-klg-sum_train_L4096_I10@2 | num samples: 25476
Total training data size: 416.1K
Evaluation datasets:
* sthsthv2/narration_val | num samples: 668
* ego4d/dialog-klg-sum_val | num samples: 592
* holoassist/dialog-klg-sum_val | num samples: 709
* ego4d/narration_val | num samples: 223
* ego4d/summary_val | num samples: 564

wandb: Currently logged in as: 594zyc (meta-intern-yichi). Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.17.7
wandb: Run data is saved locally in /opt/hpcaas/.mounts/fs-036153e63d56f4dc2/home/imzyc/project/proactive-assist/wandb/run-20240821_102007-099a2e32-77a1-438d-8762-5f5fbff7af2c
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run 20240821-L4096-I10-ep4-NOSEP-nr0.1-klgmix-1s-lora-bs512
wandb: ⭐️ View project at https://wandb.ai/meta-intern-yichi/ProAct
wandb: 🚀 View run at https://wandb.ai/meta-intern-yichi/ProAct/runs/099a2e32-77a1-438d-8762-5f5fbff7af2c

  0%|          | 0/3252 [00:00<?, ?it/s]
  0%|          | 1/3252 [00:06<6:15:00,  6.92s/it]
  0%|          | 2/3252 [00:09<3:57:44,  4.39s/it]
  0%|          | 3/3252 [00:12<3:25:32,  3.80s/it]
  0%|          | 4/3252 [00:15<3:12:21,  3.55s/it]
  0%|          | 5/3252 [00:18<3:02:11,  3.37s/it]
  0%|          | 6/3252 [00:22<3:03:40,  3.40s/it]
  0%|          | 7/3252 [00:25<2:57:12,  3.28s/it]
  0%|          | 8/3252 [00:28<2:53:18,  3.21s/it]
  0%|          | 9/3252 [00:31<2:50:30,  3.15s/it]
  0%|          | 10/3252 [00:34<2:48:12,  3.11s/it]
                                                   
{'loss': 5.6241, 'grad_norm': 12.897705078125, 'learning_rate': 2e-05, 'lm_loss': 2.9839565753936768, 'epoch': 0.01}

  0%|          | 10/3252 [00:45<2:48:12,  3.11s/it]
  0%|          | 11/3252 [00:48<5:42:19,  6.34s/it]
  0%|          | 12/3252 [00:51<4:50:08,  5.37s/it]
  0%|          | 13/3252 [00:54<4:09:06,  4.61s/it]
  0%|          | 14/3252 [00:57<3:43:39,  4.14s/it]
  0%|          | 15/3252 [01:00<3:25:56,  3.82s/it]
  0%|          | 16/3252 [01:03<3:17:05,  3.65s/it]
  1%|          | 17/3252 [01:06<3:07:10,  3.47s/it]
  1%|          | 18/3252 [01:09<3:00:15,  3.34s/it]
  1%|          | 19/3252 [01:12<2:55:08,  3.25s/it]
  1%|          | 20/3252 [01:15<2:49:12,  3.14s/it]
                                                   
{'loss': 1.8228, 'grad_norm': 2.483752965927124, 'learning_rate': 4e-05, 'lm_loss': 2.235138416290283, 'epoch': 0.02}

  1%|          | 20/3252 [01:15<2:49:12,  3.14s/it]
  1%|          | 21/3252 [01:18<2:45:00,  3.06s/it]
  1%|          | 22/3252 [01:21<2:39:18,  2.96s/it]
  1%|          | 23/3252 [01:24<2:40:30,  2.98s/it]
  1%|          | 24/3252 [01:27<2:43:34,  3.04s/it]
  1%|          | 25/3252 [01:30<2:43:34,  3.04s/it]
  1%|          | 26/3252 [01:33<2:45:45,  3.08s/it]
  1%|          | 27/3252 [01:36<2:37:29,  2.93s/it]
  1%|          | 28/3252 [01:39<2:41:10,  3.00s/it]
  1%|          | 29/3252 [01:42<2:44:57,  3.07s/it]
  1%|          | 30/3252 [01:45<2:46:34,  3.10s/it]
                                                   
{'loss': 1.3971, 'grad_norm': 1.0765433311462402, 'learning_rate': 6e-05, 'lm_loss': 0.9527426958084106, 'epoch': 0.04}

  1%|          | 30/3252 [01:45<2:46:34,  3.10s/it]
  1%|          | 31/3252 [01:48<2:35:44,  2.90s/it]
  1%|          | 32/3252 [01:50<2:30:46,  2.81s/it]
  1%|          | 33/3252 [01:53<2:34:34,  2.88s/it]
  1%|          | 34/3252 [01:56<2:32:05,  2.84s/it]
  1%|          | 35/3252 [01:59<2:32:52,  2.85s/it]
  1%|          | 36/3252 [02:02<2:35:50,  2.91s/it]
  1%|          | 37/3252 [02:05<2:35:46,  2.91s/it]
  1%|          | 38/3252 [02:08<2:37:51,  2.95s/it]
  1%|          | 39/3252 [02:10<2:28:47,  2.78s/it]
  1%|          | 40/3252 [02:13<2:32:55,  2.86s/it]
                                                   
{'loss': 1.3196, 'grad_norm': 1.9058876037597656, 'learning_rate': 8e-05, 'lm_loss': 0.7674030661582947, 'epoch': 0.05}

  1%|          | 40/3252 [02:13<2:32:55,  2.86s/it]
  1%|▏         | 41/3252 [02:16<2:33:46,  2.87s/it]
  1%|▏         | 42/3252 [02:19<2:33:25,  2.87s/it]
  1%|▏         | 43/3252 [02:22<2:29:16,  2.79s/it]
  1%|▏         | 44/3252 [02:24<2:22:56,  2.67s/it]
  1%|▏         | 45/3252 [02:27<2:26:16,  2.74s/it]
  1%|▏         | 46/3252 [02:30<2:30:54,  2.82s/it]
  1%|▏         | 47/3252 [02:32<2:23:55,  2.69s/it]
  1%|▏         | 48/3252 [02:35<2:26:53,  2.75s/it]
  2%|▏         | 49/3252 [02:38<2:26:16,  2.74s/it]
  2%|▏         | 50/3252 [02:41<2:31:59,  2.85s/it]
                                                   
{'loss': 1.2244, 'grad_norm': 4.616240501403809, 'learning_rate': 0.0001, 'lm_loss': 0.6509291529655457, 'epoch': 0.06}

  2%|▏         | 50/3252 [02:42<2:31:59,  2.85s/it]
  2%|▏         | 51/3252 [02:44<2:31:05,  2.83s/it]
  2%|▏         | 52/3252 [02:47<2:34:38,  2.90s/it]
  2%|▏         | 53/3252 [02:49<2:26:50,  2.75s/it]
  2%|▏         | 54/3252 [02:52<2:26:21,  2.75s/it]
  2%|▏         | 55/3252 [02:55<2:30:32,  2.83s/it]
  2%|▏         | 56/3252 [02:58<2:27:44,  2.77s/it]
  2%|▏         | 57/3252 [03:01<2:29:25,  2.81s/it]
  2%|▏         | 58/3252 [03:03<2:22:46,  2.68s/it]
  2%|▏         | 59/3252 [03:05<2:19:10,  2.62s/it]
  2%|▏         | 60/3252 [03:08<2:15:35,  2.55s/it]
                                                   
{'loss': 1.1823, 'grad_norm': 4.490408420562744, 'learning_rate': 0.00012, 'lm_loss': 0.8741838932037354, 'epoch': 0.07}

  2%|▏         | 60/3252 [03:08<2:15:35,  2.55s/it]
  2%|▏         | 61/3252 [03:11<2:20:39,  2.64s/it]
  2%|▏         | 62/3252 [03:13<2:17:01,  2.58s/it]
  2%|▏         | 63/3252 [03:16<2:14:02,  2.52s/it]
  2%|▏         | 64/3252 [03:19<2:22:13,  2.68s/it]
  2%|▏         | 65/3252 [03:22<2:27:50,  2.78s/it]
  2%|▏         | 66/3252 [03:24<2:21:40,  2.67s/it]
  2%|▏         | 67/3252 [03:27<2:20:10,  2.64s/it]
  2%|▏         | 68/3252 [03:29<2:16:04,  2.56s/it]
  2%|▏         | 69/3252 [03:32<2:20:46,  2.65s/it]
  2%|▏         | 70/3252 [03:34<2:16:40,  2.58s/it]
                                                   
{'loss': 1.1358, 'grad_norm': 1.0905792713165283, 'learning_rate': 0.00014, 'lm_loss': 1.232476830482483, 'epoch': 0.09}

  2%|▏         | 70/3252 [03:34<2:16:40,  2.58s/it]
  2%|▏         | 71/3252 [03:37<2:26:32,  2.76s/it]
  2%|▏         | 72/3252 [03:40<2:20:25,  2.65s/it]
  2%|▏         | 73/3252 [03:42<2:19:57,  2.64s/it]
  2%|▏         | 74/3252 [03:45<2:24:04,  2.72s/it]
  2%|▏         | 75/3252 [03:48<2:29:12,  2.82s/it]
  2%|▏         | 76/3252 [03:51<2:22:28,  2.69s/it]
  2%|▏         | 77/3252 [03:54<2:25:16,  2.75s/it]
  2%|▏         | 78/3252 [03:56<2:19:45,  2.64s/it]
  2%|▏         | 79/3252 [03:59<2:23:08,  2.71s/it]
  2%|▏         | 80/3252 [04:01<2:18:26,  2.62s/it]
                                                   
{'loss': 1.1248, 'grad_norm': 1.432718276977539, 'learning_rate': 0.00016, 'lm_loss': 0.9508834481239319, 'epoch': 0.1}

  2%|▏         | 80/3252 [04:02<2:18:26,  2.62s/it]
  2%|▏         | 81/3252 [04:04<2:26:12,  2.77s/it]
  3%|▎         | 82/3252 [04:07<2:20:13,  2.65s/it]
  3%|▎         | 83/3252 [04:09<2:16:05,  2.58s/it]
  3%|▎         | 84/3252 [04:12<2:13:15,  2.52s/it]
  3%|▎         | 85/3252 [04:14<2:11:13,  2.49s/it]
  3%|▎         | 86/3252 [04:17<2:12:43,  2.52s/it]
  3%|▎         | 87/3252 [04:19<2:10:43,  2.48s/it]
  3%|▎         | 88/3252 [04:21<2:09:24,  2.45s/it]
  3%|▎         | 89/3252 [04:24<2:16:23,  2.59s/it]
  3%|▎         | 90/3252 [04:27<2:15:32,  2.57s/it]
                                                   
{'loss': 1.0811, 'grad_norm': 0.9401188492774963, 'learning_rate': 0.00018, 'lm_loss': 0.5765393376350403, 'epoch': 0.11}

  3%|▎         | 90/3252 [04:27<2:15:32,  2.57s/it]
  3%|▎         | 91/3252 [04:29<2:14:49,  2.56s/it]
  3%|▎         | 92/3252 [04:32<2:12:16,  2.51s/it]
  3%|▎         | 93/3252 [04:35<2:15:50,  2.58s/it]
  3%|▎         | 94/3252 [04:37<2:12:50,  2.52s/it]
  3%|▎         | 95/3252 [04:39<2:10:52,  2.49s/it]
  3%|▎         | 96/3252 [04:42<2:09:17,  2.46s/it]
  3%|▎         | 97/3252 [04:44<2:08:40,  2.45s/it]
  3%|▎         | 98/3252 [04:47<2:07:50,  2.43s/it]
  3%|▎         | 99/3252 [04:49<2:07:10,  2.42s/it]
  3%|▎         | 100/3252 [04:51<2:06:46,  2.41s/it]
                                                    
{'loss': 1.0659, 'grad_norm': 0.36784738302230835, 'learning_rate': 0.0002, 'lm_loss': 0.7196485996246338, 'epoch': 0.12}

  3%|▎         | 100/3252 [04:51<2:06:46,  2.41s/it]
  3%|▎         | 101/3252 [04:54<2:08:23,  2.44s/it]
  3%|▎         | 102/3252 [04:56<2:10:05,  2.48s/it]
  3%|▎         | 103/3252 [04:59<2:19:02,  2.65s/it]
  3%|▎         | 104/3252 [05:02<2:15:06,  2.58s/it]
  3%|▎         | 105/3252 [05:04<2:12:12,  2.52s/it]
  3%|▎         | 106/3252 [05:07<2:10:35,  2.49s/it]
  3%|▎         | 107/3252 [05:09<2:09:26,  2.47s/it]
  3%|▎         | 108/3252 [05:11<2:08:17,  2.45s/it]
  3%|▎         | 109/3252 [05:14<2:07:23,  2.43s/it]
  3%|▎         | 110/3252 [05:16<2:06:58,  2.42s/it]
                                                    
{'loss': 1.0396, 'grad_norm': 1.6434264183044434, 'learning_rate': 0.00019999503300483976, 'lm_loss': 0.5776724815368652, 'epoch': 0.14}

  3%|▎         | 110/3252 [05:16<2:06:58,  2.42s/it]
  3%|▎         | 111/3252 [05:19<2:07:49,  2.44s/it]
  3%|▎         | 112/3252 [05:21<2:06:58,  2.43s/it]
  3%|▎         | 113/3252 [05:24<2:06:28,  2.42s/it]
  4%|▎         | 114/3252 [05:26<2:06:20,  2.42s/it]
  4%|▎         | 115/3252 [05:28<2:06:03,  2.41s/it]
  4%|▎         | 116/3252 [05:31<2:05:56,  2.41s/it]
  4%|▎         | 117/3252 [05:33<2:05:35,  2.40s/it]
  4%|▎         | 118/3252 [05:36<2:05:26,  2.40s/it]
  4%|▎         | 119/3252 [05:38<2:10:10,  2.49s/it]
  4%|▎         | 120/3252 [05:41<2:08:25,  2.46s/it]
                                                    
{'loss': 0.9472, 'grad_norm': 1.7971677780151367, 'learning_rate': 0.00019998013251277988, 'lm_loss': 1.0168853998184204, 'epoch': 0.15}

  4%|▎         | 120/3252 [05:41<2:08:25,  2.46s/it]
  4%|▎         | 121/3252 [05:43<2:10:03,  2.49s/it]
  4%|▍         | 122/3252 [05:46<2:08:46,  2.47s/it]
  4%|▍         | 123/3252 [05:48<2:13:44,  2.56s/it]
  4%|▍         | 124/3252 [05:51<2:13:29,  2.56s/it]
  4%|▍         | 125/3252 [05:53<2:10:53,  2.51s/it]
  4%|▍         | 126/3252 [05:56<2:09:00,  2.48s/it]
  4%|▍         | 127/3252 [05:59<2:14:57,  2.59s/it]
  4%|▍         | 128/3252 [06:01<2:19:16,  2.68s/it]
  4%|▍         | 129/3252 [06:04<2:14:59,  2.59s/it]
  4%|▍         | 130/3252 [06:07<2:19:23,  2.68s/it]
                                                    
{'loss': 0.9058, 'grad_norm': 0.46252304315567017, 'learning_rate': 0.00019995530000403377, 'lm_loss': 0.7513708472251892, 'epoch': 0.16}

  4%|▍         | 130/3252 [06:07<2:19:23,  2.68s/it]
  4%|▍         | 131/3252 [06:09<2:15:28,  2.60s/it]
  4%|▍         | 132/3252 [06:12<2:12:02,  2.54s/it]
  4%|▍         | 133/3252 [06:14<2:09:44,  2.50s/it]
  4%|▍         | 134/3252 [06:17<2:10:56,  2.52s/it]
  4%|▍         | 135/3252 [06:19<2:09:33,  2.49s/it]
  4%|▍         | 136/3252 [06:21<2:08:05,  2.47s/it]
  4%|▍         | 137/3252 [06:24<2:07:07,  2.45s/it]
  4%|▍         | 138/3252 [06:26<2:06:32,  2.44s/it]
  4%|▍         | 139/3252 [06:29<2:05:51,  2.43s/it]
  4%|▍         | 140/3252 [06:31<2:05:22,  2.42s/it]
                                                    
{'loss': 0.8575, 'grad_norm': 0.280854195356369, 'learning_rate': 0.00019992053794546047, 'lm_loss': 0.49892234802246094, 'epoch': 0.17}

  4%|▍         | 140/3252 [06:31<2:05:22,  2.42s/it]
  4%|▍         | 141/3252 [06:34<2:09:48,  2.50s/it]
  4%|▍         | 142/3252 [06:36<2:08:02,  2.47s/it]
  4%|▍         | 143/3252 [06:38<2:06:44,  2.45s/it]
  4%|▍         | 144/3252 [06:41<2:05:57,  2.43s/it]
  4%|▍         | 145/3252 [06:43<2:07:46,  2.47s/it]
  4%|▍         | 146/3252 [06:46<2:06:36,  2.45s/it]
  5%|▍         | 147/3252 [06:48<2:07:16,  2.46s/it]
  5%|▍         | 148/3252 [06:51<2:06:20,  2.44s/it]
  5%|▍         | 149/3252 [06:53<2:06:43,  2.45s/it]
  5%|▍         | 150/3252 [06:56<2:05:43,  2.43s/it]
                                                    
{'loss': 0.8027, 'grad_norm': 0.2820449471473694, 'learning_rate': 0.0001998758497903195, 'lm_loss': 0.7722914218902588, 'epoch': 0.18}

  5%|▍         | 150/3252 [06:56<2:05:43,  2.43s/it]
  5%|▍         | 151/3252 [06:59<2:15:12,  2.62s/it]
  5%|▍         | 152/3252 [07:01<2:11:36,  2.55s/it]
  5%|▍         | 153/3252 [07:03<2:09:22,  2.50s/it]
  5%|▍         | 154/3252 [07:06<2:07:47,  2.47s/it]
  5%|▍         | 155/3252 [07:08<2:06:31,  2.45s/it]
  5%|▍         | 156/3252 [07:11<2:05:54,  2.44s/it]
  5%|▍         | 157/3252 [07:13<2:05:12,  2.43s/it]
  5%|▍         | 158/3252 [07:15<2:04:48,  2.42s/it]
  5%|▍         | 159/3252 [07:18<2:04:53,  2.42s/it]
  5%|▍         | 160/3252 [07:20<2:04:23,  2.41s/it]
                                                    
{'loss': 0.7992, 'grad_norm': 1.0678768157958984, 'learning_rate': 0.00019982123997792787, 'lm_loss': 0.48010188341140747, 'epoch': 0.2}

  5%|▍         | 160/3252 [07:20<2:04:23,  2.41s/it]
  5%|▍         | 161/3252 [07:23<2:09:21,  2.51s/it]
  5%|▍         | 162/3252 [07:25<2:07:32,  2.48s/it]
  5%|▌         | 163/3252 [07:28<2:06:17,  2.45s/it]
  5%|▌         | 164/3252 [07:30<2:06:59,  2.47s/it]
  5%|▌         | 165/3252 [07:33<2:05:49,  2.45s/it]
  5%|▌         | 166/3252 [07:35<2:06:22,  2.46s/it]
  5%|▌         | 167/3252 [07:38<2:05:24,  2.44s/it]
  5%|▌         | 168/3252 [07:40<2:04:53,  2.43s/it]
  5%|▌         | 169/3252 [07:42<2:04:39,  2.43s/it]
  5%|▌         | 170/3252 [07:45<2:04:13,  2.42s/it]
                                                    
{'loss': 0.7604, 'grad_norm': 0.20208978652954102, 'learning_rate': 0.0001997567139332191, 'lm_loss': 1.9321194887161255, 'epoch': 0.21}

  5%|▌         | 170/3252 [07:45<2:04:13,  2.42s/it]
  5%|▌         | 171/3252 [07:47<2:06:50,  2.47s/it]
  5%|▌         | 172/3252 [07:50<2:05:34,  2.45s/it]
  5%|▌         | 173/3252 [07:52<2:04:45,  2.43s/it]
  5%|▌         | 174/3252 [07:55<2:04:04,  2.42s/it]
  5%|▌         | 175/3252 [07:57<2:03:33,  2.41s/it]
  5%|▌         | 176/3252 [07:59<2:03:08,  2.40s/it]
  5%|▌         | 177/3252 [08:02<2:02:56,  2.40s/it]
  5%|▌         | 178/3252 [08:04<2:02:47,  2.40s/it]
  6%|▌         | 179/3252 [08:07<2:02:54,  2.40s/it]
  6%|▌         | 180/3252 [08:09<2:02:45,  2.40s/it]
                                                    
{'loss': 0.7451, 'grad_norm': 0.20778371393680573, 'learning_rate': 0.0001996822780662041, 'lm_loss': 0.6463106274604797, 'epoch': 0.22}

  6%|▌         | 180/3252 [08:09<2:02:45,  2.40s/it]
  6%|▌         | 181/3252 [08:11<2:04:09,  2.43s/it]
  6%|▌         | 182/3252 [08:14<2:03:51,  2.42s/it]
  6%|▌         | 183/3252 [08:16<2:03:29,  2.41s/it]
  6%|▌         | 184/3252 [08:19<2:06:24,  2.47s/it]
  6%|▌         | 185/3252 [08:21<2:05:05,  2.45s/it]
  6%|▌         | 186/3252 [08:24<2:04:19,  2.43s/it]
  6%|▌         | 187/3252 [08:26<2:03:44,  2.42s/it]
  6%|▌         | 188/3252 [08:28<2:03:19,  2.42s/it]
  6%|▌         | 189/3252 [08:31<2:02:54,  2.41s/it]
  6%|▌         | 190/3252 [08:33<2:02:43,  2.40s/it]
                                                    
{'loss': 0.7366, 'grad_norm': 0.2087080031633377, 'learning_rate': 0.00019959793977133484, 'lm_loss': 0.27385222911834717, 'epoch': 0.23}

  6%|▌         | 190/3252 [08:33<2:02:43,  2.40s/it]
  6%|▌         | 191/3252 [08:36<2:03:36,  2.42s/it]
  6%|▌         | 192/3252 [08:38<2:03:05,  2.41s/it]
  6%|▌         | 193/3252 [08:40<2:02:43,  2.41s/it]
  6%|▌         | 194/3252 [08:43<2:02:24,  2.40s/it]
  6%|▌         | 195/3252 [08:45<2:02:13,  2.40s/it]
  6%|▌         | 196/3252 [08:48<2:03:07,  2.42s/it]
  6%|▌         | 197/3252 [08:50<2:02:32,  2.41s/it]
  6%|▌         | 198/3252 [08:52<2:02:14,  2.40s/it]
  6%|▌         | 199/3252 [08:55<2:01:54,  2.40s/it]
  6%|▌         | 200/3252 [08:57<2:01:46,  2.39s/it]
                                                    
{'loss': 0.735, 'grad_norm': 0.22798925638198853, 'learning_rate': 0.00019950370742676928, 'lm_loss': 0.49788013100624084, 'epoch': 0.25}

  6%|▌         | 200/3252 [08:57<2:01:46,  2.39s/it]
  6%|▌         | 201/3252 [09:00<2:03:38,  2.43s/it]
  6%|▌         | 202/3252 [09:02<2:02:54,  2.42s/it]
  6%|▌         | 203/3252 [09:05<2:04:25,  2.45s/it]
  6%|▋         | 204/3252 [09:07<2:08:48,  2.54s/it]
  6%|▋         | 205/3252 [09:10<2:06:34,  2.49s/it]
  6%|▋         | 206/3252 [09:12<2:05:07,  2.46s/it]
  6%|▋         | 207/3252 [09:15<2:04:07,  2.45s/it]
  6%|▋         | 208/3252 [09:17<2:03:17,  2.43s/it]
  6%|▋         | 209/3252 [09:19<2:02:50,  2.42s/it]
  6%|▋         | 210/3252 [09:22<2:02:29,  2.42s/it]
                                                    
{'loss': 0.713, 'grad_norm': 0.20442479848861694, 'learning_rate': 0.00019939959039353946, 'lm_loss': 0.5046742558479309, 'epoch': 0.26}

  6%|▋         | 210/3252 [09:22<2:02:29,  2.42s/it]
  6%|▋         | 211/3252 [09:24<2:02:37,  2.42s/it]
  7%|▋         | 212/3252 [09:27<2:02:26,  2.42s/it]
  7%|▋         | 213/3252 [09:29<2:02:12,  2.41s/it]
  7%|▋         | 214/3252 [09:31<2:01:57,  2.41s/it]
  7%|▋         | 215/3252 [09:34<2:01:44,  2.41s/it]
  7%|▋         | 216/3252 [09:36<2:01:33,  2.40s/it]
  7%|▋         | 217/3252 [09:39<2:01:29,  2.40s/it]
  7%|▋         | 218/3252 [09:41<2:01:14,  2.40s/it]
  7%|▋         | 219/3252 [09:43<2:01:07,  2.40s/it]
  7%|▋         | 220/3252 [09:46<2:01:01,  2.40s/it]
                                                    
{'loss': 0.6791, 'grad_norm': 0.19677932560443878, 'learning_rate': 0.00019928559901462133, 'lm_loss': 0.6121444702148438, 'epoch': 0.27}

  7%|▋         | 220/3252 [09:46<2:01:01,  2.40s/it]
  7%|▋         | 221/3252 [09:48<2:04:09,  2.46s/it]
  7%|▋         | 222/3252 [09:51<2:03:10,  2.44s/it]
  7%|▋         | 223/3252 [09:53<2:03:51,  2.45s/it]
  7%|▋         | 224/3252 [09:56<2:02:57,  2.44s/it]
  7%|▋         | 225/3252 [09:58<2:02:08,  2.42s/it]
  7%|▋         | 226/3252 [10:00<2:01:36,  2.41s/it]
  7%|▋         | 227/3252 [10:03<2:01:21,  2.41s/it]
  7%|▋         | 228/3252 [10:05<2:01:08,  2.40s/it]
  7%|▋         | 229/3252 [10:08<2:00:56,  2.40s/it]
  7%|▋         | 230/3252 [10:10<2:00:50,  2.40s/it]
                                                    
{'loss': 0.6806, 'grad_norm': 0.20167040824890137, 'learning_rate': 0.00019916174461390745, 'lm_loss': 1.7421529293060303, 'epoch': 0.28}

  7%|▋         | 230/3252 [10:10<2:00:50,  2.40s/it]
  7%|▋         | 231/3252 [10:13<2:03:38,  2.46s/it]
  7%|▋         | 232/3252 [10:15<2:02:40,  2.44s/it]
  7%|▋         | 233/3252 [10:17<2:02:01,  2.43s/it]
  7%|▋         | 234/3252 [10:20<2:01:39,  2.42s/it]
  7%|▋         | 235/3252 [10:22<2:01:56,  2.42s/it]
  7%|▋         | 236/3252 [10:25<2:01:32,  2.42s/it]
  7%|▋         | 237/3252 [10:27<2:01:17,  2.41s/it]
  7%|▋         | 238/3252 [10:30<2:03:07,  2.45s/it]
  7%|▋         | 239/3252 [10:32<2:02:12,  2.43s/it]
  7%|▋         | 240/3252 [10:34<2:01:42,  2.42s/it]
                                                    
{'loss': 0.6765, 'grad_norm': 0.1960621327161789, 'learning_rate': 0.00019902803949508203, 'lm_loss': 0.748486340045929, 'epoch': 0.3}

  7%|▋         | 240/3252 [10:34<2:01:42,  2.42s/it]
  7%|▋         | 241/3252 [10:37<2:01:47,  2.43s/it]
  7%|▋         | 242/3252 [10:39<2:01:11,  2.42s/it]
  7%|▋         | 243/3252 [10:42<2:00:40,  2.41s/it]
  8%|▊         | 244/3252 [10:44<2:02:44,  2.45s/it]
  8%|▊         | 245/3252 [10:47<2:01:51,  2.43s/it]
  8%|▊         | 246/3252 [10:49<2:01:11,  2.42s/it]
  8%|▊         | 247/3252 [10:51<2:00:43,  2.41s/it]
  8%|▊         | 248/3252 [10:54<2:00:21,  2.40s/it]
  8%|▊         | 249/3252 [10:56<2:00:09,  2.40s/it]
  8%|▊         | 250/3252 [10:58<1:59:57,  2.40s/it]
                                                    
{'loss': 0.6838, 'grad_norm': 0.21192477643489838, 'learning_rate': 0.00019888449694039864, 'lm_loss': 0.6175100207328796, 'epoch': 0.31}

  8%|▊         | 250/3252 [10:59<1:59:57,  2.40s/it]
  8%|▊         | 251/3252 [11:01<2:00:20,  2.41s/it]
  8%|▊         | 252/3252 [11:03<2:00:08,  2.40s/it]
  8%|▊         | 253/3252 [11:06<1:59:55,  2.40s/it]
  8%|▊         | 254/3252 [11:08<1:59:49,  2.40s/it]
  8%|▊         | 255/3252 [11:11<1:59:51,  2.40s/it]
  8%|▊         | 256/3252 [11:13<1:59:44,  2.40s/it]
  8%|▊         | 257/3252 [11:15<1:59:43,  2.40s/it]
  8%|▊         | 258/3252 [11:18<2:01:07,  2.43s/it]
  8%|▊         | 259/3252 [11:20<2:01:09,  2.43s/it]
  8%|▊         | 260/3252 [11:23<2:00:37,  2.42s/it]
                                                    
{'loss': 0.663, 'grad_norm': 0.3682516813278198, 'learning_rate': 0.00019873113120936074, 'lm_loss': 0.5504544973373413, 'epoch': 0.32}

  8%|▊         | 260/3252 [11:23<2:00:37,  2.42s/it]
  8%|▊         | 261/3252 [11:25<2:01:55,  2.45s/it]
  8%|▊         | 262/3252 [11:28<2:01:17,  2.43s/it]
  8%|▊         | 263/3252 [11:30<2:00:41,  2.42s/it]
  8%|▊         | 264/3252 [11:32<2:00:16,  2.42s/it]
  8%|▊         | 265/3252 [11:35<2:00:10,  2.41s/it]
  8%|▊         | 266/3252 [11:37<1:59:55,  2.41s/it]
  8%|▊         | 267/3252 [11:40<1:59:38,  2.40s/it]
  8%|▊         | 268/3252 [11:42<1:59:32,  2.40s/it]
  8%|▊         | 269/3252 [11:44<1:59:23,  2.40s/it]
  8%|▊         | 270/3252 [11:47<1:59:11,  2.40s/it]
                                                    
{'loss': 0.6652, 'grad_norm': 0.36262527108192444, 'learning_rate': 0.0001985679575373052, 'lm_loss': 0.6348951458930969, 'epoch': 0.33}

  8%|▊         | 270/3252 [11:47<1:59:11,  2.40s/it]
  8%|▊         | 271/3252 [11:49<1:59:22,  2.40s/it]
  8%|▊         | 272/3252 [11:52<1:59:12,  2.40s/it]
  8%|▊         | 273/3252 [11:54<1:59:01,  2.40s/it]
  8%|▊         | 274/3252 [11:56<1:59:08,  2.40s/it]
  8%|▊         | 275/3252 [11:59<1:59:09,  2.40s/it]
  8%|▊         | 276/3252 [12:01<1:58:55,  2.40s/it]
  9%|▊         | 277/3252 [12:04<1:58:50,  2.40s/it]
  9%|▊         | 278/3252 [12:06<2:05:03,  2.52s/it]
  9%|▊         | 279/3252 [12:09<2:03:01,  2.48s/it]
  9%|▊         | 280/3252 [12:11<2:01:42,  2.46s/it]
                                                    
{'loss': 0.6802, 'grad_norm': 0.22686289250850677, 'learning_rate': 0.00019839499213388878, 'lm_loss': 0.4849565625190735, 'epoch': 0.34}

  9%|▊         | 280/3252 [12:11<2:01:42,  2.46s/it]
  9%|▊         | 281/3252 [12:14<2:04:15,  2.51s/it]
  9%|▊         | 282/3252 [12:16<2:02:49,  2.48s/it]
  9%|▊         | 283/3252 [12:19<2:01:27,  2.45s/it]
  9%|▊         | 284/3252 [12:21<2:00:40,  2.44s/it]
  9%|▉         | 285/3252 [12:23<2:00:08,  2.43s/it]
  9%|▉         | 286/3252 [12:26<1:59:35,  2.42s/it]
  9%|▉         | 287/3252 [12:28<1:59:24,  2.42s/it]
  9%|▉         | 288/3252 [12:31<1:59:16,  2.41s/it]
  9%|▉         | 289/3252 [12:33<1:58:54,  2.41s/it]
  9%|▉         | 290/3252 [12:35<1:58:42,  2.40s/it]
                                                    
{'loss': 0.666, 'grad_norm': 0.4202888309955597, 'learning_rate': 0.000198212252181478, 'lm_loss': 0.8318759799003601, 'epoch': 0.36}

  9%|▉         | 290/3252 [12:35<1:58:42,  2.40s/it]
  9%|▉         | 291/3252 [12:38<1:58:51,  2.41s/it]
  9%|▉         | 292/3252 [12:40<1:58:39,  2.41s/it]
  9%|▉         | 293/3252 [12:43<1:58:39,  2.41s/it]
  9%|▉         | 294/3252 [12:45<1:58:44,  2.41s/it]
  9%|▉         | 295/3252 [12:47<1:58:33,  2.41s/it]
  9%|▉         | 296/3252 [12:50<1:58:18,  2.40s/it]
  9%|▉         | 297/3252 [12:52<2:00:00,  2.44s/it]
  9%|▉         | 298/3252 [12:55<1:59:11,  2.42s/it]
  9%|▉         | 299/3252 [12:57<1:58:33,  2.41s/it]
  9%|▉         | 300/3252 [13:00<1:58:35,  2.41s/it]
                                                    
{'loss': 0.6367, 'grad_norm': 0.45160332322120667, 'learning_rate': 0.000198019755833442, 'lm_loss': 0.7162055373191833, 'epoch': 0.37}

  9%|▉         | 300/3252 [13:00<1:58:35,  2.41s/it]
  9%|▉         | 301/3252 [13:02<1:58:23,  2.41s/it]
  9%|▉         | 302/3252 [13:04<1:58:15,  2.41s/it]
  9%|▉         | 303/3252 [13:07<1:58:02,  2.40s/it]
  9%|▉         | 304/3252 [13:09<1:57:56,  2.40s/it]
  9%|▉         | 305/3252 [13:11<1:57:52,  2.40s/it]
  9%|▉         | 306/3252 [13:14<1:57:41,  2.40s/it]
  9%|▉         | 307/3252 [13:16<1:57:46,  2.40s/it]
  9%|▉         | 308/3252 [13:19<1:57:42,  2.40s/it]
 10%|▉         | 309/3252 [13:21<1:57:37,  2.40s/it]
 10%|▉         | 310/3252 [13:23<1:57:42,  2.40s/it]
                                                    
{'loss': 0.6373, 'grad_norm': 0.32857778668403625, 'learning_rate': 0.00019781752221234932, 'lm_loss': 0.7886148691177368, 'epoch': 0.38}

 10%|▉         | 310/3252 [13:24<1:57:42,  2.40s/it]
 10%|▉         | 311/3252 [13:26<1:59:26,  2.44s/it]
 10%|▉         | 312/3252 [13:29<2:00:35,  2.46s/it]
 10%|▉         | 313/3252 [13:31<1:59:38,  2.44s/it]
 10%|▉         | 314/3252 [13:33<1:58:51,  2.43s/it]
 10%|▉         | 315/3252 [13:36<1:58:26,  2.42s/it]
 10%|▉         | 316/3252 [13:38<1:57:58,  2.41s/it]
 10%|▉         | 317/3252 [13:41<1:57:41,  2.41s/it]
 10%|▉         | 318/3252 [13:43<1:57:24,  2.40s/it]
 10%|▉         | 319/3252 [13:45<1:57:12,  2.40s/it]
 10%|▉         | 320/3252 [13:48<1:57:04,  2.40s/it]
                                                    
{'loss': 0.627, 'grad_norm': 0.2786678671836853, 'learning_rate': 0.00019760557140806838, 'lm_loss': 0.5034598112106323, 'epoch': 0.39}

 10%|▉         | 320/3252 [13:48<1:57:04,  2.40s/it]
 10%|▉         | 321/3252 [13:50<1:57:32,  2.41s/it]
 10%|▉         | 322/3252 [13:52<1:57:17,  2.40s/it]
 10%|▉         | 323/3252 [13:55<1:57:01,  2.40s/it]
 10%|▉         | 324/3252 [13:57<1:56:52,  2.40s/it]
 10%|▉         | 325/3252 [14:00<1:56:42,  2.39s/it]
 10%|█         | 326/3252 [14:02<1:56:40,  2.39s/it]
 10%|█         | 327/3252 [14:04<1:56:32,  2.39s/it]
 10%|█         | 328/3252 [14:07<1:56:42,  2.39s/it]
 10%|█         | 329/3252 [14:09<1:56:38,  2.39s/it]
 10%|█         | 330/3252 [14:12<1:56:38,  2.40s/it]
                                                    
{'loss': 0.6339, 'grad_norm': 0.47669708728790283, 'learning_rate': 0.00019738392447577147, 'lm_loss': 0.6805928945541382, 'epoch': 0.41}

 10%|█         | 330/3252 [14:12<1:56:38,  2.40s/it]
 10%|█         | 331/3252 [14:14<1:58:25,  2.43s/it]
 10%|█         | 332/3252 [14:17<1:57:54,  2.42s/it]
 10%|█         | 333/3252 [14:19<1:57:29,  2.41s/it]
 10%|█         | 334/3252 [14:21<1:57:13,  2.41s/it]
 10%|█         | 335/3252 [14:24<1:57:00,  2.41s/it]
 10%|█         | 336/3252 [14:26<1:56:57,  2.41s/it]
 10%|█         | 337/3252 [14:29<1:56:44,  2.40s/it]
 10%|█         | 338/3252 [14:31<1:56:42,  2.40s/it]
 10%|█         | 339/3252 [14:33<1:57:13,  2.41s/it]
 10%|█         | 340/3252 [14:36<1:56:54,  2.41s/it]
                                                    
{'loss': 0.6414, 'grad_norm': 0.24972008168697357, 'learning_rate': 0.00019715260343384347, 'lm_loss': 0.417803555727005, 'epoch': 0.42}

 10%|█         | 340/3252 [14:36<1:56:54,  2.41s/it]
 10%|█         | 341/3252 [14:38<1:58:03,  2.43s/it]
 11%|█         | 342/3252 [14:41<1:57:29,  2.42s/it]
 11%|█         | 343/3252 [14:43<1:56:57,  2.41s/it]
 11%|█         | 344/3252 [14:45<1:56:59,  2.41s/it]
 11%|█         | 345/3252 [14:48<1:56:42,  2.41s/it]
 11%|█         | 346/3252 [14:50<1:56:23,  2.40s/it]
 11%|█         | 347/3252 [14:53<1:56:06,  2.40s/it]
 11%|█         | 348/3252 [14:55<1:55:54,  2.39s/it]
 11%|█         | 349/3252 [14:57<1:55:56,  2.40s/it]
 11%|█         | 350/3252 [15:00<1:55:56,  2.40s/it]
                                                    
{'loss': 0.6021, 'grad_norm': 0.22659756243228912, 'learning_rate': 0.00019691163126169428, 'lm_loss': 1.6529167890548706, 'epoch': 0.43}

 11%|█         | 350/3252 [15:00<1:55:56,  2.40s/it]
 11%|█         | 351/3252 [15:02<1:57:24,  2.43s/it]
 11%|█         | 352/3252 [15:05<2:00:42,  2.50s/it]
 11%|█         | 353/3252 [15:07<1:59:09,  2.47s/it]
 11%|█         | 354/3252 [15:10<1:57:59,  2.44s/it]
 11%|█         | 355/3252 [15:12<1:57:21,  2.43s/it]
 11%|█         | 356/3252 [15:15<1:56:49,  2.42s/it]
 11%|█         | 357/3252 [15:17<1:56:32,  2.42s/it]
 11%|█         | 358/3252 [15:19<1:56:18,  2.41s/it]
 11%|█         | 359/3252 [15:22<1:56:13,  2.41s/it]
 11%|█         | 360/3252 [15:24<1:56:04,  2.41s/it]
                                                    
{'loss': 0.6449, 'grad_norm': 0.42482712864875793, 'learning_rate': 0.0001966610318974761, 'lm_loss': 0.5789749026298523, 'epoch': 0.44}

 11%|█         | 360/3252 [15:24<1:56:04,  2.41s/it]
 11%|█         | 361/3252 [15:27<1:58:25,  2.46s/it]
 11%|█         | 362/3252 [15:29<1:57:29,  2.44s/it]
 11%|█         | 363/3252 [15:32<1:56:49,  2.43s/it]
 11%|█         | 364/3252 [15:34<1:56:22,  2.42s/it]
 11%|█         | 365/3252 [15:36<1:56:00,  2.41s/it]
 11%|█▏        | 366/3252 [15:39<1:55:55,  2.41s/it]
 11%|█▏        | 367/3252 [15:41<1:55:33,  2.40s/it]
 11%|█▏        | 368/3252 [15:44<1:56:48,  2.43s/it]
 11%|█▏        | 369/3252 [15:46<1:56:23,  2.42s/it]
 11%|█▏        | 370/3252 [15:48<1:55:51,  2.41s/it]
                                                    
{'loss': 0.6482, 'grad_norm': 0.4018535315990448, 'learning_rate': 0.0001964008302357056, 'lm_loss': 1.6823008060455322, 'epoch': 0.46}

 11%|█▏        | 370/3252 [15:49<1:55:51,  2.41s/it]
 11%|█▏        | 371/3252 [15:51<1:58:19,  2.46s/it]
 11%|█▏        | 372/3252 [15:53<1:57:03,  2.44s/it]
 11%|█▏        | 373/3252 [15:56<1:56:14,  2.42s/it]
 12%|█▏        | 374/3252 [15:58<1:55:40,  2.41s/it]
 12%|█▏        | 375/3252 [16:01<1:55:24,  2.41s/it]
 12%|█▏        | 376/3252 [16:03<1:54:59,  2.40s/it]
 12%|█▏        | 377/3252 [16:05<1:54:59,  2.40s/it]
 12%|█▏        | 378/3252 [16:08<1:54:48,  2.40s/it]
 12%|█▏        | 379/3252 [16:10<1:54:36,  2.39s/it]
 12%|█▏        | 380/3252 [16:13<1:54:29,  2.39s/it]
                                                    
{'loss': 0.5973, 'grad_norm': 0.47940149903297424, 'learning_rate': 0.0001961310521247906, 'lm_loss': 0.627518892288208, 'epoch': 0.47}

 12%|█▏        | 380/3252 [16:13<1:54:29,  2.39s/it]
 12%|█▏        | 381/3252 [16:15<1:54:50,  2.40s/it]
 12%|█▏        | 382/3252 [16:17<1:54:41,  2.40s/it]
 12%|█▏        | 383/3252 [16:20<1:54:48,  2.40s/it]
 12%|█▏        | 384/3252 [16:22<1:54:43,  2.40s/it]
 12%|█▏        | 385/3252 [16:25<1:55:26,  2.42s/it]
 12%|█▏        | 386/3252 [16:27<1:58:33,  2.48s/it]
 12%|█▏        | 387/3252 [16:30<1:57:08,  2.45s/it]
 12%|█▏        | 388/3252 [16:32<1:56:24,  2.44s/it]
 12%|█▏        | 389/3252 [16:34<1:55:48,  2.43s/it]
 12%|█▏        | 390/3252 [16:37<1:55:17,  2.42s/it]
                                                    
{'loss': 0.5985, 'grad_norm': 0.44287705421447754, 'learning_rate': 0.00019585172436446257, 'lm_loss': 1.6898268461227417, 'epoch': 0.48}

 12%|█▏        | 390/3252 [16:37<1:55:17,  2.42s/it]
 12%|█▏        | 391/3252 [16:39<1:58:41,  2.49s/it]
 12%|█▏        | 392/3252 [16:42<1:57:11,  2.46s/it]
 12%|█▏        | 393/3252 [16:44<1:56:08,  2.44s/it]
 12%|█▏        | 394/3252 [16:47<1:55:25,  2.42s/it]
 12%|█▏        | 395/3252 [16:49<1:55:04,  2.42s/it]
 12%|█▏        | 396/3252 [16:51<1:54:45,  2.41s/it]
 12%|█▏        | 397/3252 [16:54<1:54:26,  2.41s/it]
 12%|█▏        | 398/3252 [16:56<1:54:06,  2.40s/it]
 12%|█▏        | 399/3252 [16:59<1:53:56,  2.40s/it]
 12%|█▏        | 400/3252 [17:01<1:53:46,  2.39s/it]
                                                    
{'loss': 0.591, 'grad_norm': 0.4773414731025696, 'learning_rate': 0.00019556287470311418, 'lm_loss': 0.13904161751270294, 'epoch': 0.49}

 12%|█▏        | 400/3252 [17:01<1:53:46,  2.39s/it]We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)


  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.99it/s]
                                                    

                                             
{'eval_sthsthv2/narration_val_loss': 0.6924356818199158, 'eval_sthsthv2/narration_val_runtime': 3.6266, 'eval_sthsthv2/narration_val_samples_per_second': 184.194, 'eval_sthsthv2/narration_val_steps_per_second': 0.551, 'epoch': 0.49}

 12%|█▏        | 400/3252 [17:05<1:53:46,  2.39s/it]

100%|██████████| 2/2 [00:01<00:00,  1.99it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.75it/s]
                                                    

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6380109786987305, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.1635, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 187.136, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.632, 'epoch': 0.49}

 12%|█▏        | 400/3252 [17:08<1:53:46,  2.39s/it]

100%|██████████| 2/2 [00:01<00:00,  1.75it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.98it/s]
                                                    

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.625105619430542, 'eval_holoassist/dialog-klg-sum_val_runtime': 3.0166, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 235.033, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.663, 'epoch': 0.49}

 12%|█▏        | 400/3252 [17:11<1:53:46,  2.39s/it]

100%|██████████| 2/2 [00:01<00:00,  1.98it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                    

                                     
{'eval_ego4d/narration_val_loss': 0.5742331147193909, 'eval_ego4d/narration_val_runtime': 2.2696, 'eval_ego4d/narration_val_samples_per_second': 98.254, 'eval_ego4d/narration_val_steps_per_second': 0.441, 'epoch': 0.49}

 12%|█▏        | 400/3252 [17:13<1:53:46,  2.39s/it]

100%|██████████| 1/1 [00:00<00:00,  2.80it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.49it/s]
                                                    

                                             
{'eval_ego4d/summary_val_loss': 1.148098111152649, 'eval_ego4d/summary_val_runtime': 2.9055, 'eval_ego4d/summary_val_samples_per_second': 194.112, 'eval_ego4d/summary_val_steps_per_second': 0.688, 'epoch': 0.49}

 12%|█▏        | 400/3252 [17:16<1:53:46,  2.39s/it]

100%|██████████| 2/2 [00:01<00:00,  1.49it/s]

                                             
 12%|█▏        | 401/3252 [17:19<5:33:29,  7.02s/it]
 12%|█▏        | 402/3252 [17:21<4:27:41,  5.64s/it]
 12%|█▏        | 403/3252 [17:24<3:41:30,  4.67s/it]
 12%|█▏        | 404/3252 [17:26<3:09:13,  3.99s/it]
 12%|█▏        | 405/3252 [17:29<2:48:26,  3.55s/it]
 12%|█▏        | 406/3252 [17:31<2:31:53,  3.20s/it]
 13%|█▎        | 407/3252 [17:33<2:20:28,  2.96s/it]
 13%|█▎        | 408/3252 [17:36<2:12:19,  2.79s/it]
 13%|█▎        | 409/3252 [17:38<2:06:39,  2.67s/it]
 13%|█▎        | 410/3252 [17:41<2:02:36,  2.59s/it]
                                                    
{'loss': 0.6127, 'grad_norm': 0.2275819182395935, 'learning_rate': 0.00019526453183504278, 'lm_loss': 0.3156636655330658, 'epoch': 0.5}

 13%|█▎        | 410/3252 [17:41<2:02:36,  2.59s/it]
 13%|█▎        | 411/3252 [17:43<2:04:02,  2.62s/it]
 13%|█▎        | 412/3252 [17:46<2:00:51,  2.55s/it]
 13%|█▎        | 413/3252 [17:48<1:58:29,  2.50s/it]
 13%|█▎        | 414/3252 [17:50<1:57:01,  2.47s/it]
 13%|█▎        | 415/3252 [17:53<1:55:47,  2.45s/it]
 13%|█▎        | 416/3252 [17:55<1:54:48,  2.43s/it]
 13%|█▎        | 417/3252 [17:58<1:54:10,  2.42s/it]
 13%|█▎        | 418/3252 [18:00<1:53:49,  2.41s/it]
 13%|█▎        | 419/3252 [18:02<1:53:37,  2.41s/it]
 13%|█▎        | 420/3252 [18:05<1:53:19,  2.40s/it]
                                                    
{'loss': 0.6062, 'grad_norm': 0.5651805400848389, 'learning_rate': 0.00019495672539760007, 'lm_loss': 0.43421471118927, 'epoch': 0.52}

 13%|█▎        | 420/3252 [18:05<1:53:19,  2.40s/it]
 13%|█▎        | 421/3252 [18:07<1:53:29,  2.41s/it]
 13%|█▎        | 422/3252 [18:10<1:54:49,  2.43s/it]
 13%|█▎        | 423/3252 [18:12<1:54:12,  2.42s/it]
 13%|█▎        | 424/3252 [18:14<1:53:46,  2.41s/it]
 13%|█▎        | 425/3252 [18:17<1:53:25,  2.41s/it]
 13%|█▎        | 426/3252 [18:20<1:57:49,  2.50s/it]
 13%|█▎        | 427/3252 [18:22<1:56:14,  2.47s/it]
 13%|█▎        | 428/3252 [18:24<1:55:10,  2.45s/it]
 13%|█▎        | 429/3252 [18:27<1:54:27,  2.43s/it]
 13%|█▎        | 430/3252 [18:29<1:54:10,  2.43s/it]
                                                    
{'loss': 0.5956, 'grad_norm': 0.3392122685909271, 'learning_rate': 0.00019463948596824774, 'lm_loss': 0.6162466406822205, 'epoch': 0.53}

 13%|█▎        | 430/3252 [18:29<1:54:10,  2.43s/it]
 13%|█▎        | 431/3252 [18:32<1:55:12,  2.45s/it]
 13%|█▎        | 432/3252 [18:34<1:54:29,  2.44s/it]
 13%|█▎        | 433/3252 [18:36<1:53:50,  2.42s/it]
 13%|█▎        | 434/3252 [18:39<1:53:19,  2.41s/it]
 13%|█▎        | 435/3252 [18:41<1:53:06,  2.41s/it]
 13%|█▎        | 436/3252 [18:44<1:52:51,  2.40s/it]
 13%|█▎        | 437/3252 [18:46<1:52:35,  2.40s/it]
 13%|█▎        | 438/3252 [18:48<1:52:28,  2.40s/it]
 13%|█▎        | 439/3252 [18:51<1:52:23,  2.40s/it]
 14%|█▎        | 440/3252 [18:53<1:52:21,  2.40s/it]
                                                    
{'loss': 0.6085, 'grad_norm': 0.23073139786720276, 'learning_rate': 0.00019431284506151995, 'lm_loss': 0.10168452560901642, 'epoch': 0.54}

 14%|█▎        | 440/3252 [18:53<1:52:21,  2.40s/it]
 14%|█▎        | 441/3252 [18:56<1:53:39,  2.43s/it]
 14%|█▎        | 442/3252 [18:58<1:53:17,  2.42s/it]
 14%|█▎        | 443/3252 [19:01<1:55:21,  2.46s/it]
 14%|█▎        | 444/3252 [19:03<1:55:18,  2.46s/it]
 14%|█▎        | 445/3252 [19:06<1:54:15,  2.44s/it]
 14%|█▎        | 446/3252 [19:08<1:53:46,  2.43s/it]
 14%|█▎        | 447/3252 [19:10<1:53:11,  2.42s/it]
 14%|█▍        | 448/3252 [19:13<1:52:47,  2.41s/it]
 14%|█▍        | 449/3252 [19:15<1:52:26,  2.41s/it]
 14%|█▍        | 450/3252 [19:18<1:52:10,  2.40s/it]
                                                    
{'loss': 0.5973, 'grad_norm': 0.42176660895347595, 'learning_rate': 0.0001939768351258928, 'lm_loss': 0.6505115628242493, 'epoch': 0.55}

 14%|█▍        | 450/3252 [19:18<1:52:10,  2.40s/it]
 14%|█▍        | 451/3252 [19:20<1:52:28,  2.41s/it]
 14%|█▍        | 452/3252 [19:22<1:52:55,  2.42s/it]
 14%|█▍        | 453/3252 [19:29<2:54:10,  3.73s/it]
 14%|█▍        | 454/3252 [19:32<2:35:28,  3.33s/it]
 14%|█▍        | 455/3252 [19:34<2:22:11,  3.05s/it]
 14%|█▍        | 456/3252 [19:36<2:13:01,  2.85s/it]
 14%|█▍        | 457/3252 [19:39<2:06:32,  2.72s/it]
 14%|█▍        | 458/3252 [19:41<2:02:07,  2.62s/it]
 14%|█▍        | 459/3252 [19:44<1:59:00,  2.56s/it]
 14%|█▍        | 460/3252 [19:46<1:56:46,  2.51s/it]
                                                    
{'loss': 0.5859, 'grad_norm': 0.3027350604534149, 'learning_rate': 0.00019363148954056077, 'lm_loss': 0.8104124069213867, 'epoch': 0.57}

 14%|█▍        | 460/3252 [19:46<1:56:46,  2.51s/it]
 14%|█▍        | 461/3252 [19:48<1:55:26,  2.48s/it]
 14%|█▍        | 462/3252 [19:51<1:57:23,  2.52s/it]
 14%|█▍        | 463/3252 [19:53<1:56:14,  2.50s/it]
 14%|█▍        | 464/3252 [19:56<1:54:43,  2.47s/it]
 14%|█▍        | 465/3252 [19:58<1:53:38,  2.45s/it]
 14%|█▍        | 466/3252 [20:01<1:52:59,  2.43s/it]
 14%|█▍        | 467/3252 [20:03<1:52:17,  2.42s/it]
 14%|█▍        | 468/3252 [20:05<1:51:57,  2.41s/it]
 14%|█▍        | 469/3252 [20:08<1:51:50,  2.41s/it]
 14%|█▍        | 470/3252 [20:10<1:51:34,  2.41s/it]
                                                    
{'loss': 0.593, 'grad_norm': 0.32552269101142883, 'learning_rate': 0.00019327684261212085, 'lm_loss': 0.45642831921577454, 'epoch': 0.58}

 14%|█▍        | 470/3252 [20:10<1:51:34,  2.41s/it]
 14%|█▍        | 471/3252 [20:13<1:51:54,  2.41s/it]
 15%|█▍        | 472/3252 [20:15<1:51:26,  2.41s/it]
 15%|█▍        | 473/3252 [20:17<1:51:04,  2.40s/it]
 15%|█▍        | 474/3252 [20:20<1:50:55,  2.40s/it]
 15%|█▍        | 475/3252 [20:22<1:50:55,  2.40s/it]
 15%|█▍        | 476/3252 [20:25<1:50:59,  2.40s/it]
 15%|█▍        | 477/3252 [20:27<1:50:50,  2.40s/it]
 15%|█▍        | 478/3252 [20:29<1:50:42,  2.39s/it]
 15%|█▍        | 479/3252 [20:32<1:52:53,  2.44s/it]
 15%|█▍        | 480/3252 [20:34<1:52:05,  2.43s/it]
                                                    
{'loss': 0.5911, 'grad_norm': 0.3296608030796051, 'learning_rate': 0.0001929129295711646, 'lm_loss': 0.07749775052070618, 'epoch': 0.59}

 15%|█▍        | 480/3252 [20:35<1:52:05,  2.43s/it]
 15%|█▍        | 481/3252 [20:37<1:56:17,  2.52s/it]
 15%|█▍        | 482/3252 [20:39<1:54:33,  2.48s/it]
 15%|█▍        | 483/3252 [20:42<1:53:18,  2.46s/it]
 15%|█▍        | 484/3252 [20:44<1:52:28,  2.44s/it]
 15%|█▍        | 485/3252 [20:47<1:51:50,  2.43s/it]
 15%|█▍        | 486/3252 [20:49<1:51:17,  2.41s/it]
 15%|█▍        | 487/3252 [20:51<1:50:57,  2.41s/it]
 15%|█▌        | 488/3252 [20:54<1:50:40,  2.40s/it]
 15%|█▌        | 489/3252 [20:56<1:50:34,  2.40s/it]
 15%|█▌        | 490/3252 [20:59<1:50:27,  2.40s/it]
                                                    
{'loss': 0.5606, 'grad_norm': 0.2089587152004242, 'learning_rate': 0.00019253978656877833, 'lm_loss': 1.5750391483306885, 'epoch': 0.6}

 15%|█▌        | 490/3252 [20:59<1:50:27,  2.40s/it]
 15%|█▌        | 491/3252 [21:01<1:50:32,  2.40s/it]
 15%|█▌        | 492/3252 [21:03<1:50:23,  2.40s/it]
 15%|█▌        | 493/3252 [21:06<1:50:16,  2.40s/it]
 15%|█▌        | 494/3252 [21:08<1:50:08,  2.40s/it]
 15%|█▌        | 495/3252 [21:11<1:50:03,  2.40s/it]
 15%|█▌        | 496/3252 [21:13<1:49:57,  2.39s/it]
 15%|█▌        | 497/3252 [21:15<1:50:47,  2.41s/it]
 15%|█▌        | 498/3252 [21:18<1:50:22,  2.40s/it]
 15%|█▌        | 499/3252 [21:20<1:50:04,  2.40s/it]
 15%|█▌        | 500/3252 [21:23<1:54:00,  2.49s/it]
                                                    
{'loss': 0.5742, 'grad_norm': 0.1902119517326355, 'learning_rate': 0.00019215745067295169, 'lm_loss': 0.6918820738792419, 'epoch': 0.62}

 15%|█▌        | 500/3252 [21:23<1:54:00,  2.49s/it]
 15%|█▌        | 501/3252 [21:26<1:58:58,  2.59s/it]
 15%|█▌        | 502/3252 [21:28<1:56:08,  2.53s/it]
 15%|█▌        | 503/3252 [21:31<1:54:19,  2.50s/it]
 15%|█▌        | 504/3252 [21:33<1:52:40,  2.46s/it]
 16%|█▌        | 505/3252 [21:35<1:51:42,  2.44s/it]
 16%|█▌        | 506/3252 [21:38<1:51:00,  2.43s/it]
 16%|█▌        | 507/3252 [21:40<1:50:44,  2.42s/it]
 16%|█▌        | 508/3252 [21:43<1:50:35,  2.42s/it]
 16%|█▌        | 509/3252 [21:45<1:50:11,  2.41s/it]
 16%|█▌        | 510/3252 [21:47<1:49:54,  2.41s/it]
                                                    
{'loss': 0.5958, 'grad_norm': 0.18871881067752838, 'learning_rate': 0.0001917659598648956, 'lm_loss': 0.3508395552635193, 'epoch': 0.63}

 16%|█▌        | 510/3252 [21:48<1:49:54,  2.41s/it]
 16%|█▌        | 511/3252 [21:50<1:51:32,  2.44s/it]
 16%|█▌        | 512/3252 [21:52<1:50:51,  2.43s/it]
 16%|█▌        | 513/3252 [21:55<1:50:20,  2.42s/it]
 16%|█▌        | 514/3252 [21:57<1:50:01,  2.41s/it]
 16%|█▌        | 515/3252 [21:59<1:49:44,  2.41s/it]
 16%|█▌        | 516/3252 [22:02<1:49:36,  2.40s/it]
 16%|█▌        | 517/3252 [22:04<1:49:35,  2.40s/it]
 16%|█▌        | 518/3252 [22:07<1:50:37,  2.43s/it]
 16%|█▌        | 519/3252 [22:09<1:50:07,  2.42s/it]
 16%|█▌        | 520/3252 [22:12<1:49:49,  2.41s/it]
                                                    
{'loss': 0.5602, 'grad_norm': 0.19885267317295074, 'learning_rate': 0.00019136535303526902, 'lm_loss': 0.09630747139453888, 'epoch': 0.64}

 16%|█▌        | 520/3252 [22:12<1:49:49,  2.41s/it]
 16%|█▌        | 521/3252 [22:14<1:49:57,  2.42s/it]
 16%|█▌        | 522/3252 [22:16<1:49:39,  2.41s/it]
 16%|█▌        | 523/3252 [22:19<1:49:15,  2.40s/it]
 16%|█▌        | 524/3252 [22:21<1:49:19,  2.40s/it]
 16%|█▌        | 525/3252 [22:24<1:49:10,  2.40s/it]
 16%|█▌        | 526/3252 [22:26<1:49:03,  2.40s/it]
 16%|█▌        | 527/3252 [22:28<1:49:02,  2.40s/it]
 16%|█▌        | 528/3252 [22:31<1:48:53,  2.40s/it]
 16%|█▋        | 529/3252 [22:33<1:48:59,  2.40s/it]
 16%|█▋        | 530/3252 [22:36<1:48:51,  2.40s/it]
                                                    
{'loss': 0.557, 'grad_norm': 0.3869875967502594, 'learning_rate': 0.00019095566998031572, 'lm_loss': 1.5336002111434937, 'epoch': 0.65}

 16%|█▋        | 530/3252 [22:36<1:48:51,  2.40s/it]
 16%|█▋        | 531/3252 [22:38<1:49:55,  2.42s/it]
 16%|█▋        | 532/3252 [22:40<1:49:27,  2.41s/it]
 16%|█▋        | 533/3252 [22:43<1:49:09,  2.41s/it]
 16%|█▋        | 534/3252 [22:45<1:49:05,  2.41s/it]
 16%|█▋        | 535/3252 [22:48<1:48:52,  2.40s/it]
 16%|█▋        | 536/3252 [22:50<1:50:29,  2.44s/it]
 17%|█▋        | 537/3252 [22:53<1:51:46,  2.47s/it]
 17%|█▋        | 538/3252 [22:55<1:50:40,  2.45s/it]
 17%|█▋        | 539/3252 [22:57<1:49:52,  2.43s/it]
 17%|█▋        | 540/3252 [23:00<1:49:33,  2.42s/it]
                                                    
{'loss': 0.5665, 'grad_norm': 0.1889137625694275, 'learning_rate': 0.0001905369513979106, 'lm_loss': 0.7567183375358582, 'epoch': 0.66}

 17%|█▋        | 540/3252 [23:00<1:49:33,  2.42s/it]
 17%|█▋        | 541/3252 [23:02<1:49:34,  2.42s/it]
 17%|█▋        | 542/3252 [23:05<1:49:13,  2.42s/it]
 17%|█▋        | 543/3252 [23:07<1:48:47,  2.41s/it]
 17%|█▋        | 544/3252 [23:09<1:48:35,  2.41s/it]
 17%|█▋        | 545/3252 [23:12<1:48:23,  2.40s/it]
 17%|█▋        | 546/3252 [23:14<1:48:21,  2.40s/it]
 17%|█▋        | 547/3252 [23:17<1:48:09,  2.40s/it]
 17%|█▋        | 548/3252 [23:19<1:48:04,  2.40s/it]
 17%|█▋        | 549/3252 [23:21<1:47:53,  2.39s/it]
 17%|█▋        | 550/3252 [23:24<1:47:44,  2.39s/it]
                                                    
{'loss': 0.5769, 'grad_norm': 0.3010273575782776, 'learning_rate': 0.00019010923888351718, 'lm_loss': 0.4817933142185211, 'epoch': 0.68}

 17%|█▋        | 550/3252 [23:24<1:47:44,  2.39s/it]
 17%|█▋        | 551/3252 [23:26<1:48:15,  2.40s/it]
 17%|█▋        | 552/3252 [23:29<1:48:10,  2.40s/it]
 17%|█▋        | 553/3252 [23:31<1:49:41,  2.44s/it]
 17%|█▋        | 554/3252 [23:34<1:49:11,  2.43s/it]
 17%|█▋        | 555/3252 [23:36<1:48:48,  2.42s/it]
 17%|█▋        | 556/3252 [23:38<1:48:27,  2.41s/it]
 17%|█▋        | 557/3252 [23:41<1:48:03,  2.41s/it]
 17%|█▋        | 558/3252 [23:43<1:47:54,  2.40s/it]
 17%|█▋        | 559/3252 [23:46<1:47:45,  2.40s/it]
 17%|█▋        | 560/3252 [23:48<1:47:39,  2.40s/it]
                                                    
{'loss': 0.5593, 'grad_norm': 0.19463151693344116, 'learning_rate': 0.00018967257492605522, 'lm_loss': 0.4877810478210449, 'epoch': 0.69}

 17%|█▋        | 560/3252 [23:48<1:47:39,  2.40s/it]
 17%|█▋        | 561/3252 [23:50<1:48:11,  2.41s/it]
 17%|█▋        | 562/3252 [23:53<1:47:53,  2.41s/it]
 17%|█▋        | 563/3252 [23:55<1:47:42,  2.40s/it]
 17%|█▋        | 564/3252 [23:58<1:47:35,  2.40s/it]
 17%|█▋        | 565/3252 [24:00<1:47:30,  2.40s/it]
 17%|█▋        | 566/3252 [24:02<1:47:25,  2.40s/it]
 17%|█▋        | 567/3252 [24:05<1:47:17,  2.40s/it]
 17%|█▋        | 568/3252 [24:07<1:47:18,  2.40s/it]
 17%|█▋        | 569/3252 [24:10<1:47:20,  2.40s/it]
 18%|█▊        | 570/3252 [24:12<1:47:27,  2.40s/it]
                                                    
{'loss': 0.5618, 'grad_norm': 0.18113256990909576, 'learning_rate': 0.00018922700290368, 'lm_loss': 0.3906784951686859, 'epoch': 0.7}

 18%|█▊        | 570/3252 [24:12<1:47:27,  2.40s/it]
 18%|█▊        | 571/3252 [24:14<1:48:10,  2.42s/it]
 18%|█▊        | 572/3252 [24:17<1:47:44,  2.41s/it]
 18%|█▊        | 573/3252 [24:19<1:47:34,  2.41s/it]
 18%|█▊        | 574/3252 [24:22<1:51:51,  2.51s/it]
 18%|█▊        | 575/3252 [24:24<1:50:19,  2.47s/it]
 18%|█▊        | 576/3252 [24:27<1:49:13,  2.45s/it]
 18%|█▊        | 577/3252 [24:29<1:48:28,  2.43s/it]
 18%|█▊        | 578/3252 [24:32<1:47:52,  2.42s/it]
 18%|█▊        | 579/3252 [24:34<1:47:35,  2.42s/it]
 18%|█▊        | 580/3252 [24:36<1:47:40,  2.42s/it]
                                                    
{'loss': 0.5767, 'grad_norm': 0.4464905560016632, 'learning_rate': 0.00018877256707947306, 'lm_loss': 0.1204095184803009, 'epoch': 0.71}

 18%|█▊        | 580/3252 [24:37<1:47:40,  2.42s/it]
 18%|█▊        | 581/3252 [24:39<1:48:40,  2.44s/it]
 18%|█▊        | 582/3252 [24:41<1:47:56,  2.43s/it]
 18%|█▊        | 583/3252 [24:44<1:47:25,  2.42s/it]
 18%|█▊        | 584/3252 [24:46<1:47:09,  2.41s/it]
 18%|█▊        | 585/3252 [24:48<1:47:00,  2.41s/it]
 18%|█▊        | 586/3252 [24:51<1:46:48,  2.40s/it]
 18%|█▊        | 587/3252 [24:53<1:46:46,  2.40s/it]
 18%|█▊        | 588/3252 [24:56<1:46:35,  2.40s/it]
 18%|█▊        | 589/3252 [24:58<1:46:37,  2.40s/it]
 18%|█▊        | 590/3252 [25:00<1:46:30,  2.40s/it]
                                                    
{'loss': 0.5388, 'grad_norm': 0.34453240036964417, 'learning_rate': 0.0001883093125970452, 'lm_loss': 0.07843688875436783, 'epoch': 0.73}

 18%|█▊        | 590/3252 [25:01<1:46:30,  2.40s/it]
 18%|█▊        | 591/3252 [25:03<1:48:29,  2.45s/it]
 18%|█▊        | 592/3252 [25:06<1:49:14,  2.46s/it]
 18%|█▊        | 593/3252 [25:08<1:48:13,  2.44s/it]
 18%|█▊        | 594/3252 [25:10<1:47:32,  2.43s/it]
 18%|█▊        | 595/3252 [25:13<1:47:06,  2.42s/it]
 18%|█▊        | 596/3252 [25:15<1:46:42,  2.41s/it]
 18%|█▊        | 597/3252 [25:17<1:46:18,  2.40s/it]
 18%|█▊        | 598/3252 [25:20<1:46:17,  2.40s/it]
 18%|█▊        | 599/3252 [25:22<1:46:03,  2.40s/it]
 18%|█▊        | 600/3252 [25:25<1:46:00,  2.40s/it]
                                                    
{'loss': 0.5528, 'grad_norm': 0.3488093912601471, 'learning_rate': 0.0001878372854760519, 'lm_loss': 0.5971645712852478, 'epoch': 0.74}

 18%|█▊        | 600/3252 [25:25<1:46:00,  2.40s/it]
 18%|█▊        | 601/3252 [25:27<1:46:48,  2.42s/it]
 19%|█▊        | 602/3252 [25:30<1:46:22,  2.41s/it]
 19%|█▊        | 603/3252 [25:32<1:46:02,  2.40s/it]
 19%|█▊        | 604/3252 [25:34<1:45:56,  2.40s/it]
 19%|█▊        | 605/3252 [25:37<1:45:43,  2.40s/it]
 19%|█▊        | 606/3252 [25:39<1:45:49,  2.40s/it]
 19%|█▊        | 607/3252 [25:41<1:45:41,  2.40s/it]
 19%|█▊        | 608/3252 [25:44<1:45:41,  2.40s/it]
 19%|█▊        | 609/3252 [25:46<1:45:39,  2.40s/it]
 19%|█▉        | 610/3252 [25:49<1:47:53,  2.45s/it]
                                                    
{'loss': 0.5254, 'grad_norm': 0.19726288318634033, 'learning_rate': 0.00018735653260762165, 'lm_loss': 0.907080352306366, 'epoch': 0.75}

 19%|█▉        | 610/3252 [25:49<1:47:53,  2.45s/it]
 19%|█▉        | 611/3252 [25:52<1:51:03,  2.52s/it]
 19%|█▉        | 612/3252 [25:54<1:49:22,  2.49s/it]
 19%|█▉        | 613/3252 [25:56<1:48:07,  2.46s/it]
 19%|█▉        | 614/3252 [25:59<1:47:16,  2.44s/it]
 19%|█▉        | 615/3252 [26:01<1:47:02,  2.44s/it]
 19%|█▉        | 616/3252 [26:04<1:46:23,  2.42s/it]
 19%|█▉        | 617/3252 [26:06<1:46:21,  2.42s/it]
 19%|█▉        | 618/3252 [26:08<1:45:53,  2.41s/it]
 19%|█▉        | 619/3252 [26:11<1:45:31,  2.40s/it]
 19%|█▉        | 620/3252 [26:13<1:45:21,  2.40s/it]
                                                    
{'loss': 0.5549, 'grad_norm': 0.23174436390399933, 'learning_rate': 0.00018686710174969787, 'lm_loss': 0.10675191134214401, 'epoch': 0.76}

 19%|█▉        | 620/3252 [26:13<1:45:21,  2.40s/it]
 19%|█▉        | 621/3252 [26:16<1:47:59,  2.46s/it]
 19%|█▉        | 622/3252 [26:18<1:46:55,  2.44s/it]
 19%|█▉        | 623/3252 [26:21<1:46:14,  2.42s/it]
 19%|█▉        | 624/3252 [26:23<1:45:52,  2.42s/it]
 19%|█▉        | 625/3252 [26:25<1:45:29,  2.41s/it]
 19%|█▉        | 626/3252 [26:28<1:45:11,  2.40s/it]
 19%|█▉        | 627/3252 [26:30<1:47:11,  2.45s/it]
 19%|█▉        | 628/3252 [26:33<1:46:21,  2.43s/it]
 19%|█▉        | 629/3252 [26:35<1:45:53,  2.42s/it]
 19%|█▉        | 630/3252 [26:37<1:45:26,  2.41s/it]
                                                    
{'loss': 0.5532, 'grad_norm': 0.31654998660087585, 'learning_rate': 0.00018636904152229456, 'lm_loss': 0.7807061672210693, 'epoch': 0.77}

 19%|█▉        | 630/3252 [26:38<1:45:26,  2.41s/it]
 19%|█▉        | 631/3252 [26:40<1:45:45,  2.42s/it]
 19%|█▉        | 632/3252 [26:42<1:45:21,  2.41s/it]
 19%|█▉        | 633/3252 [26:45<1:45:16,  2.41s/it]
 19%|█▉        | 634/3252 [26:47<1:44:56,  2.40s/it]
 20%|█▉        | 635/3252 [26:49<1:44:48,  2.40s/it]
 20%|█▉        | 636/3252 [26:52<1:44:44,  2.40s/it]
 20%|█▉        | 637/3252 [26:54<1:44:44,  2.40s/it]
 20%|█▉        | 638/3252 [26:57<1:44:42,  2.40s/it]
 20%|█▉        | 639/3252 [26:59<1:44:54,  2.41s/it]
 20%|█▉        | 640/3252 [27:02<1:44:46,  2.41s/it]
                                                    
{'loss': 0.5406, 'grad_norm': 0.17617149651050568, 'learning_rate': 0.00018586240140266655, 'lm_loss': 0.7994519472122192, 'epoch': 0.79}

 20%|█▉        | 640/3252 [27:02<1:44:46,  2.41s/it]
 20%|█▉        | 641/3252 [27:04<1:45:19,  2.42s/it]
 20%|█▉        | 642/3252 [27:06<1:44:56,  2.41s/it]
 20%|█▉        | 643/3252 [27:09<1:44:39,  2.41s/it]
 20%|█▉        | 644/3252 [27:11<1:44:36,  2.41s/it]
 20%|█▉        | 645/3252 [27:14<1:44:33,  2.41s/it]
 20%|█▉        | 646/3252 [27:16<1:44:22,  2.40s/it]
 20%|█▉        | 647/3252 [27:18<1:44:14,  2.40s/it]
 20%|█▉        | 648/3252 [27:21<1:48:30,  2.50s/it]
 20%|█▉        | 649/3252 [27:23<1:46:57,  2.47s/it]
 20%|█▉        | 650/3252 [27:26<1:45:57,  2.44s/it]
                                                    
{'loss': 0.55, 'grad_norm': 0.37261825799942017, 'learning_rate': 0.00018534723172039427, 'lm_loss': 1.4601987600326538, 'epoch': 0.8}

 20%|█▉        | 650/3252 [27:27<1:45:57,  2.44s/it]
 20%|██        | 651/3252 [27:29<1:51:57,  2.58s/it]
 20%|██        | 652/3252 [27:31<1:49:25,  2.53s/it]
 20%|██        | 653/3252 [27:34<1:47:41,  2.49s/it]
 20%|██        | 654/3252 [27:36<1:46:34,  2.46s/it]
 20%|██        | 655/3252 [27:38<1:45:39,  2.44s/it]
 20%|██        | 656/3252 [27:41<1:45:06,  2.43s/it]
 20%|██        | 657/3252 [27:43<1:44:42,  2.42s/it]
 20%|██        | 658/3252 [27:46<1:44:19,  2.41s/it]
 20%|██        | 659/3252 [27:48<1:44:02,  2.41s/it]
 20%|██        | 660/3252 [27:50<1:43:55,  2.41s/it]
                                                    
{'loss': 0.5532, 'grad_norm': 0.3484722375869751, 'learning_rate': 0.00018482358365238413, 'lm_loss': 0.7435799837112427, 'epoch': 0.81}

 20%|██        | 660/3252 [27:50<1:43:55,  2.41s/it]
 20%|██        | 661/3252 [27:53<1:43:55,  2.41s/it]
 20%|██        | 662/3252 [27:55<1:43:48,  2.40s/it]
 20%|██        | 663/3252 [27:58<1:43:35,  2.40s/it]
 20%|██        | 664/3252 [28:00<1:43:28,  2.40s/it]
 20%|██        | 665/3252 [28:02<1:43:26,  2.40s/it]
 20%|██        | 666/3252 [28:05<1:46:22,  2.47s/it]
 21%|██        | 667/3252 [28:07<1:45:33,  2.45s/it]
 21%|██        | 668/3252 [28:10<1:44:47,  2.43s/it]
 21%|██        | 669/3252 [28:12<1:44:08,  2.42s/it]
 21%|██        | 670/3252 [28:15<1:43:50,  2.41s/it]
                                                    
{'loss': 0.5613, 'grad_norm': 0.36168909072875977, 'learning_rate': 0.00018429150921778448, 'lm_loss': 0.4367240071296692, 'epoch': 0.82}

 21%|██        | 670/3252 [28:15<1:43:50,  2.41s/it]
 21%|██        | 671/3252 [28:17<1:44:51,  2.44s/it]
 21%|██        | 672/3252 [28:19<1:44:14,  2.42s/it]
 21%|██        | 673/3252 [28:22<1:43:44,  2.41s/it]
 21%|██        | 674/3252 [28:24<1:43:56,  2.42s/it]
 21%|██        | 675/3252 [28:27<1:43:33,  2.41s/it]
 21%|██        | 676/3252 [28:29<1:43:18,  2.41s/it]
 21%|██        | 677/3252 [28:31<1:43:08,  2.40s/it]
 21%|██        | 678/3252 [28:34<1:43:01,  2.40s/it]
 21%|██        | 679/3252 [28:36<1:42:53,  2.40s/it]
 21%|██        | 680/3252 [28:39<1:42:54,  2.40s/it]
                                                    
{'loss': 0.5491, 'grad_norm': 0.2901308238506317, 'learning_rate': 0.00018375106127281813, 'lm_loss': 0.13333012163639069, 'epoch': 0.84}

 21%|██        | 680/3252 [28:39<1:42:54,  2.40s/it]
 21%|██        | 681/3252 [28:41<1:43:23,  2.41s/it]
 21%|██        | 682/3252 [28:44<1:43:15,  2.41s/it]
 21%|██        | 683/3252 [28:46<1:43:34,  2.42s/it]
 21%|██        | 684/3252 [28:49<1:46:12,  2.48s/it]
 21%|██        | 685/3252 [28:51<1:46:27,  2.49s/it]
 21%|██        | 686/3252 [28:53<1:45:19,  2.46s/it]
 21%|██        | 687/3252 [28:56<1:44:25,  2.44s/it]
 21%|██        | 688/3252 [28:58<1:43:40,  2.43s/it]
 21%|██        | 689/3252 [29:01<1:43:12,  2.42s/it]
 21%|██        | 690/3252 [29:03<1:43:03,  2.41s/it]
                                                    
{'loss': 0.5289, 'grad_norm': 0.2079363614320755, 'learning_rate': 0.00018320229350553163, 'lm_loss': 0.1192043349146843, 'epoch': 0.85}

 21%|██        | 690/3252 [29:03<1:43:03,  2.41s/it]
 21%|██        | 691/3252 [29:06<1:43:58,  2.44s/it]
 21%|██▏       | 692/3252 [29:08<1:43:24,  2.42s/it]
 21%|██▏       | 693/3252 [29:10<1:42:54,  2.41s/it]
 21%|██▏       | 694/3252 [29:13<1:42:38,  2.41s/it]
 21%|██▏       | 695/3252 [29:15<1:42:35,  2.41s/it]
 21%|██▏       | 696/3252 [29:18<1:42:16,  2.40s/it]
 21%|██▏       | 697/3252 [29:20<1:42:07,  2.40s/it]
 21%|██▏       | 698/3252 [29:22<1:42:06,  2.40s/it]
 21%|██▏       | 699/3252 [29:25<1:41:56,  2.40s/it]
 22%|██▏       | 700/3252 [29:27<1:41:52,  2.40s/it]
                                                    
{'loss': 0.5568, 'grad_norm': 0.24045829474925995, 'learning_rate': 0.00018264526043046192, 'lm_loss': 0.023603010922670364, 'epoch': 0.86}

 22%|██▏       | 700/3252 [29:28<1:41:52,  2.40s/it]
 22%|██▏       | 701/3252 [29:30<1:48:19,  2.55s/it]
 22%|██▏       | 702/3252 [29:32<1:46:14,  2.50s/it]
 22%|██▏       | 703/3252 [29:35<1:44:57,  2.47s/it]
 22%|██▏       | 704/3252 [29:37<1:44:24,  2.46s/it]
 22%|██▏       | 705/3252 [29:40<1:43:37,  2.44s/it]
 22%|██▏       | 706/3252 [29:42<1:43:03,  2.43s/it]
 22%|██▏       | 707/3252 [29:44<1:42:34,  2.42s/it]
 22%|██▏       | 708/3252 [29:47<1:42:10,  2.41s/it]
 22%|██▏       | 709/3252 [29:49<1:41:58,  2.41s/it]
 22%|██▏       | 710/3252 [29:52<1:42:00,  2.41s/it]
                                                    
{'loss': 0.5106, 'grad_norm': 0.1991572082042694, 'learning_rate': 0.00018208001738322072, 'lm_loss': 1.4732013940811157, 'epoch': 0.87}

 22%|██▏       | 710/3252 [29:52<1:42:00,  2.41s/it]
 22%|██▏       | 711/3252 [29:54<1:43:31,  2.44s/it]
 22%|██▏       | 712/3252 [29:57<1:42:45,  2.43s/it]
 22%|██▏       | 713/3252 [29:59<1:42:13,  2.42s/it]
 22%|██▏       | 714/3252 [30:01<1:41:59,  2.41s/it]
 22%|██▏       | 715/3252 [30:04<1:41:43,  2.41s/it]
 22%|██▏       | 716/3252 [30:06<1:41:32,  2.40s/it]
 22%|██▏       | 717/3252 [30:09<1:41:26,  2.40s/it]
 22%|██▏       | 718/3252 [30:11<1:41:16,  2.40s/it]
 22%|██▏       | 719/3252 [30:13<1:41:18,  2.40s/it]
 22%|██▏       | 720/3252 [30:16<1:41:09,  2.40s/it]
                                                    
{'loss': 0.556, 'grad_norm': 0.2659837305545807, 'learning_rate': 0.0001815066205149976, 'lm_loss': 0.1956823170185089, 'epoch': 0.89}

 22%|██▏       | 720/3252 [30:16<1:41:09,  2.40s/it]
 22%|██▏       | 721/3252 [30:18<1:41:46,  2.41s/it]
 22%|██▏       | 722/3252 [30:21<1:46:30,  2.53s/it]
 22%|██▏       | 723/3252 [30:23<1:44:42,  2.48s/it]
 22%|██▏       | 724/3252 [30:26<1:43:24,  2.45s/it]
 22%|██▏       | 725/3252 [30:28<1:42:35,  2.44s/it]
 22%|██▏       | 726/3252 [30:30<1:42:01,  2.42s/it]
 22%|██▏       | 727/3252 [30:33<1:41:36,  2.41s/it]
 22%|██▏       | 728/3252 [30:35<1:41:20,  2.41s/it]
 22%|██▏       | 729/3252 [30:38<1:41:15,  2.41s/it]
 22%|██▏       | 730/3252 [30:40<1:41:08,  2.41s/it]
                                                    
{'loss': 0.5306, 'grad_norm': 0.19510652124881744, 'learning_rate': 0.000180925126786982, 'lm_loss': 1.4242780208587646, 'epoch': 0.9}

 22%|██▏       | 730/3252 [30:40<1:41:08,  2.41s/it]
 22%|██▏       | 731/3252 [30:43<1:42:28,  2.44s/it]
 23%|██▎       | 732/3252 [30:45<1:42:12,  2.43s/it]
 23%|██▎       | 733/3252 [30:47<1:41:41,  2.42s/it]
 23%|██▎       | 734/3252 [30:50<1:41:56,  2.43s/it]
 23%|██▎       | 735/3252 [30:52<1:41:29,  2.42s/it]
 23%|██▎       | 736/3252 [30:55<1:41:16,  2.41s/it]
 23%|██▎       | 737/3252 [30:57<1:40:53,  2.41s/it]
 23%|██▎       | 738/3252 [30:59<1:40:36,  2.40s/it]
 23%|██▎       | 739/3252 [31:02<1:40:26,  2.40s/it]
 23%|██▎       | 740/3252 [31:04<1:41:25,  2.42s/it]
                                                    
{'loss': 0.5297, 'grad_norm': 0.23294301331043243, 'learning_rate': 0.00018033559396470454, 'lm_loss': 0.5768815875053406, 'epoch': 0.91}

 23%|██▎       | 740/3252 [31:04<1:41:25,  2.42s/it]
 23%|██▎       | 741/3252 [31:07<1:41:09,  2.42s/it]
 23%|██▎       | 742/3252 [31:09<1:40:44,  2.41s/it]
 23%|██▎       | 743/3252 [31:11<1:40:24,  2.40s/it]
 23%|██▎       | 744/3252 [31:14<1:40:15,  2.40s/it]
 23%|██▎       | 745/3252 [31:16<1:40:05,  2.40s/it]
 23%|██▎       | 746/3252 [31:19<1:40:05,  2.40s/it]
 23%|██▎       | 747/3252 [31:21<1:40:02,  2.40s/it]
 23%|██▎       | 748/3252 [31:23<1:39:56,  2.39s/it]
 23%|██▎       | 749/3252 [31:26<1:40:01,  2.40s/it]
 23%|██▎       | 750/3252 [31:28<1:39:51,  2.39s/it]
                                                    
{'loss': 0.5223, 'grad_norm': 0.5947741866111755, 'learning_rate': 0.00017973808061229874, 'lm_loss': 1.4922882318496704, 'epoch': 0.92}

 23%|██▎       | 750/3252 [31:28<1:39:51,  2.39s/it]
 23%|██▎       | 751/3252 [31:31<1:41:56,  2.45s/it]
 23%|██▎       | 752/3252 [31:33<1:41:12,  2.43s/it]
 23%|██▎       | 753/3252 [31:36<1:40:43,  2.42s/it]
 23%|██▎       | 754/3252 [31:38<1:40:23,  2.41s/it]
 23%|██▎       | 755/3252 [31:40<1:40:09,  2.41s/it]
 23%|██▎       | 756/3252 [31:43<1:39:58,  2.40s/it]
 23%|██▎       | 757/3252 [31:45<1:40:14,  2.41s/it]
 23%|██▎       | 758/3252 [31:48<1:42:16,  2.46s/it]
 23%|██▎       | 759/3252 [31:50<1:43:03,  2.48s/it]
 23%|██▎       | 760/3252 [31:53<1:42:01,  2.46s/it]
                                                    
{'loss': 0.557, 'grad_norm': 0.2646091878414154, 'learning_rate': 0.00017913264608668316, 'lm_loss': 0.3815400302410126, 'epoch': 0.93}

 23%|██▎       | 760/3252 [31:53<1:42:01,  2.46s/it]
 23%|██▎       | 761/3252 [31:55<1:41:32,  2.45s/it]
 23%|██▎       | 762/3252 [31:58<1:40:48,  2.43s/it]
 23%|██▎       | 763/3252 [32:00<1:40:22,  2.42s/it]
 23%|██▎       | 764/3252 [32:02<1:39:56,  2.41s/it]
 24%|██▎       | 765/3252 [32:05<1:39:39,  2.40s/it]
 24%|██▎       | 766/3252 [32:07<1:39:29,  2.40s/it]
 24%|██▎       | 767/3252 [32:09<1:39:20,  2.40s/it]
 24%|██▎       | 768/3252 [32:12<1:39:12,  2.40s/it]
 24%|██▎       | 769/3252 [32:14<1:39:08,  2.40s/it]
 24%|██▎       | 770/3252 [32:17<1:39:07,  2.40s/it]
                                                    
{'loss': 0.5157, 'grad_norm': 0.1763799488544464, 'learning_rate': 0.00017851935053166503, 'lm_loss': 0.5149619579315186, 'epoch': 0.95}

 24%|██▎       | 770/3252 [32:17<1:39:07,  2.40s/it]
 24%|██▎       | 771/3252 [32:19<1:41:35,  2.46s/it]
 24%|██▎       | 772/3252 [32:22<1:40:42,  2.44s/it]
 24%|██▍       | 773/3252 [32:24<1:40:08,  2.42s/it]
 24%|██▍       | 774/3252 [32:26<1:39:47,  2.42s/it]
 24%|██▍       | 775/3252 [32:29<1:42:06,  2.47s/it]
 24%|██▍       | 776/3252 [32:31<1:41:01,  2.45s/it]
 24%|██▍       | 777/3252 [32:34<1:40:16,  2.43s/it]
 24%|██▍       | 778/3252 [32:36<1:39:48,  2.42s/it]
 24%|██▍       | 779/3252 [32:39<1:39:28,  2.41s/it]
 24%|██▍       | 780/3252 [32:41<1:39:11,  2.41s/it]
                                                    
{'loss': 0.5429, 'grad_norm': 0.21355922520160675, 'learning_rate': 0.00017789825487196537, 'lm_loss': 0.33725765347480774, 'epoch': 0.96}

 24%|██▍       | 780/3252 [32:41<1:39:11,  2.41s/it]
 24%|██▍       | 781/3252 [32:44<1:40:52,  2.45s/it]
 24%|██▍       | 782/3252 [32:46<1:40:22,  2.44s/it]
 24%|██▍       | 783/3252 [32:48<1:39:52,  2.43s/it]
 24%|██▍       | 784/3252 [32:51<1:39:27,  2.42s/it]
 24%|██▍       | 785/3252 [32:53<1:39:08,  2.41s/it]
 24%|██▍       | 786/3252 [32:56<1:38:54,  2.41s/it]
 24%|██▍       | 787/3252 [32:58<1:38:52,  2.41s/it]
 24%|██▍       | 788/3252 [33:00<1:38:50,  2.41s/it]
 24%|██▍       | 789/3252 [33:03<1:38:36,  2.40s/it]
 24%|██▍       | 790/3252 [33:05<1:38:22,  2.40s/it]
                                                    
{'loss': 0.5354, 'grad_norm': 0.17351070046424866, 'learning_rate': 0.00017726942080716692, 'lm_loss': 0.04111048951745033, 'epoch': 0.97}

 24%|██▍       | 790/3252 [33:05<1:38:22,  2.40s/it]
 24%|██▍       | 791/3252 [33:08<1:38:26,  2.40s/it]
 24%|██▍       | 792/3252 [33:10<1:38:20,  2.40s/it]
 24%|██▍       | 793/3252 [33:12<1:39:10,  2.42s/it]
 24%|██▍       | 794/3252 [33:15<1:38:46,  2.41s/it]
 24%|██▍       | 795/3252 [33:17<1:38:31,  2.41s/it]
 24%|██▍       | 796/3252 [33:20<1:42:04,  2.49s/it]
 25%|██▍       | 797/3252 [33:22<1:40:49,  2.46s/it]
 25%|██▍       | 798/3252 [33:25<1:39:55,  2.44s/it]
 25%|██▍       | 799/3252 [33:27<1:39:21,  2.43s/it]
 25%|██▍       | 800/3252 [33:30<1:39:00,  2.42s/it]
                                                    
{'loss': 0.5337, 'grad_norm': 0.18784412741661072, 'learning_rate': 0.00017663291080558476, 'lm_loss': 0.6365185976028442, 'epoch': 0.98}

 25%|██▍       | 800/3252 [33:30<1:39:00,  2.42s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.92it/s]
                                                    

                                             
{'eval_sthsthv2/narration_val_loss': 0.7665727734565735, 'eval_sthsthv2/narration_val_runtime': 3.6663, 'eval_sthsthv2/narration_val_samples_per_second': 182.202, 'eval_sthsthv2/narration_val_steps_per_second': 0.546, 'epoch': 0.98}

 25%|██▍       | 800/3252 [33:33<1:39:00,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  1.92it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.56it/s]
                                                    

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6263958811759949, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.2372, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 182.874, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.618, 'epoch': 0.98}

 25%|██▍       | 800/3252 [33:37<1:39:00,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  1.56it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.93it/s]
                                                    

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.6014955043792725, 'eval_holoassist/dialog-klg-sum_val_runtime': 3.0812, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 230.108, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.649, 'epoch': 0.98}

 25%|██▍       | 800/3252 [33:40<1:39:00,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  1.93it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                    

                                     
{'eval_ego4d/narration_val_loss': 0.5506431460380554, 'eval_ego4d/narration_val_runtime': 2.2715, 'eval_ego4d/narration_val_samples_per_second': 98.171, 'eval_ego4d/narration_val_steps_per_second': 0.44, 'epoch': 0.98}

 25%|██▍       | 800/3252 [33:42<1:39:00,  2.42s/it]

100%|██████████| 1/1 [00:00<00:00,  3.09it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.49it/s]
                                                    

                                             
{'eval_ego4d/summary_val_loss': 1.1310529708862305, 'eval_ego4d/summary_val_runtime': 2.979, 'eval_ego4d/summary_val_samples_per_second': 189.325, 'eval_ego4d/summary_val_steps_per_second': 0.671, 'epoch': 0.98}

 25%|██▍       | 800/3252 [33:45<1:39:00,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  1.49it/s]

                                             
 25%|██▍       | 801/3252 [33:48<4:50:29,  7.11s/it]
 25%|██▍       | 802/3252 [33:50<3:52:28,  5.69s/it]
 25%|██▍       | 803/3252 [33:52<3:12:01,  4.70s/it]
 25%|██▍       | 804/3252 [33:55<2:43:37,  4.01s/it]
 25%|██▍       | 805/3252 [33:57<2:23:47,  3.53s/it]
 25%|██▍       | 806/3252 [34:00<2:09:52,  3.19s/it]
 25%|██▍       | 807/3252 [34:02<2:00:24,  2.95s/it]
 25%|██▍       | 808/3252 [34:04<1:53:53,  2.80s/it]
 25%|██▍       | 809/3252 [34:07<1:49:02,  2.68s/it]
 25%|██▍       | 810/3252 [34:09<1:45:31,  2.59s/it]
                                                    
{'loss': 0.4981, 'grad_norm': 0.18632720410823822, 'learning_rate': 0.00017598878809806093, 'lm_loss': 0.5272898077964783, 'epoch': 1.0}

 25%|██▍       | 810/3252 [34:09<1:45:31,  2.59s/it]
 25%|██▍       | 811/3252 [34:12<1:44:23,  2.57s/it]
 25%|██▍       | 812/3252 [34:14<1:42:16,  2.51s/it]
 25%|██▌       | 813/3252 [34:17<1:44:38,  2.57s/it]
 25%|██▌       | 814/3252 [34:23<2:23:57,  3.54s/it]
 25%|██▌       | 815/3252 [34:25<2:10:04,  3.20s/it]
 25%|██▌       | 816/3252 [34:27<2:00:19,  2.96s/it]
 25%|██▌       | 817/3252 [34:30<1:54:08,  2.81s/it]
 25%|██▌       | 818/3252 [34:32<1:49:07,  2.69s/it]
 25%|██▌       | 819/3252 [34:35<1:45:40,  2.61s/it]
 25%|██▌       | 820/3252 [34:37<1:43:04,  2.54s/it]
                                                    
{'loss': 0.5069, 'grad_norm': 0.28473472595214844, 'learning_rate': 0.0001753371166716828, 'lm_loss': 0.6047778129577637, 'epoch': 1.01}

 25%|██▌       | 820/3252 [34:37<1:43:04,  2.54s/it]
 25%|██▌       | 821/3252 [34:39<1:41:25,  2.50s/it]
 25%|██▌       | 822/3252 [34:42<1:40:03,  2.47s/it]
 25%|██▌       | 823/3252 [34:44<1:39:06,  2.45s/it]
 25%|██▌       | 824/3252 [34:47<1:38:33,  2.44s/it]
 25%|██▌       | 825/3252 [34:49<1:38:03,  2.42s/it]
 25%|██▌       | 826/3252 [34:51<1:37:37,  2.41s/it]
 25%|██▌       | 827/3252 [34:54<1:37:41,  2.42s/it]
 25%|██▌       | 828/3252 [34:56<1:37:24,  2.41s/it]
 25%|██▌       | 829/3252 [34:59<1:37:11,  2.41s/it]
 26%|██▌       | 830/3252 [35:01<1:37:01,  2.40s/it]
                                                    
{'loss': 0.4867, 'grad_norm': 0.18448244035243988, 'learning_rate': 0.00017467796126342675, 'lm_loss': 0.009609890170395374, 'epoch': 1.02}

 26%|██▌       | 830/3252 [35:01<1:37:01,  2.40s/it]
 26%|██▌       | 831/3252 [35:04<1:37:57,  2.43s/it]
 26%|██▌       | 832/3252 [35:06<1:37:21,  2.41s/it]
 26%|██▌       | 833/3252 [35:08<1:39:04,  2.46s/it]
 26%|██▌       | 834/3252 [35:11<1:38:08,  2.44s/it]
 26%|██▌       | 835/3252 [35:13<1:37:30,  2.42s/it]
 26%|██▌       | 836/3252 [35:16<1:37:05,  2.41s/it]
 26%|██▌       | 837/3252 [35:18<1:36:51,  2.41s/it]
 26%|██▌       | 838/3252 [35:20<1:36:40,  2.40s/it]
 26%|██▌       | 839/3252 [35:23<1:36:23,  2.40s/it]
 26%|██▌       | 840/3252 [35:25<1:36:27,  2.40s/it]
                                                    
{'loss': 0.5083, 'grad_norm': 0.1804620772600174, 'learning_rate': 0.00017401138735372728, 'lm_loss': 0.5649676322937012, 'epoch': 1.03}

 26%|██▌       | 840/3252 [35:26<1:36:27,  2.40s/it]
 26%|██▌       | 841/3252 [35:28<1:40:15,  2.50s/it]
 26%|██▌       | 842/3252 [35:30<1:39:01,  2.47s/it]
 26%|██▌       | 843/3252 [35:33<1:38:08,  2.44s/it]
 26%|██▌       | 844/3252 [35:35<1:37:31,  2.43s/it]
 26%|██▌       | 845/3252 [35:38<1:37:07,  2.42s/it]
 26%|██▌       | 846/3252 [35:40<1:36:47,  2.41s/it]
 26%|██▌       | 847/3252 [35:42<1:36:36,  2.41s/it]
 26%|██▌       | 848/3252 [35:45<1:36:23,  2.41s/it]
 26%|██▌       | 849/3252 [35:47<1:36:09,  2.40s/it]
 26%|██▌       | 850/3252 [35:50<1:36:13,  2.40s/it]
                                                    
{'loss': 0.4815, 'grad_norm': 0.30262458324432373, 'learning_rate': 0.00017333746115997202, 'lm_loss': 0.7213220000267029, 'epoch': 1.05}

 26%|██▌       | 850/3252 [35:50<1:36:13,  2.40s/it]
 26%|██▌       | 851/3252 [35:52<1:38:26,  2.46s/it]
 26%|██▌       | 852/3252 [35:55<1:37:47,  2.44s/it]
 26%|██▌       | 853/3252 [35:57<1:37:02,  2.43s/it]
 26%|██▋       | 854/3252 [35:59<1:36:41,  2.42s/it]
 26%|██▋       | 855/3252 [36:02<1:36:25,  2.41s/it]
 26%|██▋       | 856/3252 [36:04<1:36:06,  2.41s/it]
 26%|██▋       | 857/3252 [36:06<1:35:47,  2.40s/it]
 26%|██▋       | 858/3252 [36:09<1:35:40,  2.40s/it]
 26%|██▋       | 859/3252 [36:11<1:35:28,  2.39s/it]
 26%|██▋       | 860/3252 [36:14<1:35:26,  2.39s/it]
                                                    
{'loss': 0.476, 'grad_norm': 0.2924748659133911, 'learning_rate': 0.0001726562496299239, 'lm_loss': 1.446886658668518, 'epoch': 1.06}

 26%|██▋       | 860/3252 [36:14<1:35:26,  2.39s/it]
 26%|██▋       | 861/3252 [36:16<1:38:03,  2.46s/it]
 27%|██▋       | 862/3252 [36:19<1:37:15,  2.44s/it]
 27%|██▋       | 863/3252 [36:21<1:36:39,  2.43s/it]
 27%|██▋       | 864/3252 [36:23<1:36:14,  2.42s/it]
 27%|██▋       | 865/3252 [36:26<1:35:59,  2.41s/it]
 27%|██▋       | 866/3252 [36:28<1:35:41,  2.41s/it]
 27%|██▋       | 867/3252 [36:31<1:35:26,  2.40s/it]
 27%|██▋       | 868/3252 [36:33<1:36:34,  2.43s/it]
 27%|██▋       | 869/3252 [36:36<1:36:10,  2.42s/it]
 27%|██▋       | 870/3252 [36:38<1:36:02,  2.42s/it]
                                                    
{'loss': 0.4856, 'grad_norm': 0.21830710768699646, 'learning_rate': 0.00017196782043507028, 'lm_loss': 0.3956383764743805, 'epoch': 1.07}

 27%|██▋       | 870/3252 [36:38<1:36:02,  2.42s/it]
 27%|██▋       | 871/3252 [36:40<1:36:35,  2.43s/it]
 27%|██▋       | 872/3252 [36:43<1:36:02,  2.42s/it]
 27%|██▋       | 873/3252 [36:45<1:35:40,  2.41s/it]
 27%|██▋       | 874/3252 [36:48<1:35:29,  2.41s/it]
 27%|██▋       | 875/3252 [36:50<1:35:17,  2.41s/it]
 27%|██▋       | 876/3252 [36:52<1:35:14,  2.41s/it]
 27%|██▋       | 877/3252 [36:55<1:35:03,  2.40s/it]
 27%|██▋       | 878/3252 [36:57<1:34:58,  2.40s/it]
 27%|██▋       | 879/3252 [37:00<1:35:14,  2.41s/it]
 27%|██▋       | 880/3252 [37:02<1:34:58,  2.40s/it]
                                                    
{'loss': 0.4926, 'grad_norm': 0.26556193828582764, 'learning_rate': 0.0001712722419639008, 'lm_loss': 0.5687783360481262, 'epoch': 1.08}

 27%|██▋       | 880/3252 [37:02<1:34:58,  2.40s/it]
 27%|██▋       | 881/3252 [37:04<1:35:39,  2.42s/it]
 27%|██▋       | 882/3252 [37:07<1:35:19,  2.41s/it]
 27%|██▋       | 883/3252 [37:09<1:35:01,  2.41s/it]
 27%|██▋       | 884/3252 [37:12<1:35:16,  2.41s/it]
 27%|██▋       | 885/3252 [37:14<1:34:54,  2.41s/it]
 27%|██▋       | 886/3252 [37:16<1:34:40,  2.40s/it]
 27%|██▋       | 887/3252 [37:19<1:34:26,  2.40s/it]
 27%|██▋       | 888/3252 [37:21<1:34:18,  2.39s/it]
 27%|██▋       | 889/3252 [37:24<1:34:27,  2.40s/it]
 27%|██▋       | 890/3252 [37:26<1:34:16,  2.39s/it]
                                                    
{'loss': 0.5042, 'grad_norm': 0.2863841652870178, 'learning_rate': 0.00017056958331511344, 'lm_loss': 0.03730394318699837, 'epoch': 1.09}

 27%|██▋       | 890/3252 [37:26<1:34:16,  2.39s/it]
 27%|██▋       | 891/3252 [37:29<1:34:56,  2.41s/it]
 27%|██▋       | 892/3252 [37:31<1:34:40,  2.41s/it]
 27%|██▋       | 893/3252 [37:33<1:34:33,  2.41s/it]
 27%|██▋       | 894/3252 [37:36<1:34:21,  2.40s/it]
 28%|██▊       | 895/3252 [37:38<1:34:15,  2.40s/it]
 28%|██▊       | 896/3252 [37:40<1:34:15,  2.40s/it]
 28%|██▊       | 897/3252 [37:43<1:34:08,  2.40s/it]
 28%|██▊       | 898/3252 [37:45<1:34:07,  2.40s/it]
 28%|██▊       | 899/3252 [37:48<1:34:02,  2.40s/it]
 28%|██▊       | 900/3252 [37:50<1:33:55,  2.40s/it]
                                                    
{'loss': 0.4844, 'grad_norm': 0.23833370208740234, 'learning_rate': 0.00016985991429075036, 'lm_loss': 0.12444683164358139, 'epoch': 1.11}

 28%|██▊       | 900/3252 [37:50<1:33:55,  2.40s/it]
 28%|██▊       | 901/3252 [37:52<1:34:14,  2.41s/it]
 28%|██▊       | 902/3252 [37:55<1:34:00,  2.40s/it]
 28%|██▊       | 903/3252 [37:57<1:33:56,  2.40s/it]
 28%|██▊       | 904/3252 [38:00<1:33:47,  2.40s/it]
 28%|██▊       | 905/3252 [38:02<1:33:49,  2.40s/it]
 28%|██▊       | 906/3252 [38:04<1:33:49,  2.40s/it]
 28%|██▊       | 907/3252 [38:07<1:36:14,  2.46s/it]
 28%|██▊       | 908/3252 [38:09<1:35:36,  2.45s/it]
 28%|██▊       | 909/3252 [38:12<1:38:26,  2.52s/it]
 28%|██▊       | 910/3252 [38:15<1:37:07,  2.49s/it]
                                                    
{'loss': 0.4955, 'grad_norm': 0.28401950001716614, 'learning_rate': 0.00016914330538926375, 'lm_loss': 1.3211818933486938, 'epoch': 1.12}

 28%|██▊       | 910/3252 [38:15<1:37:07,  2.49s/it]
 28%|██▊       | 911/3252 [38:17<1:37:36,  2.50s/it]
 28%|██▊       | 912/3252 [38:20<1:36:16,  2.47s/it]
 28%|██▊       | 913/3252 [38:22<1:35:16,  2.44s/it]
 28%|██▊       | 914/3252 [38:24<1:34:32,  2.43s/it]
 28%|██▊       | 915/3252 [38:27<1:34:06,  2.42s/it]
 28%|██▊       | 916/3252 [38:29<1:33:51,  2.41s/it]
 28%|██▊       | 917/3252 [38:31<1:33:38,  2.41s/it]
 28%|██▊       | 918/3252 [38:34<1:33:33,  2.40s/it]
 28%|██▊       | 919/3252 [38:36<1:33:24,  2.40s/it]
 28%|██▊       | 920/3252 [38:39<1:33:20,  2.40s/it]
                                                    
{'loss': 0.48, 'grad_norm': 0.1778978556394577, 'learning_rate': 0.00016841982779851252, 'lm_loss': 0.7251138091087341, 'epoch': 1.13}

 28%|██▊       | 920/3252 [38:39<1:33:20,  2.40s/it]
 28%|██▊       | 921/3252 [38:41<1:33:33,  2.41s/it]
 28%|██▊       | 922/3252 [38:44<1:33:23,  2.40s/it]
 28%|██▊       | 923/3252 [38:46<1:33:14,  2.40s/it]
 28%|██▊       | 924/3252 [38:48<1:33:10,  2.40s/it]
 28%|██▊       | 925/3252 [38:51<1:33:07,  2.40s/it]
 28%|██▊       | 926/3252 [38:53<1:34:09,  2.43s/it]
 29%|██▊       | 927/3252 [38:56<1:33:52,  2.42s/it]
 29%|██▊       | 928/3252 [38:58<1:33:28,  2.41s/it]
 29%|██▊       | 929/3252 [39:00<1:33:10,  2.41s/it]
 29%|██▊       | 930/3252 [39:03<1:32:59,  2.40s/it]
                                                    
{'loss': 0.4973, 'grad_norm': 0.2105744183063507, 'learning_rate': 0.0001676895533886905, 'lm_loss': 0.5431582927703857, 'epoch': 1.14}

 29%|██▊       | 930/3252 [39:03<1:32:59,  2.40s/it]
 29%|██▊       | 931/3252 [39:05<1:33:18,  2.41s/it]
 29%|██▊       | 932/3252 [39:08<1:33:00,  2.41s/it]
 29%|██▊       | 933/3252 [39:10<1:32:47,  2.40s/it]
 29%|██▊       | 934/3252 [39:12<1:32:35,  2.40s/it]
 29%|██▉       | 935/3252 [39:15<1:32:28,  2.39s/it]
 29%|██▉       | 936/3252 [39:17<1:32:22,  2.39s/it]
 29%|██▉       | 937/3252 [39:20<1:32:18,  2.39s/it]
 29%|██▉       | 938/3252 [39:22<1:32:14,  2.39s/it]
 29%|██▉       | 939/3252 [39:24<1:32:10,  2.39s/it]
 29%|██▉       | 940/3252 [39:27<1:32:19,  2.40s/it]
                                                    
{'loss': 0.4865, 'grad_norm': 0.23167584836483002, 'learning_rate': 0.00016695255470518686, 'lm_loss': 0.68333899974823, 'epoch': 1.16}

 29%|██▉       | 940/3252 [39:27<1:32:19,  2.40s/it]
 29%|██▉       | 941/3252 [39:29<1:34:26,  2.45s/it]
 29%|██▉       | 942/3252 [39:32<1:33:50,  2.44s/it]
 29%|██▉       | 943/3252 [39:34<1:34:54,  2.47s/it]
 29%|██▉       | 944/3252 [39:37<1:34:04,  2.45s/it]
 29%|██▉       | 945/3252 [39:39<1:33:26,  2.43s/it]
 29%|██▉       | 946/3252 [39:41<1:33:10,  2.42s/it]
 29%|██▉       | 947/3252 [39:44<1:32:49,  2.42s/it]
 29%|██▉       | 948/3252 [39:46<1:32:32,  2.41s/it]
 29%|██▉       | 949/3252 [39:49<1:32:19,  2.41s/it]
 29%|██▉       | 950/3252 [39:51<1:32:52,  2.42s/it]
                                                    
{'loss': 0.4693, 'grad_norm': 0.22470398247241974, 'learning_rate': 0.00016620890496137953, 'lm_loss': 0.588444173336029, 'epoch': 1.17}

 29%|██▉       | 950/3252 [39:51<1:32:52,  2.42s/it]
 29%|██▉       | 951/3252 [39:54<1:34:06,  2.45s/it]
 29%|██▉       | 952/3252 [39:56<1:33:24,  2.44s/it]
 29%|██▉       | 953/3252 [39:58<1:32:57,  2.43s/it]
 29%|██▉       | 954/3252 [40:01<1:32:36,  2.42s/it]
 29%|██▉       | 955/3252 [40:03<1:32:14,  2.41s/it]
 29%|██▉       | 956/3252 [40:06<1:31:56,  2.40s/it]
 29%|██▉       | 957/3252 [40:08<1:31:44,  2.40s/it]
 29%|██▉       | 958/3252 [40:10<1:31:37,  2.40s/it]
 29%|██▉       | 959/3252 [40:13<1:31:33,  2.40s/it]
 30%|██▉       | 960/3252 [40:15<1:31:21,  2.39s/it]
                                                    
{'loss': 0.469, 'grad_norm': 0.22361581027507782, 'learning_rate': 0.000165458678031362, 'lm_loss': 0.5419159531593323, 'epoch': 1.18}

 30%|██▉       | 960/3252 [40:15<1:31:21,  2.39s/it]
 30%|██▉       | 961/3252 [40:18<1:31:31,  2.40s/it]
 30%|██▉       | 962/3252 [40:20<1:31:32,  2.40s/it]
 30%|██▉       | 963/3252 [40:22<1:32:35,  2.43s/it]
 30%|██▉       | 964/3252 [40:25<1:32:14,  2.42s/it]
 30%|██▉       | 965/3252 [40:27<1:33:18,  2.45s/it]
 30%|██▉       | 966/3252 [40:30<1:32:39,  2.43s/it]
 30%|██▉       | 967/3252 [40:32<1:32:11,  2.42s/it]
 30%|██▉       | 968/3252 [40:35<1:31:46,  2.41s/it]
 30%|██▉       | 969/3252 [40:37<1:31:30,  2.40s/it]
 30%|██▉       | 970/3252 [40:39<1:31:22,  2.40s/it]
                                                    
{'loss': 0.5035, 'grad_norm': 0.30727845430374146, 'learning_rate': 0.00016470194844260498, 'lm_loss': 0.5975921154022217, 'epoch': 1.19}

 30%|██▉       | 970/3252 [40:39<1:31:22,  2.40s/it]
 30%|██▉       | 971/3252 [40:42<1:31:52,  2.42s/it]
 30%|██▉       | 972/3252 [40:44<1:31:35,  2.41s/it]
 30%|██▉       | 973/3252 [40:47<1:31:26,  2.41s/it]
 30%|██▉       | 974/3252 [40:49<1:31:16,  2.40s/it]
 30%|██▉       | 975/3252 [40:51<1:31:05,  2.40s/it]
 30%|███       | 976/3252 [40:54<1:31:00,  2.40s/it]
 30%|███       | 977/3252 [40:56<1:30:49,  2.40s/it]
 30%|███       | 978/3252 [40:59<1:30:53,  2.40s/it]
 30%|███       | 979/3252 [41:01<1:30:53,  2.40s/it]
 30%|███       | 980/3252 [41:03<1:30:48,  2.40s/it]
                                                    
{'loss': 0.4974, 'grad_norm': 0.1852865070104599, 'learning_rate': 0.00016393879136855248, 'lm_loss': 0.4269799590110779, 'epoch': 1.21}

 30%|███       | 980/3252 [41:04<1:30:48,  2.40s/it]
 30%|███       | 981/3252 [41:06<1:35:13,  2.52s/it]
 30%|███       | 982/3252 [41:09<1:33:46,  2.48s/it]
 30%|███       | 983/3252 [41:11<1:35:59,  2.54s/it]
 30%|███       | 984/3252 [41:14<1:34:23,  2.50s/it]
 30%|███       | 985/3252 [41:16<1:33:18,  2.47s/it]
 30%|███       | 986/3252 [41:18<1:32:20,  2.45s/it]
 30%|███       | 987/3252 [41:21<1:31:50,  2.43s/it]
 30%|███       | 988/3252 [41:23<1:31:32,  2.43s/it]
 30%|███       | 989/3252 [41:26<1:31:05,  2.41s/it]
 30%|███       | 990/3252 [41:28<1:30:58,  2.41s/it]
                                                    
{'loss': 0.4845, 'grad_norm': 0.17630791664123535, 'learning_rate': 0.0001631692826211544, 'lm_loss': 0.03200293332338333, 'epoch': 1.22}

 30%|███       | 990/3252 [41:28<1:30:58,  2.41s/it]
 30%|███       | 991/3252 [41:31<1:31:38,  2.43s/it]
 31%|███       | 992/3252 [41:33<1:31:14,  2.42s/it]
 31%|███       | 993/3252 [41:35<1:31:00,  2.42s/it]
 31%|███       | 994/3252 [41:38<1:30:43,  2.41s/it]
 31%|███       | 995/3252 [41:40<1:30:32,  2.41s/it]
 31%|███       | 996/3252 [41:43<1:30:24,  2.40s/it]
 31%|███       | 997/3252 [41:45<1:30:17,  2.40s/it]
 31%|███       | 998/3252 [41:47<1:30:08,  2.40s/it]
 31%|███       | 999/3252 [41:50<1:30:04,  2.40s/it]
 31%|███       | 1000/3252 [41:52<1:31:06,  2.43s/it]
                                                     
{'loss': 0.4857, 'grad_norm': 0.24350880086421967, 'learning_rate': 0.00016239349864333527, 'lm_loss': 1.261782169342041, 'epoch': 1.23}

 31%|███       | 1000/3252 [41:52<1:31:06,  2.43s/it]
 31%|███       | 1001/3252 [41:55<1:32:32,  2.47s/it]
 31%|███       | 1002/3252 [41:57<1:31:41,  2.45s/it]
 31%|███       | 1003/3252 [42:00<1:31:09,  2.43s/it]
 31%|███       | 1004/3252 [42:02<1:30:47,  2.42s/it]
 31%|███       | 1005/3252 [42:04<1:30:35,  2.42s/it]
 31%|███       | 1006/3252 [42:07<1:30:16,  2.41s/it]
 31%|███       | 1007/3252 [42:09<1:30:06,  2.41s/it]
 31%|███       | 1008/3252 [42:12<1:29:56,  2.40s/it]
 31%|███       | 1009/3252 [42:14<1:29:49,  2.40s/it]
 31%|███       | 1010/3252 [42:16<1:29:47,  2.40s/it]
                                                     
{'loss': 0.4904, 'grad_norm': 0.2370385080575943, 'learning_rate': 0.00016161151650140025, 'lm_loss': 0.7873631715774536, 'epoch': 1.24}

 31%|███       | 1010/3252 [42:16<1:29:47,  2.40s/it]
 31%|███       | 1011/3252 [42:19<1:30:06,  2.41s/it]
 31%|███       | 1012/3252 [42:21<1:29:51,  2.41s/it]
 31%|███       | 1013/3252 [42:24<1:29:37,  2.40s/it]
 31%|███       | 1014/3252 [42:26<1:29:28,  2.40s/it]
 31%|███       | 1015/3252 [42:28<1:30:18,  2.42s/it]
 31%|███       | 1016/3252 [42:31<1:30:11,  2.42s/it]
 31%|███▏      | 1017/3252 [42:33<1:31:29,  2.46s/it]
 31%|███▏      | 1018/3252 [42:36<1:30:52,  2.44s/it]
 31%|███▏      | 1019/3252 [42:38<1:30:18,  2.43s/it]
 31%|███▏      | 1020/3252 [42:41<1:29:58,  2.42s/it]
                                                     
{'loss': 0.4674, 'grad_norm': 0.3515241742134094, 'learning_rate': 0.0001608234138773797, 'lm_loss': 0.31127193570137024, 'epoch': 1.25}

 31%|███▏      | 1020/3252 [42:41<1:29:58,  2.42s/it]
 31%|███▏      | 1021/3252 [42:43<1:30:02,  2.42s/it]
 31%|███▏      | 1022/3252 [42:45<1:29:40,  2.41s/it]
 31%|███▏      | 1023/3252 [42:48<1:29:27,  2.41s/it]
 31%|███▏      | 1024/3252 [42:50<1:29:10,  2.40s/it]
 32%|███▏      | 1025/3252 [42:53<1:28:55,  2.40s/it]
 32%|███▏      | 1026/3252 [42:55<1:28:48,  2.39s/it]
 32%|███▏      | 1027/3252 [42:57<1:28:39,  2.39s/it]
 32%|███▏      | 1028/3252 [43:00<1:28:37,  2.39s/it]
 32%|███▏      | 1029/3252 [43:02<1:28:40,  2.39s/it]
 32%|███▏      | 1030/3252 [43:05<1:28:37,  2.39s/it]
                                                     
{'loss': 0.487, 'grad_norm': 0.2496965080499649, 'learning_rate': 0.000160029269061312, 'lm_loss': 0.6259192824363708, 'epoch': 1.27}

 32%|███▏      | 1030/3252 [43:05<1:28:37,  2.39s/it]
 32%|███▏      | 1031/3252 [43:07<1:29:14,  2.41s/it]
 32%|███▏      | 1032/3252 [43:09<1:29:01,  2.41s/it]
 32%|███▏      | 1033/3252 [43:12<1:28:54,  2.40s/it]
 32%|███▏      | 1034/3252 [43:14<1:28:41,  2.40s/it]
 32%|███▏      | 1035/3252 [43:17<1:28:39,  2.40s/it]
 32%|███▏      | 1036/3252 [43:19<1:28:33,  2.40s/it]
 32%|███▏      | 1037/3252 [43:21<1:29:23,  2.42s/it]
 32%|███▏      | 1038/3252 [43:24<1:29:08,  2.42s/it]
 32%|███▏      | 1039/3252 [43:26<1:28:53,  2.41s/it]
 32%|███▏      | 1040/3252 [43:29<1:28:41,  2.41s/it]
                                                     
{'loss': 0.4757, 'grad_norm': 0.23281724750995636, 'learning_rate': 0.0001592291609434663, 'lm_loss': 0.26991167664527893, 'epoch': 1.28}

 32%|███▏      | 1040/3252 [43:29<1:28:41,  2.41s/it]
 32%|███▏      | 1041/3252 [43:31<1:29:01,  2.42s/it]
 32%|███▏      | 1042/3252 [43:33<1:28:43,  2.41s/it]
 32%|███▏      | 1043/3252 [43:36<1:28:31,  2.40s/it]
 32%|███▏      | 1044/3252 [43:38<1:28:19,  2.40s/it]
 32%|███▏      | 1045/3252 [43:41<1:28:14,  2.40s/it]
 32%|███▏      | 1046/3252 [43:43<1:28:08,  2.40s/it]
 32%|███▏      | 1047/3252 [43:45<1:28:02,  2.40s/it]
 32%|███▏      | 1048/3252 [43:48<1:28:00,  2.40s/it]
 32%|███▏      | 1049/3252 [43:50<1:27:58,  2.40s/it]
 32%|███▏      | 1050/3252 [43:53<1:27:55,  2.40s/it]
                                                     
{'loss': 0.4876, 'grad_norm': 0.17325462400913239, 'learning_rate': 0.00015842316900650558, 'lm_loss': 0.44989433884620667, 'epoch': 1.29}

 32%|███▏      | 1050/3252 [43:53<1:27:55,  2.40s/it]
 32%|███▏      | 1051/3252 [43:55<1:28:32,  2.41s/it]
 32%|███▏      | 1052/3252 [43:57<1:28:16,  2.41s/it]
 32%|███▏      | 1053/3252 [44:00<1:28:03,  2.40s/it]
 32%|███▏      | 1054/3252 [44:02<1:28:02,  2.40s/it]
 32%|███▏      | 1055/3252 [44:05<1:30:16,  2.47s/it]
 32%|███▏      | 1056/3252 [44:07<1:29:35,  2.45s/it]
 33%|███▎      | 1057/3252 [44:10<1:32:21,  2.52s/it]
 33%|███▎      | 1058/3252 [44:12<1:31:01,  2.49s/it]
 33%|███▎      | 1059/3252 [44:15<1:29:55,  2.46s/it]
 33%|███▎      | 1060/3252 [44:17<1:29:07,  2.44s/it]
                                                     
{'loss': 0.4622, 'grad_norm': 0.1921130120754242, 'learning_rate': 0.00015761137331759084, 'lm_loss': 0.46096938848495483, 'epoch': 1.3}

 33%|███▎      | 1060/3252 [44:17<1:29:07,  2.44s/it]
 33%|███▎      | 1061/3252 [44:20<1:31:13,  2.50s/it]
 33%|███▎      | 1062/3252 [44:22<1:30:04,  2.47s/it]
 33%|███▎      | 1063/3252 [44:25<1:29:18,  2.45s/it]
 33%|███▎      | 1064/3252 [44:27<1:28:43,  2.43s/it]
 33%|███▎      | 1065/3252 [44:29<1:28:21,  2.42s/it]
 33%|███▎      | 1066/3252 [44:32<1:28:06,  2.42s/it]
 33%|███▎      | 1067/3252 [44:34<1:27:51,  2.41s/it]
 33%|███▎      | 1068/3252 [44:37<1:27:45,  2.41s/it]
 33%|███▎      | 1069/3252 [44:39<1:27:35,  2.41s/it]
 33%|███▎      | 1070/3252 [44:41<1:27:35,  2.41s/it]
                                                     
{'loss': 0.4831, 'grad_norm': 0.1921919882297516, 'learning_rate': 0.00015679385452042722, 'lm_loss': 0.5624741911888123, 'epoch': 1.32}

 33%|███▎      | 1070/3252 [44:41<1:27:35,  2.41s/it]
 33%|███▎      | 1071/3252 [44:44<1:27:43,  2.41s/it]
 33%|███▎      | 1072/3252 [44:46<1:27:25,  2.41s/it]
 33%|███▎      | 1073/3252 [44:49<1:27:22,  2.41s/it]
 33%|███▎      | 1074/3252 [44:51<1:27:51,  2.42s/it]
 33%|███▎      | 1075/3252 [44:53<1:27:31,  2.41s/it]
 33%|███▎      | 1076/3252 [44:56<1:27:14,  2.41s/it]
 33%|███▎      | 1077/3252 [44:58<1:27:08,  2.40s/it]
 33%|███▎      | 1078/3252 [45:01<1:26:54,  2.40s/it]
 33%|███▎      | 1079/3252 [45:03<1:26:44,  2.40s/it]
 33%|███▎      | 1080/3252 [45:05<1:26:37,  2.39s/it]
                                                     
{'loss': 0.4889, 'grad_norm': 0.20126207172870636, 'learning_rate': 0.00015597069382725293, 'lm_loss': 0.04485642537474632, 'epoch': 1.33}

 33%|███▎      | 1080/3252 [45:06<1:26:37,  2.39s/it]
 33%|███▎      | 1081/3252 [45:08<1:27:27,  2.42s/it]
 33%|███▎      | 1082/3252 [45:10<1:27:11,  2.41s/it]
 33%|███▎      | 1083/3252 [45:13<1:27:04,  2.41s/it]
 33%|███▎      | 1084/3252 [45:15<1:27:00,  2.41s/it]
 33%|███▎      | 1085/3252 [45:18<1:26:52,  2.41s/it]
 33%|███▎      | 1086/3252 [45:20<1:26:42,  2.40s/it]
 33%|███▎      | 1087/3252 [45:22<1:26:30,  2.40s/it]
 33%|███▎      | 1088/3252 [45:25<1:26:32,  2.40s/it]
 33%|███▎      | 1089/3252 [45:27<1:26:25,  2.40s/it]
 34%|███▎      | 1090/3252 [45:30<1:26:23,  2.40s/it]
                                                     
{'loss': 0.4491, 'grad_norm': 0.24277453124523163, 'learning_rate': 0.0001551419730107716, 'lm_loss': 0.042949844151735306, 'epoch': 1.34}

 34%|███▎      | 1090/3252 [45:30<1:26:23,  2.40s/it]
 34%|███▎      | 1091/3252 [45:32<1:28:20,  2.45s/it]
 34%|███▎      | 1092/3252 [45:34<1:27:37,  2.43s/it]
 34%|███▎      | 1093/3252 [45:37<1:27:10,  2.42s/it]
 34%|███▎      | 1094/3252 [45:39<1:26:50,  2.41s/it]
 34%|███▎      | 1095/3252 [45:42<1:26:48,  2.41s/it]
 34%|███▎      | 1096/3252 [45:44<1:26:24,  2.40s/it]
 34%|███▎      | 1097/3252 [45:46<1:26:14,  2.40s/it]
 34%|███▍      | 1098/3252 [45:49<1:26:05,  2.40s/it]
 34%|███▍      | 1099/3252 [45:51<1:25:53,  2.39s/it]
 34%|███▍      | 1100/3252 [45:54<1:25:43,  2.39s/it]
                                                     
{'loss': 0.4887, 'grad_norm': 0.57740718126297, 'learning_rate': 0.00015430777439602876, 'lm_loss': 0.05568463355302811, 'epoch': 1.35}

 34%|███▍      | 1100/3252 [45:54<1:25:43,  2.39s/it]
 34%|███▍      | 1101/3252 [45:56<1:26:24,  2.41s/it]
 34%|███▍      | 1102/3252 [45:58<1:26:11,  2.41s/it]
 34%|███▍      | 1103/3252 [46:01<1:26:00,  2.40s/it]
 34%|███▍      | 1104/3252 [46:03<1:25:56,  2.40s/it]
 34%|███▍      | 1105/3252 [46:06<1:25:53,  2.40s/it]
 34%|███▍      | 1106/3252 [46:08<1:26:05,  2.41s/it]
 34%|███▍      | 1107/3252 [46:10<1:26:01,  2.41s/it]
 34%|███▍      | 1108/3252 [46:13<1:25:53,  2.40s/it]
 34%|███▍      | 1109/3252 [46:15<1:25:55,  2.41s/it]
 34%|███▍      | 1110/3252 [46:18<1:25:42,  2.40s/it]
                                                     
{'loss': 0.4711, 'grad_norm': 0.2566615343093872, 'learning_rate': 0.00015346818085223423, 'lm_loss': 1.4637768268585205, 'epoch': 1.37}

 34%|███▍      | 1110/3252 [46:18<1:25:42,  2.40s/it]
 34%|███▍      | 1111/3252 [46:20<1:27:09,  2.44s/it]
 34%|███▍      | 1112/3252 [46:23<1:26:32,  2.43s/it]
 34%|███▍      | 1113/3252 [46:25<1:26:13,  2.42s/it]
 34%|███▍      | 1114/3252 [46:27<1:25:59,  2.41s/it]
 34%|███▍      | 1115/3252 [46:30<1:25:52,  2.41s/it]
 34%|███▍      | 1116/3252 [46:32<1:25:54,  2.41s/it]
 34%|███▍      | 1117/3252 [46:35<1:25:41,  2.41s/it]
 34%|███▍      | 1118/3252 [46:37<1:25:32,  2.41s/it]
 34%|███▍      | 1119/3252 [46:39<1:25:17,  2.40s/it]
 34%|███▍      | 1120/3252 [46:42<1:25:10,  2.40s/it]
                                                     
{'loss': 0.4603, 'grad_norm': 0.2698190212249756, 'learning_rate': 0.00015262327578452925, 'lm_loss': 0.06902853399515152, 'epoch': 1.38}

 34%|███▍      | 1120/3252 [46:42<1:25:10,  2.40s/it]
 34%|███▍      | 1121/3252 [46:44<1:26:49,  2.44s/it]
 35%|███▍      | 1122/3252 [46:47<1:26:19,  2.43s/it]
 35%|███▍      | 1123/3252 [46:49<1:25:52,  2.42s/it]
 35%|███▍      | 1124/3252 [46:52<1:25:45,  2.42s/it]
 35%|███▍      | 1125/3252 [46:54<1:25:23,  2.41s/it]
 35%|███▍      | 1126/3252 [46:56<1:25:13,  2.41s/it]
 35%|███▍      | 1127/3252 [46:59<1:25:07,  2.40s/it]
 35%|███▍      | 1128/3252 [47:01<1:24:55,  2.40s/it]
 35%|███▍      | 1129/3252 [47:04<1:27:55,  2.48s/it]
 35%|███▍      | 1130/3252 [47:06<1:26:59,  2.46s/it]
                                                     
{'loss': 0.4644, 'grad_norm': 0.21758610010147095, 'learning_rate': 0.00015177314312570154, 'lm_loss': 0.08074460923671722, 'epoch': 1.39}

 35%|███▍      | 1130/3252 [47:06<1:26:59,  2.46s/it]
 35%|███▍      | 1131/3252 [47:09<1:30:09,  2.55s/it]
 35%|███▍      | 1132/3252 [47:11<1:28:34,  2.51s/it]
 35%|███▍      | 1133/3252 [47:14<1:27:14,  2.47s/it]
 35%|███▍      | 1134/3252 [47:16<1:26:19,  2.45s/it]
 35%|███▍      | 1135/3252 [47:19<1:25:44,  2.43s/it]
 35%|███▍      | 1136/3252 [47:21<1:25:21,  2.42s/it]
 35%|███▍      | 1137/3252 [47:23<1:25:08,  2.42s/it]
 35%|███▍      | 1138/3252 [47:26<1:25:01,  2.41s/it]
 35%|███▌      | 1139/3252 [47:28<1:24:52,  2.41s/it]
 35%|███▌      | 1140/3252 [47:31<1:24:48,  2.41s/it]
                                                     
{'loss': 0.466, 'grad_norm': 0.19413666427135468, 'learning_rate': 0.00015091786732784716, 'lm_loss': 0.1639384925365448, 'epoch': 1.4}

 35%|███▌      | 1140/3252 [47:31<1:24:48,  2.41s/it]
 35%|███▌      | 1141/3252 [47:33<1:25:29,  2.43s/it]
 35%|███▌      | 1142/3252 [47:35<1:25:06,  2.42s/it]
 35%|███▌      | 1143/3252 [47:38<1:24:50,  2.41s/it]
 35%|███▌      | 1144/3252 [47:40<1:24:32,  2.41s/it]
 35%|███▌      | 1145/3252 [47:43<1:24:22,  2.40s/it]
 35%|███▌      | 1146/3252 [47:45<1:24:14,  2.40s/it]
 35%|███▌      | 1147/3252 [47:47<1:24:12,  2.40s/it]
 35%|███▌      | 1148/3252 [47:50<1:25:16,  2.43s/it]
 35%|███▌      | 1149/3252 [47:52<1:24:51,  2.42s/it]
 35%|███▌      | 1150/3252 [47:55<1:24:34,  2.41s/it]
                                                     
{'loss': 0.463, 'grad_norm': 0.18646155297756195, 'learning_rate': 0.000150057533353981, 'lm_loss': 0.7995421290397644, 'epoch': 1.41}

 35%|███▌      | 1150/3252 [47:55<1:24:34,  2.41s/it]
 35%|███▌      | 1151/3252 [47:57<1:27:10,  2.49s/it]
 35%|███▌      | 1152/3252 [48:00<1:26:02,  2.46s/it]
 35%|███▌      | 1153/3252 [48:02<1:25:25,  2.44s/it]
 35%|███▌      | 1154/3252 [48:05<1:24:55,  2.43s/it]
 36%|███▌      | 1155/3252 [48:07<1:24:38,  2.42s/it]
 36%|███▌      | 1156/3252 [48:09<1:24:31,  2.42s/it]
 36%|███▌      | 1157/3252 [48:12<1:25:37,  2.45s/it]
 36%|███▌      | 1158/3252 [48:14<1:24:59,  2.44s/it]
 36%|███▌      | 1159/3252 [48:17<1:24:27,  2.42s/it]
 36%|███▌      | 1160/3252 [48:19<1:24:02,  2.41s/it]
                                                     
{'loss': 0.4567, 'grad_norm': 0.19865061342716217, 'learning_rate': 0.0001491922266695968, 'lm_loss': 0.5346617698669434, 'epoch': 1.43}

 36%|███▌      | 1160/3252 [48:19<1:24:02,  2.41s/it]
 36%|███▌      | 1161/3252 [48:22<1:24:09,  2.41s/it]
 36%|███▌      | 1162/3252 [48:24<1:23:57,  2.41s/it]
 36%|███▌      | 1163/3252 [48:26<1:24:21,  2.42s/it]
 36%|███▌      | 1164/3252 [48:29<1:24:48,  2.44s/it]
 36%|███▌      | 1165/3252 [48:31<1:26:10,  2.48s/it]
 36%|███▌      | 1166/3252 [48:34<1:25:22,  2.46s/it]
 36%|███▌      | 1167/3252 [48:36<1:24:40,  2.44s/it]
 36%|███▌      | 1168/3252 [48:39<1:24:12,  2.42s/it]
 36%|███▌      | 1169/3252 [48:41<1:23:52,  2.42s/it]
 36%|███▌      | 1170/3252 [48:43<1:23:36,  2.41s/it]
                                                     
{'loss': 0.4657, 'grad_norm': 0.18268738687038422, 'learning_rate': 0.0001483220332341768, 'lm_loss': 0.549036979675293, 'epoch': 1.44}

 36%|███▌      | 1170/3252 [48:44<1:23:36,  2.41s/it]
 36%|███▌      | 1171/3252 [48:46<1:26:04,  2.48s/it]
 36%|███▌      | 1172/3252 [48:48<1:25:13,  2.46s/it]
 36%|███▌      | 1173/3252 [48:51<1:24:34,  2.44s/it]
 36%|███▌      | 1174/3252 [48:53<1:24:07,  2.43s/it]
 36%|███▌      | 1175/3252 [48:56<1:23:49,  2.42s/it]
 36%|███▌      | 1176/3252 [48:58<1:23:28,  2.41s/it]
 36%|███▌      | 1177/3252 [49:00<1:23:15,  2.41s/it]
 36%|███▌      | 1178/3252 [49:03<1:23:21,  2.41s/it]
 36%|███▋      | 1179/3252 [49:05<1:23:05,  2.41s/it]
 36%|███▋      | 1180/3252 [49:08<1:22:56,  2.40s/it]
                                                     
{'loss': 0.4607, 'grad_norm': 0.17528881132602692, 'learning_rate': 0.00014744703949265268, 'lm_loss': 0.16923624277114868, 'epoch': 1.45}

 36%|███▋      | 1180/3252 [49:08<1:22:56,  2.40s/it]
 36%|███▋      | 1181/3252 [49:10<1:23:04,  2.41s/it]
 36%|███▋      | 1182/3252 [49:12<1:22:54,  2.40s/it]
 36%|███▋      | 1183/3252 [49:15<1:22:46,  2.40s/it]
 36%|███▋      | 1184/3252 [49:17<1:22:42,  2.40s/it]
 36%|███▋      | 1185/3252 [49:20<1:24:25,  2.45s/it]
 36%|███▋      | 1186/3252 [49:22<1:23:54,  2.44s/it]
 37%|███▋      | 1187/3252 [49:25<1:23:24,  2.42s/it]
 37%|███▋      | 1188/3252 [49:27<1:23:09,  2.42s/it]
 37%|███▋      | 1189/3252 [49:29<1:22:52,  2.41s/it]
 37%|███▋      | 1190/3252 [49:32<1:22:44,  2.41s/it]
                                                     
{'loss': 0.4679, 'grad_norm': 0.21560712158679962, 'learning_rate': 0.00014656733236681802, 'lm_loss': 0.16830316185951233, 'epoch': 1.46}

 37%|███▋      | 1190/3252 [49:32<1:22:44,  2.41s/it]
 37%|███▋      | 1191/3252 [49:34<1:23:18,  2.43s/it]
 37%|███▋      | 1192/3252 [49:37<1:23:02,  2.42s/it]
 37%|███▋      | 1193/3252 [49:39<1:22:49,  2.41s/it]
 37%|███▋      | 1194/3252 [49:42<1:22:42,  2.41s/it]
 37%|███▋      | 1195/3252 [49:44<1:22:33,  2.41s/it]
 37%|███▋      | 1196/3252 [49:46<1:22:31,  2.41s/it]
 37%|███▋      | 1197/3252 [49:49<1:22:12,  2.40s/it]
 37%|███▋      | 1198/3252 [49:51<1:22:04,  2.40s/it]
 37%|███▋      | 1199/3252 [49:53<1:22:02,  2.40s/it]
 37%|███▋      | 1200/3252 [49:56<1:22:04,  2.40s/it]
                                                     
{'loss': 0.4355, 'grad_norm': 0.3305585980415344, 'learning_rate': 0.0001456829992466935, 'lm_loss': 0.6739005446434021, 'epoch': 1.48}

 37%|███▋      | 1200/3252 [49:56<1:22:04,  2.40s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.22it/s]
                                                     

                                             
{'eval_sthsthv2/narration_val_loss': 0.773794949054718, 'eval_sthsthv2/narration_val_runtime': 3.4238, 'eval_sthsthv2/narration_val_samples_per_second': 195.107, 'eval_sthsthv2/narration_val_steps_per_second': 0.584, 'epoch': 1.48}

 37%|███▋      | 1200/3252 [50:00<1:22:04,  2.40s/it]

100%|██████████| 2/2 [00:00<00:00,  2.22it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.53it/s]
                                                     

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6315209865570068, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.2238, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 183.634, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.62, 'epoch': 1.48}

 37%|███▋      | 1200/3252 [50:03<1:22:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.53it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.04it/s]
                                                     

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.5992239117622375, 'eval_holoassist/dialog-klg-sum_val_runtime': 3.0454, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 232.806, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.657, 'epoch': 1.48}

 37%|███▋      | 1200/3252 [50:06<1:22:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  2.04it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                     

                                     
{'eval_ego4d/narration_val_loss': 0.5381870865821838, 'eval_ego4d/narration_val_runtime': 2.2274, 'eval_ego4d/narration_val_samples_per_second': 100.119, 'eval_ego4d/narration_val_steps_per_second': 0.449, 'epoch': 1.48}

 37%|███▋      | 1200/3252 [50:08<1:22:04,  2.40s/it]

100%|██████████| 1/1 [00:00<00:00,  3.64it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.47it/s]
                                                     

                                             
{'eval_ego4d/summary_val_loss': 1.1461888551712036, 'eval_ego4d/summary_val_runtime': 2.9669, 'eval_ego4d/summary_val_samples_per_second': 190.099, 'eval_ego4d/summary_val_steps_per_second': 0.674, 'epoch': 1.48}

 37%|███▋      | 1200/3252 [50:11<1:22:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.47it/s]

                                             
 37%|███▋      | 1201/3252 [50:14<4:00:00,  7.02s/it]
 37%|███▋      | 1202/3252 [50:16<3:12:27,  5.63s/it]
 37%|███▋      | 1203/3252 [50:19<2:39:58,  4.68s/it]
 37%|███▋      | 1204/3252 [50:21<2:16:28,  4.00s/it]
 37%|███▋      | 1205/3252 [50:24<2:01:37,  3.56s/it]
 37%|███▋      | 1206/3252 [50:26<1:49:40,  3.22s/it]
 37%|███▋      | 1207/3252 [50:28<1:41:13,  2.97s/it]
 37%|███▋      | 1208/3252 [50:31<1:35:24,  2.80s/it]
 37%|███▋      | 1209/3252 [50:33<1:31:16,  2.68s/it]
 37%|███▋      | 1210/3252 [50:36<1:28:24,  2.60s/it]
                                                     
{'loss': 0.4619, 'grad_norm': 0.267246812582016, 'learning_rate': 0.00014479412798184575, 'lm_loss': 0.8040302991867065, 'epoch': 1.49}

 37%|███▋      | 1210/3252 [50:36<1:28:24,  2.60s/it]
 37%|███▋      | 1211/3252 [50:38<1:27:40,  2.58s/it]
 37%|███▋      | 1212/3252 [50:40<1:25:44,  2.52s/it]
 37%|███▋      | 1213/3252 [50:43<1:24:23,  2.48s/it]
 37%|███▋      | 1214/3252 [50:45<1:23:23,  2.46s/it]
 37%|███▋      | 1215/3252 [50:48<1:22:41,  2.44s/it]
 37%|███▋      | 1216/3252 [50:50<1:22:12,  2.42s/it]
 37%|███▋      | 1217/3252 [50:52<1:21:49,  2.41s/it]
 37%|███▋      | 1218/3252 [50:55<1:21:29,  2.40s/it]
 37%|███▋      | 1219/3252 [50:57<1:21:17,  2.40s/it]
 38%|███▊      | 1220/3252 [51:00<1:21:12,  2.40s/it]
                                                     
{'loss': 0.4569, 'grad_norm': 0.17447713017463684, 'learning_rate': 0.00014390080687266013, 'lm_loss': 0.6667245030403137, 'epoch': 1.5}

 38%|███▊      | 1220/3252 [51:00<1:21:12,  2.40s/it]
 38%|███▊      | 1221/3252 [51:02<1:21:05,  2.40s/it]
 38%|███▊      | 1222/3252 [51:04<1:21:04,  2.40s/it]
 38%|███▊      | 1223/3252 [51:07<1:21:07,  2.40s/it]
 38%|███▊      | 1224/3252 [51:09<1:22:04,  2.43s/it]
 38%|███▊      | 1225/3252 [51:12<1:21:34,  2.41s/it]
 38%|███▊      | 1226/3252 [51:14<1:21:20,  2.41s/it]
 38%|███▊      | 1227/3252 [51:16<1:21:08,  2.40s/it]
 38%|███▊      | 1228/3252 [51:19<1:21:11,  2.41s/it]
 38%|███▊      | 1229/3252 [51:21<1:21:09,  2.41s/it]
 38%|███▊      | 1230/3252 [51:24<1:21:03,  2.41s/it]
                                                     
{'loss': 0.4453, 'grad_norm': 0.20868530869483948, 'learning_rate': 0.00014300312466156918, 'lm_loss': 0.005393315106630325, 'epoch': 1.51}

 38%|███▊      | 1230/3252 [51:24<1:21:03,  2.41s/it]
 38%|███▊      | 1231/3252 [51:26<1:22:13,  2.44s/it]
 38%|███▊      | 1232/3252 [51:29<1:21:39,  2.43s/it]
 38%|███▊      | 1233/3252 [51:31<1:21:17,  2.42s/it]
 38%|███▊      | 1234/3252 [51:33<1:21:04,  2.41s/it]
 38%|███▊      | 1235/3252 [51:36<1:20:56,  2.41s/it]
 38%|███▊      | 1236/3252 [51:38<1:20:49,  2.41s/it]
 38%|███▊      | 1237/3252 [51:41<1:20:37,  2.40s/it]
 38%|███▊      | 1238/3252 [51:43<1:20:30,  2.40s/it]
 38%|███▊      | 1239/3252 [51:45<1:20:26,  2.40s/it]
 38%|███▊      | 1240/3252 [51:48<1:20:54,  2.41s/it]
                                                     
{'loss': 0.4419, 'grad_norm': 0.3748989999294281, 'learning_rate': 0.00014210117052423685, 'lm_loss': 0.6862514019012451, 'epoch': 1.53}

 38%|███▊      | 1240/3252 [51:48<1:20:54,  2.41s/it]
 38%|███▊      | 1241/3252 [51:50<1:21:47,  2.44s/it]
 38%|███▊      | 1242/3252 [51:53<1:22:23,  2.46s/it]
 38%|███▊      | 1243/3252 [51:55<1:21:39,  2.44s/it]
 38%|███▊      | 1244/3252 [51:58<1:21:08,  2.42s/it]
 38%|███▊      | 1245/3252 [52:00<1:20:43,  2.41s/it]
 38%|███▊      | 1246/3252 [52:02<1:20:28,  2.41s/it]
 38%|███▊      | 1247/3252 [52:05<1:20:24,  2.41s/it]
 38%|███▊      | 1248/3252 [52:07<1:20:05,  2.40s/it]
 38%|███▊      | 1249/3252 [52:10<1:19:56,  2.39s/it]
 38%|███▊      | 1250/3252 [52:12<1:19:55,  2.40s/it]
                                                     
{'loss': 0.4485, 'grad_norm': 0.3960059583187103, 'learning_rate': 0.00014119503406069978, 'lm_loss': 0.25357523560523987, 'epoch': 1.54}

 38%|███▊      | 1250/3252 [52:12<1:19:55,  2.40s/it]
 38%|███▊      | 1251/3252 [52:14<1:20:08,  2.40s/it]
 38%|███▊      | 1252/3252 [52:17<1:20:02,  2.40s/it]
 39%|███▊      | 1253/3252 [52:19<1:20:02,  2.40s/it]
 39%|███▊      | 1254/3252 [52:22<1:19:52,  2.40s/it]
 39%|███▊      | 1255/3252 [52:24<1:19:50,  2.40s/it]
 39%|███▊      | 1256/3252 [52:26<1:19:43,  2.40s/it]
 39%|███▊      | 1257/3252 [52:29<1:19:38,  2.40s/it]
 39%|███▊      | 1258/3252 [52:31<1:19:32,  2.39s/it]
 39%|███▊      | 1259/3252 [52:34<1:20:25,  2.42s/it]
 39%|███▊      | 1260/3252 [52:36<1:20:12,  2.42s/it]
                                                     
{'loss': 0.4461, 'grad_norm': 0.1852470487356186, 'learning_rate': 0.00014028480528646667, 'lm_loss': 0.40021049976348877, 'epoch': 1.55}

 39%|███▊      | 1260/3252 [52:36<1:20:12,  2.42s/it]
 39%|███▉      | 1261/3252 [52:39<1:21:32,  2.46s/it]
 39%|███▉      | 1262/3252 [52:41<1:20:51,  2.44s/it]
 39%|███▉      | 1263/3252 [52:43<1:20:22,  2.42s/it]
 39%|███▉      | 1264/3252 [52:46<1:20:13,  2.42s/it]
 39%|███▉      | 1265/3252 [52:48<1:19:56,  2.41s/it]
 39%|███▉      | 1266/3252 [52:51<1:19:46,  2.41s/it]
 39%|███▉      | 1267/3252 [52:53<1:19:29,  2.40s/it]
 39%|███▉      | 1268/3252 [52:55<1:19:23,  2.40s/it]
 39%|███▉      | 1269/3252 [52:58<1:19:22,  2.40s/it]
 39%|███▉      | 1270/3252 [53:00<1:19:36,  2.41s/it]
                                                     
{'loss': 0.4454, 'grad_norm': 0.1838463395833969, 'learning_rate': 0.00013937057462357576, 'lm_loss': 0.008208313025534153, 'epoch': 1.56}

 39%|███▉      | 1270/3252 [53:00<1:19:36,  2.41s/it]
 39%|███▉      | 1271/3252 [53:03<1:20:35,  2.44s/it]
 39%|███▉      | 1272/3252 [53:05<1:20:11,  2.43s/it]
 39%|███▉      | 1273/3252 [53:07<1:19:45,  2.42s/it]
 39%|███▉      | 1274/3252 [53:10<1:19:32,  2.41s/it]
 39%|███▉      | 1275/3252 [53:12<1:19:47,  2.42s/it]
 39%|███▉      | 1276/3252 [53:15<1:19:48,  2.42s/it]
 39%|███▉      | 1277/3252 [53:17<1:19:31,  2.42s/it]
 39%|███▉      | 1278/3252 [53:20<1:20:15,  2.44s/it]
 39%|███▉      | 1279/3252 [53:22<1:19:52,  2.43s/it]
 39%|███▉      | 1280/3252 [53:25<1:20:47,  2.46s/it]
                                                     
{'loss': 0.463, 'grad_norm': 0.2053581029176712, 'learning_rate': 0.00013845243289161262, 'lm_loss': 0.008799935691058636, 'epoch': 1.57}

 39%|███▉      | 1280/3252 [53:25<1:20:47,  2.46s/it]
 39%|███▉      | 1281/3252 [53:27<1:20:29,  2.45s/it]
 39%|███▉      | 1282/3252 [53:29<1:19:57,  2.44s/it]
 39%|███▉      | 1283/3252 [53:32<1:19:32,  2.42s/it]
 39%|███▉      | 1284/3252 [53:34<1:19:11,  2.41s/it]
 40%|███▉      | 1285/3252 [53:37<1:18:54,  2.41s/it]
 40%|███▉      | 1286/3252 [53:39<1:18:41,  2.40s/it]
 40%|███▉      | 1287/3252 [53:41<1:18:35,  2.40s/it]
 40%|███▉      | 1288/3252 [53:44<1:18:27,  2.40s/it]
 40%|███▉      | 1289/3252 [53:46<1:18:20,  2.39s/it]
 40%|███▉      | 1290/3252 [53:49<1:18:30,  2.40s/it]
                                                     
{'loss': 0.4388, 'grad_norm': 0.2973194718360901, 'learning_rate': 0.00013753047129868808, 'lm_loss': 0.20929266512393951, 'epoch': 1.59}

 40%|███▉      | 1290/3252 [53:49<1:18:30,  2.40s/it]
 40%|███▉      | 1291/3252 [53:51<1:19:12,  2.42s/it]
 40%|███▉      | 1292/3252 [53:53<1:18:52,  2.41s/it]
 40%|███▉      | 1293/3252 [53:56<1:18:40,  2.41s/it]
 40%|███▉      | 1294/3252 [53:58<1:18:26,  2.40s/it]
 40%|███▉      | 1295/3252 [54:01<1:18:21,  2.40s/it]
 40%|███▉      | 1296/3252 [54:03<1:18:14,  2.40s/it]
 40%|███▉      | 1297/3252 [54:05<1:18:10,  2.40s/it]
 40%|███▉      | 1298/3252 [54:08<1:19:15,  2.43s/it]
 40%|███▉      | 1299/3252 [54:10<1:18:50,  2.42s/it]
 40%|███▉      | 1300/3252 [54:13<1:21:31,  2.51s/it]
                                                     
{'loss': 0.4612, 'grad_norm': 0.19541998207569122, 'learning_rate': 0.00013660478143237746, 'lm_loss': 1.423551082611084, 'epoch': 1.6}

 40%|███▉      | 1300/3252 [54:13<1:21:31,  2.51s/it]
 40%|████      | 1301/3252 [54:15<1:20:47,  2.48s/it]
 40%|████      | 1302/3252 [54:18<1:19:54,  2.46s/it]
 40%|████      | 1303/3252 [54:20<1:19:16,  2.44s/it]
 40%|████      | 1304/3252 [54:23<1:18:53,  2.43s/it]
 40%|████      | 1305/3252 [54:25<1:18:30,  2.42s/it]
 40%|████      | 1306/3252 [54:27<1:18:11,  2.41s/it]
 40%|████      | 1307/3252 [54:30<1:17:55,  2.40s/it]
 40%|████      | 1308/3252 [54:32<1:17:47,  2.40s/it]
 40%|████      | 1309/3252 [54:35<1:17:38,  2.40s/it]
 40%|████      | 1310/3252 [54:37<1:17:37,  2.40s/it]
                                                     
{'loss': 0.4305, 'grad_norm': 0.23916029930114746, 'learning_rate': 0.00013567545525062255, 'lm_loss': 0.35733217000961304, 'epoch': 1.61}

 40%|████      | 1310/3252 [54:37<1:17:37,  2.40s/it]
 40%|████      | 1311/3252 [54:39<1:18:00,  2.41s/it]
 40%|████      | 1312/3252 [54:42<1:17:41,  2.40s/it]
 40%|████      | 1313/3252 [54:44<1:17:32,  2.40s/it]
 40%|████      | 1314/3252 [54:47<1:17:25,  2.40s/it]
 40%|████      | 1315/3252 [54:49<1:17:48,  2.41s/it]
 40%|████      | 1316/3252 [54:51<1:17:38,  2.41s/it]
 40%|████      | 1317/3252 [54:54<1:19:14,  2.46s/it]
 41%|████      | 1318/3252 [54:56<1:18:37,  2.44s/it]
 41%|████      | 1319/3252 [54:59<1:18:09,  2.43s/it]
 41%|████      | 1320/3252 [55:01<1:18:01,  2.42s/it]
                                                     
{'loss': 0.4472, 'grad_norm': 0.28414905071258545, 'learning_rate': 0.00013474258507259622, 'lm_loss': 0.0461992584168911, 'epoch': 1.62}

 41%|████      | 1320/3252 [55:01<1:18:01,  2.42s/it]
 41%|████      | 1321/3252 [55:04<1:18:35,  2.44s/it]
 41%|████      | 1322/3252 [55:06<1:18:04,  2.43s/it]
 41%|████      | 1323/3252 [55:09<1:17:45,  2.42s/it]
 41%|████      | 1324/3252 [55:11<1:17:27,  2.41s/it]
 41%|████      | 1325/3252 [55:13<1:17:12,  2.40s/it]
 41%|████      | 1326/3252 [55:16<1:17:04,  2.40s/it]
 41%|████      | 1327/3252 [55:18<1:17:15,  2.41s/it]
 41%|████      | 1328/3252 [55:20<1:17:01,  2.40s/it]
 41%|████      | 1329/3252 [55:23<1:16:58,  2.40s/it]
 41%|████      | 1330/3252 [55:25<1:16:48,  2.40s/it]
                                                     
{'loss': 0.4463, 'grad_norm': 0.1776627153158188, 'learning_rate': 0.00013380626356953175, 'lm_loss': 0.5639379024505615, 'epoch': 1.64}

 41%|████      | 1330/3252 [55:26<1:16:48,  2.40s/it]
 41%|████      | 1331/3252 [55:28<1:17:40,  2.43s/it]
 41%|████      | 1332/3252 [55:30<1:17:22,  2.42s/it]
 41%|████      | 1333/3252 [55:33<1:17:01,  2.41s/it]
 41%|████      | 1334/3252 [55:35<1:18:40,  2.46s/it]
 41%|████      | 1335/3252 [55:38<1:17:59,  2.44s/it]
 41%|████      | 1336/3252 [55:40<1:17:23,  2.42s/it]
 41%|████      | 1337/3252 [55:42<1:16:56,  2.41s/it]
 41%|████      | 1338/3252 [55:45<1:16:43,  2.41s/it]
 41%|████      | 1339/3252 [55:47<1:16:34,  2.40s/it]
 41%|████      | 1340/3252 [55:49<1:16:27,  2.40s/it]
                                                     
{'loss': 0.4408, 'grad_norm': 0.18398594856262207, 'learning_rate': 0.00013286658375551654, 'lm_loss': 1.350284218788147, 'epoch': 1.65}

 41%|████      | 1340/3252 [55:50<1:16:27,  2.40s/it]
 41%|████      | 1341/3252 [55:52<1:16:41,  2.41s/it]
 41%|████▏     | 1342/3252 [55:54<1:16:35,  2.41s/it]
 41%|████▏     | 1343/3252 [55:57<1:16:26,  2.40s/it]
 41%|████▏     | 1344/3252 [55:59<1:16:15,  2.40s/it]
 41%|████▏     | 1345/3252 [56:01<1:16:13,  2.40s/it]
 41%|████▏     | 1346/3252 [56:04<1:16:10,  2.40s/it]
 41%|████▏     | 1347/3252 [56:06<1:16:10,  2.40s/it]
 41%|████▏     | 1348/3252 [56:09<1:16:05,  2.40s/it]
 41%|████▏     | 1349/3252 [56:11<1:16:05,  2.40s/it]
 42%|████▏     | 1350/3252 [56:14<1:16:27,  2.41s/it]
                                                     
{'loss': 0.4391, 'grad_norm': 0.18909013271331787, 'learning_rate': 0.0001319236389782524, 'lm_loss': 0.035625897347927094, 'epoch': 1.66}

 42%|████▏     | 1350/3252 [56:14<1:16:27,  2.41s/it]
 42%|████▏     | 1351/3252 [56:16<1:17:11,  2.44s/it]
 42%|████▏     | 1352/3252 [56:18<1:17:08,  2.44s/it]
 42%|████▏     | 1353/3252 [56:21<1:16:43,  2.42s/it]
 42%|████▏     | 1354/3252 [56:23<1:17:54,  2.46s/it]
 42%|████▏     | 1355/3252 [56:26<1:17:07,  2.44s/it]
 42%|████▏     | 1356/3252 [56:28<1:16:44,  2.43s/it]
 42%|████▏     | 1357/3252 [56:31<1:16:25,  2.42s/it]
 42%|████▏     | 1358/3252 [56:33<1:16:07,  2.41s/it]
 42%|████▏     | 1359/3252 [56:35<1:15:54,  2.41s/it]
 42%|████▏     | 1360/3252 [56:38<1:15:44,  2.40s/it]
                                                     
{'loss': 0.4273, 'grad_norm': 0.20881766080856323, 'learning_rate': 0.00013097752290978218, 'lm_loss': 0.7139782905578613, 'epoch': 1.67}

 42%|████▏     | 1360/3252 [56:38<1:15:44,  2.40s/it]
 42%|████▏     | 1361/3252 [56:40<1:15:52,  2.41s/it]
 42%|████▏     | 1362/3252 [56:43<1:15:42,  2.40s/it]
 42%|████▏     | 1363/3252 [56:45<1:15:48,  2.41s/it]
 42%|████▏     | 1364/3252 [56:47<1:15:37,  2.40s/it]
 42%|████▏     | 1365/3252 [56:50<1:15:30,  2.40s/it]
 42%|████▏     | 1366/3252 [56:52<1:15:27,  2.40s/it]
 42%|████▏     | 1367/3252 [56:55<1:15:17,  2.40s/it]
 42%|████▏     | 1368/3252 [56:57<1:15:10,  2.39s/it]
 42%|████▏     | 1369/3252 [56:59<1:15:07,  2.39s/it]
 42%|████▏     | 1370/3252 [57:02<1:15:04,  2.39s/it]
                                                     
{'loss': 0.4113, 'grad_norm': 0.20014755427837372, 'learning_rate': 0.00013002832953718462, 'lm_loss': 0.45815110206604004, 'epoch': 1.69}

 42%|████▏     | 1370/3252 [57:02<1:15:04,  2.39s/it]
 42%|████▏     | 1371/3252 [57:04<1:15:24,  2.41s/it]
 42%|████▏     | 1372/3252 [57:07<1:15:25,  2.41s/it]
 42%|████▏     | 1373/3252 [57:09<1:15:16,  2.40s/it]
 42%|████▏     | 1374/3252 [57:12<1:17:51,  2.49s/it]
 42%|████▏     | 1375/3252 [57:14<1:16:59,  2.46s/it]
 42%|████▏     | 1376/3252 [57:16<1:16:17,  2.44s/it]
 42%|████▏     | 1377/3252 [57:19<1:15:48,  2.43s/it]
 42%|████▏     | 1378/3252 [57:21<1:15:26,  2.42s/it]
 42%|████▏     | 1379/3252 [57:24<1:15:16,  2.41s/it]
 42%|████▏     | 1380/3252 [57:26<1:15:06,  2.41s/it]
                                                     
{'loss': 0.4341, 'grad_norm': 0.2139282375574112, 'learning_rate': 0.0001290761531532374, 'lm_loss': 0.3515787124633789, 'epoch': 1.7}

 42%|████▏     | 1380/3252 [57:26<1:15:06,  2.41s/it]
 42%|████▏     | 1381/3252 [57:29<1:16:04,  2.44s/it]
 42%|████▏     | 1382/3252 [57:31<1:15:31,  2.42s/it]
 43%|████▎     | 1383/3252 [57:33<1:15:16,  2.42s/it]
 43%|████▎     | 1384/3252 [57:36<1:14:54,  2.41s/it]
 43%|████▎     | 1385/3252 [57:38<1:14:44,  2.40s/it]
 43%|████▎     | 1386/3252 [57:41<1:14:38,  2.40s/it]
 43%|████▎     | 1387/3252 [57:43<1:14:30,  2.40s/it]
 43%|████▎     | 1388/3252 [57:45<1:14:24,  2.40s/it]
 43%|████▎     | 1389/3252 [57:48<1:16:28,  2.46s/it]
 43%|████▎     | 1390/3252 [57:50<1:15:53,  2.45s/it]
                                                     
{'loss': 0.438, 'grad_norm': 0.30156782269477844, 'learning_rate': 0.0001281210883470504, 'lm_loss': 0.5917488932609558, 'epoch': 1.71}

 43%|████▎     | 1390/3252 [57:50<1:15:53,  2.45s/it]
 43%|████▎     | 1391/3252 [57:53<1:16:57,  2.48s/it]
 43%|████▎     | 1392/3252 [57:55<1:16:03,  2.45s/it]
 43%|████▎     | 1393/3252 [57:58<1:15:23,  2.43s/it]
 43%|████▎     | 1394/3252 [58:00<1:15:06,  2.43s/it]
 43%|████▎     | 1395/3252 [58:02<1:14:54,  2.42s/it]
 43%|████▎     | 1396/3252 [58:05<1:14:37,  2.41s/it]
 43%|████▎     | 1397/3252 [58:07<1:14:25,  2.41s/it]
 43%|████▎     | 1398/3252 [58:10<1:14:16,  2.40s/it]
 43%|████▎     | 1399/3252 [58:12<1:14:09,  2.40s/it]
 43%|████▎     | 1400/3252 [58:14<1:14:02,  2.40s/it]
                                                     
{'loss': 0.4365, 'grad_norm': 0.181928351521492, 'learning_rate': 0.000127163229994669, 'lm_loss': 0.016362264752388, 'epoch': 1.72}

 43%|████▎     | 1400/3252 [58:15<1:14:02,  2.40s/it]
 43%|████▎     | 1401/3252 [58:17<1:15:51,  2.46s/it]
 43%|████▎     | 1402/3252 [58:19<1:15:32,  2.45s/it]
 43%|████▎     | 1403/3252 [58:22<1:15:28,  2.45s/it]
 43%|████▎     | 1404/3252 [58:24<1:14:56,  2.43s/it]
 43%|████▎     | 1405/3252 [58:27<1:14:33,  2.42s/it]
 43%|████▎     | 1406/3252 [58:29<1:14:21,  2.42s/it]
 43%|████▎     | 1407/3252 [58:32<1:14:09,  2.41s/it]
 43%|████▎     | 1408/3252 [58:34<1:14:57,  2.44s/it]
 43%|████▎     | 1409/3252 [58:36<1:14:32,  2.43s/it]
 43%|████▎     | 1410/3252 [58:39<1:14:14,  2.42s/it]
                                                     
{'loss': 0.4407, 'grad_norm': 0.18773388862609863, 'learning_rate': 0.00012620267324964918, 'lm_loss': 0.20787790417671204, 'epoch': 1.73}

 43%|████▎     | 1410/3252 [58:39<1:14:14,  2.42s/it]
 43%|████▎     | 1411/3252 [58:41<1:15:34,  2.46s/it]
 43%|████▎     | 1412/3252 [58:44<1:14:59,  2.45s/it]
 43%|████▎     | 1413/3252 [58:46<1:14:31,  2.43s/it]
 43%|████▎     | 1414/3252 [58:49<1:14:08,  2.42s/it]
 44%|████▎     | 1415/3252 [58:51<1:13:50,  2.41s/it]
 44%|████▎     | 1416/3252 [58:53<1:13:40,  2.41s/it]
 44%|████▎     | 1417/3252 [58:56<1:13:30,  2.40s/it]
 44%|████▎     | 1418/3252 [58:58<1:13:19,  2.40s/it]
 44%|████▎     | 1419/3252 [59:01<1:13:15,  2.40s/it]
 44%|████▎     | 1420/3252 [59:03<1:13:03,  2.39s/it]
                                                     
{'loss': 0.4307, 'grad_norm': 0.35151833295822144, 'learning_rate': 0.00012523951353360504, 'lm_loss': 0.5580066442489624, 'epoch': 1.75}

 44%|████▎     | 1420/3252 [59:03<1:13:03,  2.39s/it]
 44%|████▎     | 1421/3252 [59:05<1:13:12,  2.40s/it]
 44%|████▎     | 1422/3252 [59:08<1:13:11,  2.40s/it]
 44%|████▍     | 1423/3252 [59:10<1:13:08,  2.40s/it]
 44%|████▍     | 1424/3252 [59:13<1:13:03,  2.40s/it]
 44%|████▍     | 1425/3252 [59:15<1:12:56,  2.40s/it]
 44%|████▍     | 1426/3252 [59:18<1:14:33,  2.45s/it]
 44%|████▍     | 1427/3252 [59:20<1:14:03,  2.43s/it]
 44%|████▍     | 1428/3252 [59:22<1:14:32,  2.45s/it]
 44%|████▍     | 1429/3252 [59:25<1:13:56,  2.43s/it]
 44%|████▍     | 1430/3252 [59:27<1:13:31,  2.42s/it]
                                                     
{'loss': 0.4394, 'grad_norm': 0.17565235495567322, 'learning_rate': 0.00012427384652672954, 'lm_loss': 1.4685629606246948, 'epoch': 1.76}

 44%|████▍     | 1430/3252 [59:27<1:13:31,  2.42s/it]
 44%|████▍     | 1431/3252 [59:30<1:14:06,  2.44s/it]
 44%|████▍     | 1432/3252 [59:32<1:13:38,  2.43s/it]
 44%|████▍     | 1433/3252 [59:34<1:13:19,  2.42s/it]
 44%|████▍     | 1434/3252 [59:37<1:13:05,  2.41s/it]
 44%|████▍     | 1435/3252 [59:39<1:12:53,  2.41s/it]
 44%|████▍     | 1436/3252 [59:42<1:13:07,  2.42s/it]
 44%|████▍     | 1437/3252 [59:44<1:13:03,  2.42s/it]
 44%|████▍     | 1438/3252 [59:47<1:12:45,  2.41s/it]
 44%|████▍     | 1439/3252 [59:49<1:12:46,  2.41s/it]
 44%|████▍     | 1440/3252 [59:51<1:12:34,  2.40s/it]
                                                     
{'loss': 0.4551, 'grad_norm': 0.19769492745399475, 'learning_rate': 0.00012330576815828965, 'lm_loss': 0.009769360534846783, 'epoch': 1.77}

 44%|████▍     | 1440/3252 [59:52<1:12:34,  2.40s/it]
 44%|████▍     | 1441/3252 [59:54<1:14:56,  2.48s/it]
 44%|████▍     | 1442/3252 [59:56<1:14:20,  2.46s/it]
 44%|████▍     | 1443/3252 [59:59<1:13:41,  2.44s/it]
 44%|████▍     | 1444/3252 [1:00:01<1:13:13,  2.43s/it]
 44%|████▍     | 1445/3252 [1:00:04<1:12:46,  2.42s/it]
 44%|████▍     | 1446/3252 [1:00:06<1:13:22,  2.44s/it]
 44%|████▍     | 1447/3252 [1:00:08<1:12:56,  2.42s/it]
 45%|████▍     | 1448/3252 [1:00:11<1:15:34,  2.51s/it]
 45%|████▍     | 1449/3252 [1:00:14<1:14:31,  2.48s/it]
 45%|████▍     | 1450/3252 [1:00:16<1:13:42,  2.45s/it]
                                                       
{'loss': 0.4389, 'grad_norm': 0.3376274108886719, 'learning_rate': 0.00012233537459709683, 'lm_loss': 0.015752727165818214, 'epoch': 1.78}

 45%|████▍     | 1450/3252 [1:00:16<1:13:42,  2.45s/it]
 45%|████▍     | 1451/3252 [1:00:18<1:13:25,  2.45s/it]
 45%|████▍     | 1452/3252 [1:00:21<1:12:58,  2.43s/it]
 45%|████▍     | 1453/3252 [1:00:23<1:12:37,  2.42s/it]
 45%|████▍     | 1454/3252 [1:00:26<1:12:25,  2.42s/it]
 45%|████▍     | 1455/3252 [1:00:28<1:12:30,  2.42s/it]
 45%|████▍     | 1456/3252 [1:00:30<1:12:13,  2.41s/it]
 45%|████▍     | 1457/3252 [1:00:33<1:11:58,  2.41s/it]
 45%|████▍     | 1458/3252 [1:00:35<1:11:47,  2.40s/it]
 45%|████▍     | 1459/3252 [1:00:38<1:11:40,  2.40s/it]
 45%|████▍     | 1460/3252 [1:00:40<1:11:37,  2.40s/it]
                                                       
{'loss': 0.4355, 'grad_norm': 0.18028321862220764, 'learning_rate': 0.00012136276224195348, 'lm_loss': 0.5585598349571228, 'epoch': 1.8}

 45%|████▍     | 1460/3252 [1:00:40<1:11:37,  2.40s/it]
 45%|████▍     | 1461/3252 [1:00:43<1:12:37,  2.43s/it]
 45%|████▍     | 1462/3252 [1:00:45<1:12:10,  2.42s/it]
 45%|████▍     | 1463/3252 [1:00:47<1:12:06,  2.42s/it]
 45%|████▌     | 1464/3252 [1:00:50<1:12:01,  2.42s/it]
 45%|████▌     | 1465/3252 [1:00:52<1:13:20,  2.46s/it]
 45%|████▌     | 1466/3252 [1:00:55<1:12:40,  2.44s/it]
 45%|████▌     | 1467/3252 [1:00:57<1:12:14,  2.43s/it]
 45%|████▌     | 1468/3252 [1:00:59<1:11:54,  2.42s/it]
 45%|████▌     | 1469/3252 [1:01:02<1:11:44,  2.41s/it]
 45%|████▌     | 1470/3252 [1:01:04<1:11:35,  2.41s/it]
                                                       
{'loss': 0.4133, 'grad_norm': 0.17994898557662964, 'learning_rate': 0.00012038802771207684, 'lm_loss': 0.0027090609073638916, 'epoch': 1.81}

 45%|████▌     | 1470/3252 [1:01:04<1:11:35,  2.41s/it]
 45%|████▌     | 1471/3252 [1:01:07<1:11:57,  2.42s/it]
 45%|████▌     | 1472/3252 [1:01:09<1:11:41,  2.42s/it]
 45%|████▌     | 1473/3252 [1:01:12<1:11:32,  2.41s/it]
 45%|████▌     | 1474/3252 [1:01:14<1:11:23,  2.41s/it]
 45%|████▌     | 1475/3252 [1:01:16<1:11:16,  2.41s/it]
 45%|████▌     | 1476/3252 [1:01:19<1:11:02,  2.40s/it]
 45%|████▌     | 1477/3252 [1:01:21<1:11:00,  2.40s/it]
 45%|████▌     | 1478/3252 [1:01:24<1:10:49,  2.40s/it]
 45%|████▌     | 1479/3252 [1:01:26<1:10:47,  2.40s/it]
 46%|████▌     | 1480/3252 [1:01:28<1:11:00,  2.40s/it]
                                                       
{'loss': 0.426, 'grad_norm': 0.19005639851093292, 'learning_rate': 0.00011941126783750074, 'lm_loss': 0.0343930758535862, 'epoch': 1.82}

 46%|████▌     | 1480/3252 [1:01:28<1:11:00,  2.40s/it]
 46%|████▌     | 1481/3252 [1:01:31<1:11:01,  2.41s/it]
 46%|████▌     | 1482/3252 [1:01:33<1:11:42,  2.43s/it]
 46%|████▌     | 1483/3252 [1:01:36<1:11:37,  2.43s/it]
 46%|████▌     | 1484/3252 [1:01:38<1:11:18,  2.42s/it]
 46%|████▌     | 1485/3252 [1:01:40<1:11:05,  2.41s/it]
 46%|████▌     | 1486/3252 [1:01:43<1:10:49,  2.41s/it]
 46%|████▌     | 1487/3252 [1:01:45<1:10:36,  2.40s/it]
 46%|████▌     | 1488/3252 [1:01:48<1:10:36,  2.40s/it]
 46%|████▌     | 1489/3252 [1:01:50<1:10:29,  2.40s/it]
 46%|████▌     | 1490/3252 [1:01:52<1:10:24,  2.40s/it]
                                                       
{'loss': 0.4362, 'grad_norm': 0.3715980350971222, 'learning_rate': 0.0001184325796494566, 'lm_loss': 0.08115934580564499, 'epoch': 1.83}

 46%|████▌     | 1490/3252 [1:01:53<1:10:24,  2.40s/it]
 46%|████▌     | 1491/3252 [1:01:55<1:12:40,  2.48s/it]
 46%|████▌     | 1492/3252 [1:01:57<1:11:57,  2.45s/it]
 46%|████▌     | 1493/3252 [1:02:00<1:11:24,  2.44s/it]
 46%|████▌     | 1494/3252 [1:02:02<1:10:56,  2.42s/it]
 46%|████▌     | 1495/3252 [1:02:05<1:10:45,  2.42s/it]
 46%|████▌     | 1496/3252 [1:02:07<1:10:31,  2.41s/it]
 46%|████▌     | 1497/3252 [1:02:09<1:10:16,  2.40s/it]
 46%|████▌     | 1498/3252 [1:02:12<1:10:12,  2.40s/it]
 46%|████▌     | 1499/3252 [1:02:14<1:10:03,  2.40s/it]
 46%|████▌     | 1500/3252 [1:02:17<1:11:27,  2.45s/it]
                                                       
{'loss': 0.4147, 'grad_norm': 0.3025594651699066, 'learning_rate': 0.00011745206037073426, 'lm_loss': 0.00614673737436533, 'epoch': 1.85}

 46%|████▌     | 1500/3252 [1:02:17<1:11:27,  2.45s/it]
 46%|████▌     | 1501/3252 [1:02:19<1:11:13,  2.44s/it]
 46%|████▌     | 1502/3252 [1:02:22<1:11:54,  2.47s/it]
 46%|████▌     | 1503/3252 [1:02:24<1:11:16,  2.44s/it]
 46%|████▌     | 1504/3252 [1:02:27<1:10:41,  2.43s/it]
 46%|████▋     | 1505/3252 [1:02:29<1:10:23,  2.42s/it]
 46%|████▋     | 1506/3252 [1:02:31<1:10:09,  2.41s/it]
 46%|████▋     | 1507/3252 [1:02:34<1:10:01,  2.41s/it]
 46%|████▋     | 1508/3252 [1:02:36<1:09:47,  2.40s/it]
 46%|████▋     | 1509/3252 [1:02:39<1:09:42,  2.40s/it]
 46%|████▋     | 1510/3252 [1:02:41<1:10:11,  2.42s/it]
                                                       
{'loss': 0.4325, 'grad_norm': 0.2847006618976593, 'learning_rate': 0.00011646980740602393, 'lm_loss': 0.6942877769470215, 'epoch': 1.86}

 46%|████▋     | 1510/3252 [1:02:41<1:10:11,  2.42s/it]
 46%|████▋     | 1511/3252 [1:02:43<1:10:22,  2.43s/it]
 46%|████▋     | 1512/3252 [1:02:46<1:10:05,  2.42s/it]
 47%|████▋     | 1513/3252 [1:02:48<1:09:50,  2.41s/it]
 47%|████▋     | 1514/3252 [1:02:51<1:09:40,  2.41s/it]
 47%|████▋     | 1515/3252 [1:02:53<1:09:32,  2.40s/it]
 47%|████▋     | 1516/3252 [1:02:55<1:09:35,  2.41s/it]
 47%|████▋     | 1517/3252 [1:02:58<1:09:24,  2.40s/it]
 47%|████▋     | 1518/3252 [1:03:00<1:09:14,  2.40s/it]
 47%|████▋     | 1519/3252 [1:03:03<1:09:08,  2.39s/it]
 47%|████▋     | 1520/3252 [1:03:05<1:09:57,  2.42s/it]
                                                       
{'loss': 0.4263, 'grad_norm': 0.19931887090206146, 'learning_rate': 0.00011548591833224014, 'lm_loss': 0.7166070938110352, 'epoch': 1.87}

 47%|████▋     | 1520/3252 [1:03:05<1:09:57,  2.42s/it]
 47%|████▋     | 1521/3252 [1:03:08<1:10:14,  2.43s/it]
 47%|████▋     | 1522/3252 [1:03:10<1:12:12,  2.50s/it]
 47%|████▋     | 1523/3252 [1:03:13<1:11:12,  2.47s/it]
 47%|████▋     | 1524/3252 [1:03:15<1:10:29,  2.45s/it]
 47%|████▋     | 1525/3252 [1:03:17<1:10:01,  2.43s/it]
 47%|████▋     | 1526/3252 [1:03:20<1:09:42,  2.42s/it]
 47%|████▋     | 1527/3252 [1:03:22<1:09:22,  2.41s/it]
 47%|████▋     | 1528/3252 [1:03:25<1:09:14,  2.41s/it]
 47%|████▋     | 1529/3252 [1:03:27<1:09:05,  2.41s/it]
 47%|████▋     | 1530/3252 [1:03:29<1:08:57,  2.40s/it]
                                                       
{'loss': 0.4061, 'grad_norm': 0.22854746878147125, 'learning_rate': 0.00011450049088882815, 'lm_loss': 0.6868299841880798, 'epoch': 1.88}

 47%|████▋     | 1530/3252 [1:03:30<1:08:57,  2.40s/it]
 47%|████▋     | 1531/3252 [1:03:32<1:09:50,  2.43s/it]
 47%|████▋     | 1532/3252 [1:03:34<1:09:30,  2.42s/it]
 47%|████▋     | 1533/3252 [1:03:37<1:09:13,  2.42s/it]
 47%|████▋     | 1534/3252 [1:03:39<1:09:06,  2.41s/it]
 47%|████▋     | 1535/3252 [1:03:41<1:08:51,  2.41s/it]
 47%|████▋     | 1536/3252 [1:03:44<1:08:36,  2.40s/it]
 47%|████▋     | 1537/3252 [1:03:46<1:09:11,  2.42s/it]
 47%|████▋     | 1538/3252 [1:03:49<1:08:59,  2.42s/it]
 47%|████▋     | 1539/3252 [1:03:51<1:09:49,  2.45s/it]
 47%|████▋     | 1540/3252 [1:03:54<1:09:17,  2.43s/it]
                                                       
{'loss': 0.418, 'grad_norm': 0.1819072812795639, 'learning_rate': 0.00011351362296805485, 'lm_loss': 0.24727904796600342, 'epoch': 1.89}

 47%|████▋     | 1540/3252 [1:03:54<1:09:17,  2.43s/it]
 47%|████▋     | 1541/3252 [1:03:56<1:09:21,  2.43s/it]
 47%|████▋     | 1542/3252 [1:03:58<1:09:06,  2.43s/it]
 47%|████▋     | 1543/3252 [1:04:01<1:08:45,  2.41s/it]
 47%|████▋     | 1544/3252 [1:04:03<1:08:29,  2.41s/it]
 48%|████▊     | 1545/3252 [1:04:06<1:08:34,  2.41s/it]
 48%|████▊     | 1546/3252 [1:04:08<1:08:29,  2.41s/it]
 48%|████▊     | 1547/3252 [1:04:10<1:08:16,  2.40s/it]
 48%|████▊     | 1548/3252 [1:04:13<1:08:13,  2.40s/it]
 48%|████▊     | 1549/3252 [1:04:15<1:08:12,  2.40s/it]
 48%|████▊     | 1550/3252 [1:04:18<1:08:08,  2.40s/it]
                                                       
{'loss': 0.4136, 'grad_norm': 0.24497298896312714, 'learning_rate': 0.00011252541260528398, 'lm_loss': 0.3714967668056488, 'epoch': 1.91}

 48%|████▊     | 1550/3252 [1:04:18<1:08:08,  2.40s/it]
 48%|████▊     | 1551/3252 [1:04:20<1:08:56,  2.43s/it]
 48%|████▊     | 1552/3252 [1:04:23<1:08:38,  2.42s/it]
 48%|████▊     | 1553/3252 [1:04:25<1:08:17,  2.41s/it]
 48%|████▊     | 1554/3252 [1:04:27<1:08:20,  2.41s/it]
 48%|████▊     | 1555/3252 [1:04:30<1:08:18,  2.42s/it]
 48%|████▊     | 1556/3252 [1:04:32<1:09:26,  2.46s/it]
 48%|████▊     | 1557/3252 [1:04:35<1:09:03,  2.44s/it]
 48%|████▊     | 1558/3252 [1:04:37<1:08:33,  2.43s/it]
 48%|████▊     | 1559/3252 [1:04:40<1:08:29,  2.43s/it]
 48%|████▊     | 1560/3252 [1:04:42<1:08:14,  2.42s/it]
                                                       
{'loss': 0.41, 'grad_norm': 0.19017598032951355, 'learning_rate': 0.0001115359579692373, 'lm_loss': 0.5813034176826477, 'epoch': 1.92}

 48%|████▊     | 1560/3252 [1:04:42<1:08:14,  2.42s/it]
 48%|████▊     | 1561/3252 [1:04:44<1:08:10,  2.42s/it]
 48%|████▊     | 1562/3252 [1:04:47<1:07:56,  2.41s/it]
 48%|████▊     | 1563/3252 [1:04:49<1:07:44,  2.41s/it]
 48%|████▊     | 1564/3252 [1:04:52<1:07:35,  2.40s/it]
 48%|████▊     | 1565/3252 [1:04:54<1:07:28,  2.40s/it]
 48%|████▊     | 1566/3252 [1:04:56<1:07:20,  2.40s/it]
 48%|████▊     | 1567/3252 [1:04:59<1:07:15,  2.39s/it]
 48%|████▊     | 1568/3252 [1:05:01<1:07:34,  2.41s/it]
 48%|████▊     | 1569/3252 [1:05:04<1:07:21,  2.40s/it]
 48%|████▊     | 1570/3252 [1:05:06<1:07:14,  2.40s/it]
                                                       
{'loss': 0.4178, 'grad_norm': 0.2428624927997589, 'learning_rate': 0.00011054535735224256, 'lm_loss': 0.6147779822349548, 'epoch': 1.93}

 48%|████▊     | 1570/3252 [1:05:06<1:07:14,  2.40s/it]
 48%|████▊     | 1571/3252 [1:05:08<1:07:23,  2.41s/it]
 48%|████▊     | 1572/3252 [1:05:11<1:07:20,  2.40s/it]
 48%|████▊     | 1573/3252 [1:05:13<1:07:13,  2.40s/it]
 48%|████▊     | 1574/3252 [1:05:16<1:08:06,  2.44s/it]
 48%|████▊     | 1575/3252 [1:05:18<1:07:43,  2.42s/it]
 48%|████▊     | 1576/3252 [1:05:21<1:08:42,  2.46s/it]
 48%|████▊     | 1577/3252 [1:05:23<1:08:09,  2.44s/it]
 49%|████▊     | 1578/3252 [1:05:25<1:07:42,  2.43s/it]
 49%|████▊     | 1579/3252 [1:05:28<1:07:27,  2.42s/it]
 49%|████▊     | 1580/3252 [1:05:30<1:07:05,  2.41s/it]
                                                       
{'loss': 0.4213, 'grad_norm': 0.2038167417049408, 'learning_rate': 0.00010955370916046918, 'lm_loss': 0.2880806624889374, 'epoch': 1.94}

 49%|████▊     | 1580/3252 [1:05:30<1:07:05,  2.41s/it]
 49%|████▊     | 1581/3252 [1:05:33<1:07:14,  2.41s/it]
 49%|████▊     | 1582/3252 [1:05:35<1:07:21,  2.42s/it]
 49%|████▊     | 1583/3252 [1:05:37<1:07:05,  2.41s/it]
 49%|████▊     | 1584/3252 [1:05:40<1:07:12,  2.42s/it]
 49%|████▊     | 1585/3252 [1:05:42<1:07:06,  2.42s/it]
 49%|████▉     | 1586/3252 [1:05:45<1:06:51,  2.41s/it]
 49%|████▉     | 1587/3252 [1:05:47<1:06:43,  2.40s/it]
 49%|████▉     | 1588/3252 [1:05:50<1:06:36,  2.40s/it]
 49%|████▉     | 1589/3252 [1:05:52<1:06:28,  2.40s/it]
 49%|████▉     | 1590/3252 [1:05:54<1:06:21,  2.40s/it]
                                                       
{'loss': 0.4144, 'grad_norm': 0.16891565918922424, 'learning_rate': 0.00010856111190415262, 'lm_loss': 0.023549430072307587, 'epoch': 1.96}

 49%|████▉     | 1590/3252 [1:05:54<1:06:21,  2.40s/it]
 49%|████▉     | 1591/3252 [1:05:57<1:07:03,  2.42s/it]
 49%|████▉     | 1592/3252 [1:05:59<1:06:48,  2.41s/it]
 49%|████▉     | 1593/3252 [1:06:02<1:06:32,  2.41s/it]
 49%|████▉     | 1594/3252 [1:06:04<1:07:30,  2.44s/it]
 49%|████▉     | 1595/3252 [1:06:06<1:07:05,  2.43s/it]
 49%|████▉     | 1596/3252 [1:06:09<1:09:17,  2.51s/it]
 49%|████▉     | 1597/3252 [1:06:12<1:08:14,  2.47s/it]
 49%|████▉     | 1598/3252 [1:06:14<1:07:33,  2.45s/it]
 49%|████▉     | 1599/3252 [1:06:16<1:07:04,  2.43s/it]
 49%|████▉     | 1600/3252 [1:06:19<1:06:39,  2.42s/it]
                                                       
{'loss': 0.4179, 'grad_norm': 0.22166217863559723, 'learning_rate': 0.0001075676641878081, 'lm_loss': 0.9289381504058838, 'epoch': 1.97}

 49%|████▉     | 1600/3252 [1:06:19<1:06:39,  2.42s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.20it/s]
                                                       

                                             
{'eval_sthsthv2/narration_val_loss': 0.818312406539917, 'eval_sthsthv2/narration_val_runtime': 3.623, 'eval_sthsthv2/narration_val_samples_per_second': 184.376, 'eval_sthsthv2/narration_val_steps_per_second': 0.552, 'epoch': 1.97}

 49%|████▉     | 1600/3252 [1:06:23<1:06:39,  2.42s/it]

100%|██████████| 2/2 [00:00<00:00,  2.20it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.69it/s]
                                                       

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6278453469276428, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.1705, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 186.721, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.631, 'epoch': 1.97}

 49%|████▉     | 1600/3252 [1:06:26<1:06:39,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  1.69it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.12it/s]
                                                       

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.5878801345825195, 'eval_holoassist/dialog-klg-sum_val_runtime': 3.0326, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 233.791, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.659, 'epoch': 1.97}

 49%|████▉     | 1600/3252 [1:06:29<1:06:39,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  2.12it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                       

                                     
{'eval_ego4d/narration_val_loss': 0.5244945883750916, 'eval_ego4d/narration_val_runtime': 2.1671, 'eval_ego4d/narration_val_samples_per_second': 102.902, 'eval_ego4d/narration_val_steps_per_second': 0.461, 'epoch': 1.97}

 49%|████▉     | 1600/3252 [1:06:31<1:06:39,  2.42s/it]

100%|██████████| 1/1 [00:00<00:00,  3.87it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.73it/s]
                                                       

                                             
{'eval_ego4d/summary_val_loss': 1.1260652542114258, 'eval_ego4d/summary_val_runtime': 2.8111, 'eval_ego4d/summary_val_samples_per_second': 200.63, 'eval_ego4d/summary_val_steps_per_second': 0.711, 'epoch': 1.97}

 49%|████▉     | 1600/3252 [1:06:34<1:06:39,  2.42s/it]

100%|██████████| 2/2 [00:01<00:00,  1.73it/s]

                                             
 49%|████▉     | 1601/3252 [1:06:36<3:10:35,  6.93s/it]
 49%|████▉     | 1602/3252 [1:06:39<2:33:05,  5.57s/it]
 49%|████▉     | 1603/3252 [1:06:41<2:06:49,  4.61s/it]
 49%|████▉     | 1604/3252 [1:06:43<1:48:26,  3.95s/it]
 49%|████▉     | 1605/3252 [1:06:46<1:35:35,  3.48s/it]
 49%|████▉     | 1606/3252 [1:06:48<1:26:45,  3.16s/it]
 49%|████▉     | 1607/3252 [1:06:51<1:20:25,  2.93s/it]
 49%|████▉     | 1608/3252 [1:06:53<1:16:01,  2.77s/it]
 49%|████▉     | 1609/3252 [1:06:55<1:12:50,  2.66s/it]
 50%|████▉     | 1610/3252 [1:06:58<1:10:41,  2.58s/it]
                                                       
{'loss': 0.3974, 'grad_norm': 0.20393383502960205, 'learning_rate': 0.00010657346470043575, 'lm_loss': 0.5326628684997559, 'epoch': 1.98}

 50%|████▉     | 1610/3252 [1:06:58<1:10:41,  2.58s/it]
 50%|████▉     | 1611/3252 [1:07:00<1:10:25,  2.57s/it]
 50%|████▉     | 1612/3252 [1:07:03<1:09:26,  2.54s/it]
 50%|████▉     | 1613/3252 [1:07:05<1:08:14,  2.50s/it]
 50%|████▉     | 1614/3252 [1:07:08<1:08:48,  2.52s/it]
 50%|████▉     | 1615/3252 [1:07:10<1:07:43,  2.48s/it]
 50%|████▉     | 1616/3252 [1:07:13<1:07:02,  2.46s/it]
 50%|████▉     | 1617/3252 [1:07:15<1:06:26,  2.44s/it]
 50%|████▉     | 1618/3252 [1:07:17<1:06:01,  2.42s/it]
 50%|████▉     | 1619/3252 [1:07:20<1:05:44,  2.42s/it]
 50%|████▉     | 1620/3252 [1:07:22<1:05:25,  2.41s/it]
                                                       
{'loss': 0.4252, 'grad_norm': 0.31501978635787964, 'learning_rate': 0.00010557861220571625, 'lm_loss': 0.466970294713974, 'epoch': 1.99}

 50%|████▉     | 1620/3252 [1:07:22<1:05:25,  2.41s/it]
 50%|████▉     | 1621/3252 [1:07:25<1:06:23,  2.44s/it]
 50%|████▉     | 1622/3252 [1:07:27<1:05:53,  2.43s/it]
 50%|████▉     | 1623/3252 [1:07:29<1:05:32,  2.41s/it]
 50%|████▉     | 1624/3252 [1:07:32<1:05:13,  2.40s/it]
 50%|████▉     | 1625/3252 [1:07:34<1:05:03,  2.40s/it]
 50%|█████     | 1626/3252 [1:07:37<1:06:31,  2.45s/it]
 50%|█████     | 1627/3252 [1:07:43<1:34:14,  3.48s/it]
 50%|█████     | 1628/3252 [1:07:45<1:25:32,  3.16s/it]
 50%|█████     | 1629/3252 [1:07:47<1:19:12,  2.93s/it]
 50%|█████     | 1630/3252 [1:07:50<1:15:10,  2.78s/it]
                                                       
{'loss': 0.4133, 'grad_norm': 0.2253994345664978, 'learning_rate': 0.00010458320553220029, 'lm_loss': 0.0017214288236573339, 'epoch': 2.0}

 50%|█████     | 1630/3252 [1:07:50<1:15:10,  2.78s/it]
 50%|█████     | 1631/3252 [1:07:52<1:12:18,  2.68s/it]
 50%|█████     | 1632/3252 [1:07:55<1:11:33,  2.65s/it]
 50%|█████     | 1633/3252 [1:07:57<1:09:25,  2.57s/it]
 50%|█████     | 1634/3252 [1:08:00<1:07:55,  2.52s/it]
 50%|█████     | 1635/3252 [1:08:02<1:06:51,  2.48s/it]
 50%|█████     | 1636/3252 [1:08:04<1:06:11,  2.46s/it]
 50%|█████     | 1637/3252 [1:08:07<1:05:42,  2.44s/it]
 50%|█████     | 1638/3252 [1:08:09<1:05:18,  2.43s/it]
 50%|█████     | 1639/3252 [1:08:12<1:04:59,  2.42s/it]
 50%|█████     | 1640/3252 [1:08:14<1:04:46,  2.41s/it]
                                                       
{'loss': 0.3871, 'grad_norm': 0.22563104331493378, 'learning_rate': 0.00010358734356349036, 'lm_loss': 0.3027110695838928, 'epoch': 2.02}

 50%|█████     | 1640/3252 [1:08:14<1:04:46,  2.41s/it]
 50%|█████     | 1641/3252 [1:08:17<1:04:54,  2.42s/it]
 50%|█████     | 1642/3252 [1:08:19<1:04:46,  2.41s/it]
 51%|█████     | 1643/3252 [1:08:21<1:04:35,  2.41s/it]
 51%|█████     | 1644/3252 [1:08:24<1:04:25,  2.40s/it]
 51%|█████     | 1645/3252 [1:08:26<1:04:20,  2.40s/it]
 51%|█████     | 1646/3252 [1:08:29<1:04:16,  2.40s/it]
 51%|█████     | 1647/3252 [1:08:31<1:04:42,  2.42s/it]
 51%|█████     | 1648/3252 [1:08:33<1:04:28,  2.41s/it]
 51%|█████     | 1649/3252 [1:08:36<1:05:55,  2.47s/it]
 51%|█████     | 1650/3252 [1:08:38<1:05:18,  2.45s/it]
                                                       
{'loss': 0.3874, 'grad_norm': 0.23233307898044586, 'learning_rate': 0.00010259112522841807, 'lm_loss': 1.0535151958465576, 'epoch': 2.03}

 51%|█████     | 1650/3252 [1:08:38<1:05:18,  2.45s/it]
 51%|█████     | 1651/3252 [1:08:41<1:05:39,  2.46s/it]
 51%|█████     | 1652/3252 [1:08:43<1:05:03,  2.44s/it]
 51%|█████     | 1653/3252 [1:08:46<1:04:42,  2.43s/it]
 51%|█████     | 1654/3252 [1:08:48<1:04:22,  2.42s/it]
 51%|█████     | 1655/3252 [1:08:50<1:04:09,  2.41s/it]
 51%|█████     | 1656/3252 [1:08:53<1:03:59,  2.41s/it]
 51%|█████     | 1657/3252 [1:08:55<1:03:50,  2.40s/it]
 51%|█████     | 1658/3252 [1:08:58<1:03:53,  2.40s/it]
 51%|█████     | 1659/3252 [1:09:00<1:04:13,  2.42s/it]
 51%|█████     | 1660/3252 [1:09:02<1:04:05,  2.42s/it]
                                                       
{'loss': 0.3863, 'grad_norm': 0.21239104866981506, 'learning_rate': 0.00010159464949121641, 'lm_loss': 0.006788274738937616, 'epoch': 2.04}

 51%|█████     | 1660/3252 [1:09:03<1:04:05,  2.42s/it]
 51%|█████     | 1661/3252 [1:09:05<1:05:31,  2.47s/it]
 51%|█████     | 1662/3252 [1:09:07<1:04:54,  2.45s/it]
 51%|█████     | 1663/3252 [1:09:10<1:04:36,  2.44s/it]
 51%|█████     | 1664/3252 [1:09:12<1:04:17,  2.43s/it]
 51%|█████     | 1665/3252 [1:09:15<1:04:03,  2.42s/it]
 51%|█████     | 1666/3252 [1:09:17<1:03:47,  2.41s/it]
 51%|█████▏    | 1667/3252 [1:09:20<1:04:14,  2.43s/it]
 51%|█████▏    | 1668/3252 [1:09:22<1:03:50,  2.42s/it]
 51%|█████▏    | 1669/3252 [1:09:24<1:03:33,  2.41s/it]
 51%|█████▏    | 1670/3252 [1:09:27<1:04:55,  2.46s/it]
                                                       
{'loss': 0.3813, 'grad_norm': 0.19213765859603882, 'learning_rate': 0.00010059801534168867, 'lm_loss': 0.014080728404223919, 'epoch': 2.05}

 51%|█████▏    | 1670/3252 [1:09:27<1:04:55,  2.46s/it]
 51%|█████▏    | 1671/3252 [1:09:29<1:04:51,  2.46s/it]
 51%|█████▏    | 1672/3252 [1:09:32<1:04:17,  2.44s/it]
 51%|█████▏    | 1673/3252 [1:09:34<1:03:54,  2.43s/it]
 51%|█████▏    | 1674/3252 [1:09:37<1:03:33,  2.42s/it]
 52%|█████▏    | 1675/3252 [1:09:39<1:03:35,  2.42s/it]
 52%|█████▏    | 1676/3252 [1:09:41<1:03:22,  2.41s/it]
 52%|█████▏    | 1677/3252 [1:09:44<1:03:13,  2.41s/it]
 52%|█████▏    | 1678/3252 [1:09:46<1:03:08,  2.41s/it]
 52%|█████▏    | 1679/3252 [1:09:49<1:03:04,  2.41s/it]
 52%|█████▏    | 1680/3252 [1:09:51<1:02:58,  2.40s/it]
                                                       
{'loss': 0.367, 'grad_norm': 0.1960364431142807, 'learning_rate': 9.960132178537471e-05, 'lm_loss': 0.46636614203453064, 'epoch': 2.07}

 52%|█████▏    | 1680/3252 [1:09:51<1:02:58,  2.40s/it]
 52%|█████▏    | 1681/3252 [1:09:54<1:03:43,  2.43s/it]
 52%|█████▏    | 1682/3252 [1:09:56<1:03:18,  2.42s/it]
 52%|█████▏    | 1683/3252 [1:09:58<1:03:07,  2.41s/it]
 52%|█████▏    | 1684/3252 [1:10:01<1:02:56,  2.41s/it]
 52%|█████▏    | 1685/3252 [1:10:03<1:02:46,  2.40s/it]
 52%|█████▏    | 1686/3252 [1:10:05<1:02:44,  2.40s/it]
 52%|█████▏    | 1687/3252 [1:10:08<1:03:54,  2.45s/it]
 52%|█████▏    | 1688/3252 [1:10:11<1:05:46,  2.52s/it]
 52%|█████▏    | 1689/3252 [1:10:13<1:04:46,  2.49s/it]
 52%|█████▏    | 1690/3252 [1:10:16<1:04:06,  2.46s/it]
                                                       
{'loss': 0.3929, 'grad_norm': 0.21494226157665253, 'learning_rate': 9.860466783371604e-05, 'lm_loss': 0.5966855883598328, 'epoch': 2.08}

 52%|█████▏    | 1690/3252 [1:10:16<1:04:06,  2.46s/it]
 52%|█████▏    | 1691/3252 [1:10:18<1:04:53,  2.49s/it]
 52%|█████▏    | 1692/3252 [1:10:21<1:04:07,  2.47s/it]
 52%|█████▏    | 1693/3252 [1:10:23<1:03:33,  2.45s/it]
 52%|█████▏    | 1694/3252 [1:10:25<1:03:09,  2.43s/it]
 52%|█████▏    | 1695/3252 [1:10:28<1:02:52,  2.42s/it]
 52%|█████▏    | 1696/3252 [1:10:30<1:02:36,  2.41s/it]
 52%|█████▏    | 1697/3252 [1:10:33<1:02:26,  2.41s/it]
 52%|█████▏    | 1698/3252 [1:10:35<1:02:27,  2.41s/it]
 52%|█████▏    | 1699/3252 [1:10:37<1:02:15,  2.41s/it]
 52%|█████▏    | 1700/3252 [1:10:40<1:02:11,  2.40s/it]
                                                       
{'loss': 0.3653, 'grad_norm': 0.3561396896839142, 'learning_rate': 9.760815249421973e-05, 'lm_loss': 0.34834641218185425, 'epoch': 2.09}

 52%|█████▏    | 1700/3252 [1:10:40<1:02:11,  2.40s/it]
 52%|█████▏    | 1701/3252 [1:10:42<1:02:52,  2.43s/it]
 52%|█████▏    | 1702/3252 [1:10:45<1:02:34,  2.42s/it]
 52%|█████▏    | 1703/3252 [1:10:47<1:02:21,  2.42s/it]
 52%|█████▏    | 1704/3252 [1:10:49<1:02:08,  2.41s/it]
 52%|█████▏    | 1705/3252 [1:10:52<1:02:00,  2.40s/it]
 52%|█████▏    | 1706/3252 [1:10:54<1:02:49,  2.44s/it]
 52%|█████▏    | 1707/3252 [1:10:57<1:02:34,  2.43s/it]
 53%|█████▎    | 1708/3252 [1:10:59<1:02:16,  2.42s/it]
 53%|█████▎    | 1709/3252 [1:11:02<1:02:02,  2.41s/it]
 53%|█████▎    | 1710/3252 [1:11:04<1:01:51,  2.41s/it]
                                                       
{'loss': 0.383, 'grad_norm': 0.23635420203208923, 'learning_rate': 9.661187476062308e-05, 'lm_loss': 0.3170147240161896, 'epoch': 2.1}

 53%|█████▎    | 1710/3252 [1:11:04<1:01:51,  2.41s/it]
 53%|█████▎    | 1711/3252 [1:11:06<1:02:40,  2.44s/it]
 53%|█████▎    | 1712/3252 [1:11:09<1:02:16,  2.43s/it]
 53%|█████▎    | 1713/3252 [1:11:11<1:01:56,  2.41s/it]
 53%|█████▎    | 1714/3252 [1:11:14<1:01:45,  2.41s/it]
 53%|█████▎    | 1715/3252 [1:11:16<1:01:39,  2.41s/it]
 53%|█████▎    | 1716/3252 [1:11:18<1:01:32,  2.40s/it]
 53%|█████▎    | 1717/3252 [1:11:21<1:01:22,  2.40s/it]
 53%|█████▎    | 1718/3252 [1:11:23<1:01:26,  2.40s/it]
 53%|█████▎    | 1719/3252 [1:11:26<1:01:46,  2.42s/it]
 53%|█████▎    | 1720/3252 [1:11:28<1:01:34,  2.41s/it]
                                                       
{'loss': 0.3703, 'grad_norm': 0.20143522322177887, 'learning_rate': 9.561593360305977e-05, 'lm_loss': 0.0022752571385353804, 'epoch': 2.12}

 53%|█████▎    | 1720/3252 [1:11:28<1:01:34,  2.41s/it]
 53%|█████▎    | 1721/3252 [1:11:30<1:01:33,  2.41s/it]
 53%|█████▎    | 1722/3252 [1:11:33<1:02:31,  2.45s/it]
 53%|█████▎    | 1723/3252 [1:11:36<1:03:32,  2.49s/it]
 53%|█████▎    | 1724/3252 [1:11:38<1:02:44,  2.46s/it]
 53%|█████▎    | 1725/3252 [1:11:40<1:02:10,  2.44s/it]
 53%|█████▎    | 1726/3252 [1:11:43<1:01:52,  2.43s/it]
 53%|█████▎    | 1727/3252 [1:11:45<1:01:34,  2.42s/it]
 53%|█████▎    | 1728/3252 [1:11:48<1:01:23,  2.42s/it]
 53%|█████▎    | 1729/3252 [1:11:50<1:01:20,  2.42s/it]
 53%|█████▎    | 1730/3252 [1:11:52<1:01:19,  2.42s/it]
                                                       
{'loss': 0.375, 'grad_norm': 0.1944246143102646, 'learning_rate': 9.462042795822799e-05, 'lm_loss': 0.5088287591934204, 'epoch': 2.13}

 53%|█████▎    | 1730/3252 [1:11:53<1:01:19,  2.42s/it]
 53%|█████▎    | 1731/3252 [1:11:55<1:01:18,  2.42s/it]
 53%|█████▎    | 1732/3252 [1:11:57<1:00:59,  2.41s/it]
 53%|█████▎    | 1733/3252 [1:12:00<1:00:54,  2.41s/it]
 53%|█████▎    | 1734/3252 [1:12:02<1:00:47,  2.40s/it]
 53%|█████▎    | 1735/3252 [1:12:04<1:00:51,  2.41s/it]
 53%|█████▎    | 1736/3252 [1:12:07<1:00:41,  2.40s/it]
 53%|█████▎    | 1737/3252 [1:12:09<1:00:36,  2.40s/it]
 53%|█████▎    | 1738/3252 [1:12:12<1:00:38,  2.40s/it]
 53%|█████▎    | 1739/3252 [1:12:14<1:00:34,  2.40s/it]
 54%|█████▎    | 1740/3252 [1:12:16<1:00:33,  2.40s/it]
                                                       
{'loss': 0.3745, 'grad_norm': 0.21747708320617676, 'learning_rate': 9.362545671956214e-05, 'lm_loss': 0.012448767200112343, 'epoch': 2.14}

 54%|█████▎    | 1740/3252 [1:12:17<1:00:33,  2.40s/it]
 54%|█████▎    | 1741/3252 [1:12:19<1:01:30,  2.44s/it]
 54%|█████▎    | 1742/3252 [1:12:21<1:01:04,  2.43s/it]
 54%|█████▎    | 1743/3252 [1:12:24<1:00:49,  2.42s/it]
 54%|█████▎    | 1744/3252 [1:12:26<1:02:39,  2.49s/it]
 54%|█████▎    | 1745/3252 [1:12:29<1:01:48,  2.46s/it]
 54%|█████▎    | 1746/3252 [1:12:31<1:01:19,  2.44s/it]
 54%|█████▎    | 1747/3252 [1:12:34<1:00:52,  2.43s/it]
 54%|█████▍    | 1748/3252 [1:12:36<1:00:37,  2.42s/it]
 54%|█████▍    | 1749/3252 [1:12:38<1:00:22,  2.41s/it]
 54%|█████▍    | 1750/3252 [1:12:41<1:00:14,  2.41s/it]
                                                       
{'loss': 0.3766, 'grad_norm': 0.47089165449142456, 'learning_rate': 9.263111872740866e-05, 'lm_loss': 0.09503118693828583, 'epoch': 2.15}

 54%|█████▍    | 1750/3252 [1:12:41<1:00:14,  2.41s/it]
 54%|█████▍    | 1751/3252 [1:12:43<1:00:18,  2.41s/it]
 54%|█████▍    | 1752/3252 [1:12:46<1:00:06,  2.40s/it]
 54%|█████▍    | 1753/3252 [1:12:48<1:00:01,  2.40s/it]
 54%|█████▍    | 1754/3252 [1:12:50<1:00:03,  2.41s/it]
 54%|█████▍    | 1755/3252 [1:12:53<1:00:01,  2.41s/it]
 54%|█████▍    | 1756/3252 [1:12:55<59:52,  2.40s/it]  
 54%|█████▍    | 1757/3252 [1:12:58<59:51,  2.40s/it]
 54%|█████▍    | 1758/3252 [1:13:00<59:56,  2.41s/it]
 54%|█████▍    | 1759/3252 [1:13:02<59:44,  2.40s/it]
 54%|█████▍    | 1760/3252 [1:13:05<59:39,  2.40s/it]
                                                     
{'loss': 0.3727, 'grad_norm': 0.19560199975967407, 'learning_rate': 9.163751275920762e-05, 'lm_loss': 0.001456643221899867, 'epoch': 2.16}

 54%|█████▍    | 1760/3252 [1:13:05<59:39,  2.40s/it]
 54%|█████▍    | 1761/3252 [1:13:07<1:00:50,  2.45s/it]
 54%|█████▍    | 1762/3252 [1:13:10<1:02:15,  2.51s/it]
 54%|█████▍    | 1763/3252 [1:13:13<1:02:12,  2.51s/it]
 54%|█████▍    | 1764/3252 [1:13:15<1:01:15,  2.47s/it]
 54%|█████▍    | 1765/3252 [1:13:17<1:00:38,  2.45s/it]
 54%|█████▍    | 1766/3252 [1:13:20<1:00:11,  2.43s/it]
 54%|█████▍    | 1767/3252 [1:13:22<59:53,  2.42s/it]  
 54%|█████▍    | 1768/3252 [1:13:25<59:40,  2.41s/it]
 54%|█████▍    | 1769/3252 [1:13:27<59:28,  2.41s/it]
 54%|█████▍    | 1770/3252 [1:13:29<59:24,  2.40s/it]
                                                     
{'loss': 0.3891, 'grad_norm': 0.20559324324131012, 'learning_rate': 9.064473751967957e-05, 'lm_loss': 0.41371479630470276, 'epoch': 2.18}

 54%|█████▍    | 1770/3252 [1:13:30<59:24,  2.40s/it]
 54%|█████▍    | 1771/3252 [1:13:32<1:01:32,  2.49s/it]
 54%|█████▍    | 1772/3252 [1:13:34<1:00:46,  2.46s/it]
 55%|█████▍    | 1773/3252 [1:13:37<1:00:14,  2.44s/it]
 55%|█████▍    | 1774/3252 [1:13:39<59:49,  2.43s/it]  
 55%|█████▍    | 1775/3252 [1:13:42<59:32,  2.42s/it]
 55%|█████▍    | 1776/3252 [1:13:44<59:18,  2.41s/it]
 55%|█████▍    | 1777/3252 [1:13:46<59:14,  2.41s/it]
 55%|█████▍    | 1778/3252 [1:13:49<59:09,  2.41s/it]
 55%|█████▍    | 1779/3252 [1:13:51<58:57,  2.40s/it]
 55%|█████▍    | 1780/3252 [1:13:54<59:41,  2.43s/it]
                                                     
{'loss': 0.3752, 'grad_norm': 0.20937596261501312, 'learning_rate': 8.965289163102078e-05, 'lm_loss': 0.0030694762244820595, 'epoch': 2.19}

 55%|█████▍    | 1780/3252 [1:13:54<59:41,  2.43s/it]
 55%|█████▍    | 1781/3252 [1:13:56<1:00:19,  2.46s/it]
 55%|█████▍    | 1782/3252 [1:13:59<59:52,  2.44s/it]  
 55%|█████▍    | 1783/3252 [1:14:01<59:30,  2.43s/it]
 55%|█████▍    | 1784/3252 [1:14:03<59:14,  2.42s/it]
 55%|█████▍    | 1785/3252 [1:14:06<59:03,  2.42s/it]
 55%|█████▍    | 1786/3252 [1:14:08<58:56,  2.41s/it]
 55%|█████▍    | 1787/3252 [1:14:11<58:48,  2.41s/it]
 55%|█████▍    | 1788/3252 [1:14:13<58:47,  2.41s/it]
 55%|█████▌    | 1789/3252 [1:14:15<58:38,  2.40s/it]
 55%|█████▌    | 1790/3252 [1:14:18<58:31,  2.40s/it]
                                                     
{'loss': 0.3767, 'grad_norm': 0.21602711081504822, 'learning_rate': 8.866207362310583e-05, 'lm_loss': 0.023014836013317108, 'epoch': 2.2}

 55%|█████▌    | 1790/3252 [1:14:18<58:31,  2.40s/it]
 55%|█████▌    | 1791/3252 [1:14:20<59:05,  2.43s/it]
 55%|█████▌    | 1792/3252 [1:14:23<58:48,  2.42s/it]
 55%|█████▌    | 1793/3252 [1:14:25<58:36,  2.41s/it]
 55%|█████▌    | 1794/3252 [1:14:27<58:26,  2.41s/it]
 55%|█████▌    | 1795/3252 [1:14:30<58:17,  2.40s/it]
 55%|█████▌    | 1796/3252 [1:14:32<58:10,  2.40s/it]
 55%|█████▌    | 1797/3252 [1:14:35<59:16,  2.44s/it]
 55%|█████▌    | 1798/3252 [1:14:37<58:53,  2.43s/it]
 55%|█████▌    | 1799/3252 [1:14:40<58:57,  2.43s/it]
 55%|█████▌    | 1800/3252 [1:14:42<58:38,  2.42s/it]
                                                     
{'loss': 0.3793, 'grad_norm': 0.19425424933433533, 'learning_rate': 8.76723819236997e-05, 'lm_loss': 0.49767062067985535, 'epoch': 2.21}

 55%|█████▌    | 1800/3252 [1:14:42<58:38,  2.42s/it]
 55%|█████▌    | 1801/3252 [1:14:45<1:00:20,  2.49s/it]
 55%|█████▌    | 1802/3252 [1:14:47<59:35,  2.47s/it]  
 55%|█████▌    | 1803/3252 [1:14:50<59:03,  2.45s/it]
 55%|█████▌    | 1804/3252 [1:14:52<58:49,  2.44s/it]
 56%|█████▌    | 1805/3252 [1:14:54<58:37,  2.43s/it]
 56%|█████▌    | 1806/3252 [1:14:57<58:29,  2.43s/it]
 56%|█████▌    | 1807/3252 [1:14:59<58:14,  2.42s/it]
 56%|█████▌    | 1808/3252 [1:15:02<58:04,  2.41s/it]
 56%|█████▌    | 1809/3252 [1:15:04<57:52,  2.41s/it]
 56%|█████▌    | 1810/3252 [1:15:06<57:48,  2.41s/it]
                                                     
{'loss': 0.3629, 'grad_norm': 0.2057396024465561, 'learning_rate': 8.668391484868003e-05, 'lm_loss': 0.005481955129653215, 'epoch': 2.23}

 56%|█████▌    | 1810/3252 [1:15:06<57:48,  2.41s/it]
 56%|█████▌    | 1811/3252 [1:15:09<57:53,  2.41s/it]
 56%|█████▌    | 1812/3252 [1:15:11<57:45,  2.41s/it]
 56%|█████▌    | 1813/3252 [1:15:14<57:34,  2.40s/it]
 56%|█████▌    | 1814/3252 [1:15:16<57:29,  2.40s/it]
 56%|█████▌    | 1815/3252 [1:15:18<58:09,  2.43s/it]
 56%|█████▌    | 1816/3252 [1:15:21<57:50,  2.42s/it]
 56%|█████▌    | 1817/3252 [1:15:23<57:36,  2.41s/it]
 56%|█████▌    | 1818/3252 [1:15:26<59:07,  2.47s/it]
 56%|█████▌    | 1819/3252 [1:15:28<58:30,  2.45s/it]
 56%|█████▌    | 1820/3252 [1:15:31<58:01,  2.43s/it]
                                                     
{'loss': 0.3628, 'grad_norm': 0.2197989970445633, 'learning_rate': 8.569677059227032e-05, 'lm_loss': 0.2908264398574829, 'epoch': 2.24}

 56%|█████▌    | 1820/3252 [1:15:31<58:01,  2.43s/it]
 56%|█████▌    | 1821/3252 [1:15:33<58:45,  2.46s/it]
 56%|█████▌    | 1822/3252 [1:15:36<58:12,  2.44s/it]
 56%|█████▌    | 1823/3252 [1:15:38<57:46,  2.43s/it]
 56%|█████▌    | 1824/3252 [1:15:40<57:29,  2.42s/it]
 56%|█████▌    | 1825/3252 [1:15:43<57:17,  2.41s/it]
 56%|█████▌    | 1826/3252 [1:15:45<57:10,  2.41s/it]
 56%|█████▌    | 1827/3252 [1:15:48<57:05,  2.40s/it]
 56%|█████▌    | 1828/3252 [1:15:50<56:57,  2.40s/it]
 56%|█████▌    | 1829/3252 [1:15:52<56:55,  2.40s/it]
 56%|█████▋    | 1830/3252 [1:15:55<56:53,  2.40s/it]
                                                     
{'loss': 0.3628, 'grad_norm': 0.21709097921848297, 'learning_rate': 8.471104721728551e-05, 'lm_loss': 0.3709929883480072, 'epoch': 2.25}

 56%|█████▋    | 1830/3252 [1:15:55<56:53,  2.40s/it]
 56%|█████▋    | 1831/3252 [1:15:57<56:59,  2.41s/it]
 56%|█████▋    | 1832/3252 [1:16:00<56:53,  2.40s/it]
 56%|█████▋    | 1833/3252 [1:16:02<56:47,  2.40s/it]
 56%|█████▋    | 1834/3252 [1:16:04<56:45,  2.40s/it]
 56%|█████▋    | 1835/3252 [1:16:07<56:42,  2.40s/it]
 56%|█████▋    | 1836/3252 [1:16:09<58:27,  2.48s/it]
 56%|█████▋    | 1837/3252 [1:16:12<58:39,  2.49s/it]
 57%|█████▋    | 1838/3252 [1:16:14<57:56,  2.46s/it]
 57%|█████▋    | 1839/3252 [1:16:17<57:28,  2.44s/it]
 57%|█████▋    | 1840/3252 [1:16:19<57:14,  2.43s/it]
                                                     
{'loss': 0.3754, 'grad_norm': 0.20598646998405457, 'learning_rate': 8.372684264539024e-05, 'lm_loss': 0.00932758953422308, 'epoch': 2.26}

 57%|█████▋    | 1840/3252 [1:16:19<57:14,  2.43s/it]
 57%|█████▋    | 1841/3252 [1:16:22<57:54,  2.46s/it]
 57%|█████▋    | 1842/3252 [1:16:24<57:24,  2.44s/it]
 57%|█████▋    | 1843/3252 [1:16:26<57:02,  2.43s/it]
 57%|█████▋    | 1844/3252 [1:16:29<56:49,  2.42s/it]
 57%|█████▋    | 1845/3252 [1:16:31<56:35,  2.41s/it]
 57%|█████▋    | 1846/3252 [1:16:34<56:20,  2.40s/it]
 57%|█████▋    | 1847/3252 [1:16:36<56:15,  2.40s/it]
 57%|█████▋    | 1848/3252 [1:16:38<56:10,  2.40s/it]
 57%|█████▋    | 1849/3252 [1:16:41<56:04,  2.40s/it]
 57%|█████▋    | 1850/3252 [1:16:43<55:58,  2.40s/it]
                                                     
{'loss': 0.3553, 'grad_norm': 0.26240217685699463, 'learning_rate': 8.274425464737139e-05, 'lm_loss': 0.403486967086792, 'epoch': 2.28}

 57%|█████▋    | 1850/3252 [1:16:43<55:58,  2.40s/it]
 57%|█████▋    | 1851/3252 [1:16:46<56:22,  2.41s/it]
 57%|█████▋    | 1852/3252 [1:16:48<56:12,  2.41s/it]
 57%|█████▋    | 1853/3252 [1:16:50<56:06,  2.41s/it]
 57%|█████▋    | 1854/3252 [1:16:53<56:47,  2.44s/it]
 57%|█████▋    | 1855/3252 [1:16:55<56:25,  2.42s/it]
 57%|█████▋    | 1856/3252 [1:16:58<56:12,  2.42s/it]
 57%|█████▋    | 1857/3252 [1:17:00<56:07,  2.41s/it]
 57%|█████▋    | 1858/3252 [1:17:03<55:59,  2.41s/it]
 57%|█████▋    | 1859/3252 [1:17:05<55:51,  2.41s/it]
 57%|█████▋    | 1860/3252 [1:17:07<55:45,  2.40s/it]
                                                     
{'loss': 0.3782, 'grad_norm': 0.2922963500022888, 'learning_rate': 8.176338083342561e-05, 'lm_loss': 0.4289160370826721, 'epoch': 2.29}

 57%|█████▋    | 1860/3252 [1:17:07<55:45,  2.40s/it]
 57%|█████▋    | 1861/3252 [1:17:10<55:43,  2.40s/it]
 57%|█████▋    | 1862/3252 [1:17:12<55:39,  2.40s/it]
 57%|█████▋    | 1863/3252 [1:17:15<55:33,  2.40s/it]
 57%|█████▋    | 1864/3252 [1:17:17<55:30,  2.40s/it]
 57%|█████▋    | 1865/3252 [1:17:19<55:30,  2.40s/it]
 57%|█████▋    | 1866/3252 [1:17:22<55:26,  2.40s/it]
 57%|█████▋    | 1867/3252 [1:17:24<55:21,  2.40s/it]
 57%|█████▋    | 1868/3252 [1:17:27<55:25,  2.40s/it]
 57%|█████▋    | 1869/3252 [1:17:29<55:20,  2.40s/it]
 58%|█████▊    | 1870/3252 [1:17:31<55:13,  2.40s/it]
                                                     
{'loss': 0.373, 'grad_norm': 0.24309758841991425, 'learning_rate': 8.078431864346265e-05, 'lm_loss': 0.3727644085884094, 'epoch': 2.3}

 58%|█████▊    | 1870/3252 [1:17:32<55:13,  2.40s/it]
 58%|█████▊    | 1871/3252 [1:17:34<56:40,  2.46s/it]
 58%|█████▊    | 1872/3252 [1:17:36<56:12,  2.44s/it]
 58%|█████▊    | 1873/3252 [1:17:39<55:57,  2.43s/it]
 58%|█████▊    | 1874/3252 [1:17:41<55:38,  2.42s/it]
 58%|█████▊    | 1875/3252 [1:17:44<55:25,  2.41s/it]
 58%|█████▊    | 1876/3252 [1:17:46<55:16,  2.41s/it]
 58%|█████▊    | 1877/3252 [1:17:48<55:11,  2.41s/it]
 58%|█████▊    | 1878/3252 [1:17:51<55:04,  2.40s/it]
 58%|█████▊    | 1879/3252 [1:17:53<54:57,  2.40s/it]
 58%|█████▊    | 1880/3252 [1:17:56<54:54,  2.40s/it]
                                                     
{'loss': 0.3563, 'grad_norm': 0.20691296458244324, 'learning_rate': 7.980716533742558e-05, 'lm_loss': 0.1487201303243637, 'epoch': 2.31}

 58%|█████▊    | 1880/3252 [1:17:56<54:54,  2.40s/it]
 58%|█████▊    | 1881/3252 [1:17:58<56:20,  2.47s/it]
 58%|█████▊    | 1882/3252 [1:18:01<55:49,  2.44s/it]
 58%|█████▊    | 1883/3252 [1:18:03<55:28,  2.43s/it]
 58%|█████▊    | 1884/3252 [1:18:05<55:09,  2.42s/it]
 58%|█████▊    | 1885/3252 [1:18:08<54:55,  2.41s/it]
 58%|█████▊    | 1886/3252 [1:18:10<54:42,  2.40s/it]
 58%|█████▊    | 1887/3252 [1:18:13<54:38,  2.40s/it]
 58%|█████▊    | 1888/3252 [1:18:15<54:33,  2.40s/it]
 58%|█████▊    | 1889/3252 [1:18:18<55:28,  2.44s/it]
 58%|█████▊    | 1890/3252 [1:18:20<55:06,  2.43s/it]
                                                     
{'loss': 0.3669, 'grad_norm': 0.1982564777135849, 'learning_rate': 7.88320179856294e-05, 'lm_loss': 1.0048162937164307, 'epoch': 2.32}

 58%|█████▊    | 1890/3252 [1:18:20<55:06,  2.43s/it]
 58%|█████▊    | 1891/3252 [1:18:22<55:42,  2.46s/it]
 58%|█████▊    | 1892/3252 [1:18:25<56:32,  2.49s/it]
 58%|█████▊    | 1893/3252 [1:18:27<55:47,  2.46s/it]
 58%|█████▊    | 1894/3252 [1:18:30<55:16,  2.44s/it]
 58%|█████▊    | 1895/3252 [1:18:32<55:01,  2.43s/it]
 58%|█████▊    | 1896/3252 [1:18:35<54:46,  2.42s/it]
 58%|█████▊    | 1897/3252 [1:18:37<54:30,  2.41s/it]
 58%|█████▊    | 1898/3252 [1:18:39<54:24,  2.41s/it]
 58%|█████▊    | 1899/3252 [1:18:42<54:13,  2.41s/it]
 58%|█████▊    | 1900/3252 [1:18:44<54:08,  2.40s/it]
                                                     
{'loss': 0.3603, 'grad_norm': 0.20033453404903412, 'learning_rate': 7.785897345911754e-05, 'lm_loss': 0.3127318024635315, 'epoch': 2.34}

 58%|█████▊    | 1900/3252 [1:18:45<54:08,  2.40s/it]
 58%|█████▊    | 1901/3252 [1:18:47<56:02,  2.49s/it]
 58%|█████▊    | 1902/3252 [1:18:49<55:22,  2.46s/it]
 59%|█████▊    | 1903/3252 [1:18:52<54:52,  2.44s/it]
 59%|█████▊    | 1904/3252 [1:18:54<54:35,  2.43s/it]
 59%|█████▊    | 1905/3252 [1:18:56<54:21,  2.42s/it]
 59%|█████▊    | 1906/3252 [1:18:59<54:04,  2.41s/it]
 59%|█████▊    | 1907/3252 [1:19:01<53:51,  2.40s/it]
 59%|█████▊    | 1908/3252 [1:19:04<53:43,  2.40s/it]
 59%|█████▊    | 1909/3252 [1:19:06<53:34,  2.39s/it]
 59%|█████▊    | 1910/3252 [1:19:09<55:07,  2.46s/it]
                                                     
{'loss': 0.3505, 'grad_norm': 0.28005707263946533, 'learning_rate': 7.68881284200391e-05, 'lm_loss': 0.041736386716365814, 'epoch': 2.35}

 59%|█████▊    | 1910/3252 [1:19:09<55:07,  2.46s/it]
 59%|█████▉    | 1911/3252 [1:19:12<57:51,  2.59s/it]
 59%|█████▉    | 1912/3252 [1:19:14<56:25,  2.53s/it]
 59%|█████▉    | 1913/3252 [1:19:16<55:26,  2.48s/it]
 59%|█████▉    | 1914/3252 [1:19:19<54:47,  2.46s/it]
 59%|█████▉    | 1915/3252 [1:19:21<54:19,  2.44s/it]
 59%|█████▉    | 1916/3252 [1:19:23<53:56,  2.42s/it]
 59%|█████▉    | 1917/3252 [1:19:26<53:44,  2.42s/it]
 59%|█████▉    | 1918/3252 [1:19:28<53:32,  2.41s/it]
 59%|█████▉    | 1919/3252 [1:19:31<53:23,  2.40s/it]
 59%|█████▉    | 1920/3252 [1:19:33<53:21,  2.40s/it]
                                                     
{'loss': 0.3568, 'grad_norm': 0.20755290985107422, 'learning_rate': 7.591957931204628e-05, 'lm_loss': 0.024085169658064842, 'epoch': 2.36}

 59%|█████▉    | 1920/3252 [1:19:33<53:21,  2.40s/it]
 59%|█████▉    | 1921/3252 [1:19:36<53:41,  2.42s/it]
 59%|█████▉    | 1922/3252 [1:19:38<53:28,  2.41s/it]
 59%|█████▉    | 1923/3252 [1:19:40<53:19,  2.41s/it]
 59%|█████▉    | 1924/3252 [1:19:43<53:15,  2.41s/it]
 59%|█████▉    | 1925/3252 [1:19:45<53:15,  2.41s/it]
 59%|█████▉    | 1926/3252 [1:19:48<53:18,  2.41s/it]
 59%|█████▉    | 1927/3252 [1:19:50<53:08,  2.41s/it]
 59%|█████▉    | 1928/3252 [1:19:53<54:28,  2.47s/it]
 59%|█████▉    | 1929/3252 [1:19:55<54:09,  2.46s/it]
 59%|█████▉    | 1930/3252 [1:19:57<53:44,  2.44s/it]
                                                     
{'loss': 0.3611, 'grad_norm': 0.27971139550209045, 'learning_rate': 7.495342235071377e-05, 'lm_loss': 0.6024420857429504, 'epoch': 2.37}

 59%|█████▉    | 1930/3252 [1:19:58<53:44,  2.44s/it]
 59%|█████▉    | 1931/3252 [1:20:00<54:30,  2.48s/it]
 59%|█████▉    | 1932/3252 [1:20:02<53:53,  2.45s/it]
 59%|█████▉    | 1933/3252 [1:20:05<53:34,  2.44s/it]
 59%|█████▉    | 1934/3252 [1:20:07<53:15,  2.42s/it]
 60%|█████▉    | 1935/3252 [1:20:10<53:00,  2.41s/it]
 60%|█████▉    | 1936/3252 [1:20:12<52:49,  2.41s/it]
 60%|█████▉    | 1937/3252 [1:20:14<52:40,  2.40s/it]
 60%|█████▉    | 1938/3252 [1:20:17<52:32,  2.40s/it]
 60%|█████▉    | 1939/3252 [1:20:19<52:24,  2.39s/it]
 60%|█████▉    | 1940/3252 [1:20:21<52:24,  2.40s/it]
                                                     
{'loss': 0.3618, 'grad_norm': 0.23900099098682404, 'learning_rate': 7.398975351398053e-05, 'lm_loss': 0.6581873297691345, 'epoch': 2.39}

 60%|█████▉    | 1940/3252 [1:20:22<52:24,  2.40s/it]
 60%|█████▉    | 1941/3252 [1:20:24<52:30,  2.40s/it]
 60%|█████▉    | 1942/3252 [1:20:26<52:33,  2.41s/it]
 60%|█████▉    | 1943/3252 [1:20:29<52:23,  2.40s/it]
 60%|█████▉    | 1944/3252 [1:20:31<52:17,  2.40s/it]
 60%|█████▉    | 1945/3252 [1:20:34<53:00,  2.43s/it]
 60%|█████▉    | 1946/3252 [1:20:36<52:42,  2.42s/it]
 60%|█████▉    | 1947/3252 [1:20:38<52:29,  2.41s/it]
 60%|█████▉    | 1948/3252 [1:20:41<52:21,  2.41s/it]
 60%|█████▉    | 1949/3252 [1:20:43<52:12,  2.40s/it]
 60%|█████▉    | 1950/3252 [1:20:46<52:04,  2.40s/it]
                                                     
{'loss': 0.3606, 'grad_norm': 0.2611124515533447, 'learning_rate': 7.302866853261552e-05, 'lm_loss': 0.008802708238363266, 'epoch': 2.4}

 60%|█████▉    | 1950/3252 [1:20:46<52:04,  2.40s/it]
 60%|█████▉    | 1951/3252 [1:20:48<52:53,  2.44s/it]
 60%|██████    | 1952/3252 [1:20:50<52:35,  2.43s/it]
 60%|██████    | 1953/3252 [1:20:53<52:24,  2.42s/it]
 60%|██████    | 1954/3252 [1:20:55<52:32,  2.43s/it]
 60%|██████    | 1955/3252 [1:20:58<52:19,  2.42s/it]
 60%|██████    | 1956/3252 [1:21:00<52:15,  2.42s/it]
 60%|██████    | 1957/3252 [1:21:03<52:11,  2.42s/it]
 60%|██████    | 1958/3252 [1:21:05<52:00,  2.41s/it]
 60%|██████    | 1959/3252 [1:21:07<51:50,  2.41s/it]
 60%|██████    | 1960/3252 [1:21:10<51:44,  2.40s/it]
                                                     
{'loss': 0.3644, 'grad_norm': 0.22068430483341217, 'learning_rate': 7.20702628807078e-05, 'lm_loss': 0.5229380130767822, 'epoch': 2.41}

 60%|██████    | 1960/3252 [1:21:10<51:44,  2.40s/it]
 60%|██████    | 1961/3252 [1:21:12<52:47,  2.45s/it]
 60%|██████    | 1962/3252 [1:21:15<52:23,  2.44s/it]
 60%|██████    | 1963/3252 [1:21:17<52:59,  2.47s/it]
 60%|██████    | 1964/3252 [1:21:20<52:25,  2.44s/it]
 60%|██████    | 1965/3252 [1:21:22<52:05,  2.43s/it]
 60%|██████    | 1966/3252 [1:21:25<53:16,  2.49s/it]
 60%|██████    | 1967/3252 [1:21:27<52:36,  2.46s/it]
 61%|██████    | 1968/3252 [1:21:29<52:09,  2.44s/it]
 61%|██████    | 1969/3252 [1:21:32<51:51,  2.43s/it]
 61%|██████    | 1970/3252 [1:21:34<51:36,  2.42s/it]
                                                     
{'loss': 0.3508, 'grad_norm': 0.21821144223213196, 'learning_rate': 7.111463176618205e-05, 'lm_loss': 0.5263713002204895, 'epoch': 2.42}

 61%|██████    | 1970/3252 [1:21:34<51:36,  2.42s/it]
 61%|██████    | 1971/3252 [1:21:37<52:00,  2.44s/it]
 61%|██████    | 1972/3252 [1:21:39<51:43,  2.42s/it]
 61%|██████    | 1973/3252 [1:21:42<51:28,  2.41s/it]
 61%|██████    | 1974/3252 [1:21:44<51:19,  2.41s/it]
 61%|██████    | 1975/3252 [1:21:46<51:15,  2.41s/it]
 61%|██████    | 1976/3252 [1:21:49<51:09,  2.41s/it]
 61%|██████    | 1977/3252 [1:21:51<51:06,  2.40s/it]
 61%|██████    | 1978/3252 [1:21:54<51:01,  2.40s/it]
 61%|██████    | 1979/3252 [1:21:56<50:56,  2.40s/it]
 61%|██████    | 1980/3252 [1:21:58<50:55,  2.40s/it]
                                                     
{'loss': 0.3562, 'grad_norm': 0.20427218079566956, 'learning_rate': 7.016187012134068e-05, 'lm_loss': 0.030502071604132652, 'epoch': 2.44}

 61%|██████    | 1980/3252 [1:21:59<50:55,  2.40s/it]
 61%|██████    | 1981/3252 [1:22:01<52:04,  2.46s/it]
 61%|██████    | 1982/3252 [1:22:03<51:37,  2.44s/it]
 61%|██████    | 1983/3252 [1:22:06<51:23,  2.43s/it]
 61%|██████    | 1984/3252 [1:22:08<53:10,  2.52s/it]
 61%|██████    | 1985/3252 [1:22:11<53:14,  2.52s/it]
 61%|██████    | 1986/3252 [1:22:13<52:22,  2.48s/it]
 61%|██████    | 1987/3252 [1:22:16<51:44,  2.45s/it]
 61%|██████    | 1988/3252 [1:22:18<51:20,  2.44s/it]
 61%|██████    | 1989/3252 [1:22:21<51:06,  2.43s/it]
 61%|██████    | 1990/3252 [1:22:23<51:01,  2.43s/it]
                                                     
{'loss': 0.3662, 'grad_norm': 0.22642341256141663, 'learning_rate': 6.921207259343323e-05, 'lm_loss': 0.9926157593727112, 'epoch': 2.45}

 61%|██████    | 1990/3252 [1:22:23<51:01,  2.43s/it]
 61%|██████    | 1991/3252 [1:22:25<51:23,  2.45s/it]
 61%|██████▏   | 1992/3252 [1:22:28<51:00,  2.43s/it]
 61%|██████▏   | 1993/3252 [1:22:30<50:45,  2.42s/it]
 61%|██████▏   | 1994/3252 [1:22:33<50:36,  2.41s/it]
 61%|██████▏   | 1995/3252 [1:22:35<50:27,  2.41s/it]
 61%|██████▏   | 1996/3252 [1:22:37<50:18,  2.40s/it]
 61%|██████▏   | 1997/3252 [1:22:40<50:12,  2.40s/it]
 61%|██████▏   | 1998/3252 [1:22:42<50:07,  2.40s/it]
 61%|██████▏   | 1999/3252 [1:22:45<50:04,  2.40s/it]
 62%|██████▏   | 2000/3252 [1:22:47<50:00,  2.40s/it]
                                                     
{'loss': 0.3559, 'grad_norm': 0.23163264989852905, 'learning_rate': 6.826533353525428e-05, 'lm_loss': 1.0270631313323975, 'epoch': 2.46}

 62%|██████▏   | 2000/3252 [1:22:47<50:00,  2.40s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.92it/s]
                                                     

                                             
{'eval_sthsthv2/narration_val_loss': 0.7798206210136414, 'eval_sthsthv2/narration_val_runtime': 3.7853, 'eval_sthsthv2/narration_val_samples_per_second': 176.471, 'eval_sthsthv2/narration_val_steps_per_second': 0.528, 'epoch': 2.46}

 62%|██████▏   | 2000/3252 [1:22:51<50:00,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.92it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.63it/s]
                                                     

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6367776989936829, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.1697, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 186.77, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.631, 'epoch': 2.46}

 62%|██████▏   | 2000/3252 [1:22:54<50:00,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.63it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.80it/s]
                                                     

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.5886566042900085, 'eval_holoassist/dialog-klg-sum_val_runtime': 3.0211, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 234.681, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.662, 'epoch': 2.46}

 62%|██████▏   | 2000/3252 [1:22:57<50:00,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.80it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                     

                                     
{'eval_ego4d/narration_val_loss': 0.5185419917106628, 'eval_ego4d/narration_val_runtime': 2.392, 'eval_ego4d/narration_val_samples_per_second': 93.226, 'eval_ego4d/narration_val_steps_per_second': 0.418, 'epoch': 2.46}

 62%|██████▏   | 2000/3252 [1:23:00<50:00,  2.40s/it]

100%|██████████| 1/1 [00:00<00:00,  2.49it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.54it/s]
                                                     

                                             
{'eval_ego4d/summary_val_loss': 1.1400789022445679, 'eval_ego4d/summary_val_runtime': 2.9083, 'eval_ego4d/summary_val_samples_per_second': 193.926, 'eval_ego4d/summary_val_steps_per_second': 0.688, 'epoch': 2.46}

 62%|██████▏   | 2000/3252 [1:23:02<50:00,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.54it/s]

                                             
 62%|██████▏   | 2001/3252 [1:23:05<2:27:02,  7.05s/it]
 62%|██████▏   | 2002/3252 [1:23:07<1:58:41,  5.70s/it]
 62%|██████▏   | 2003/3252 [1:23:10<1:37:57,  4.71s/it]
 62%|██████▏   | 2004/3252 [1:23:12<1:23:31,  4.02s/it]
 62%|██████▏   | 2005/3252 [1:23:15<1:13:20,  3.53s/it]
 62%|██████▏   | 2006/3252 [1:23:17<1:06:11,  3.19s/it]
 62%|██████▏   | 2007/3252 [1:23:19<1:01:16,  2.95s/it]
 62%|██████▏   | 2008/3252 [1:23:22<57:46,  2.79s/it]  
 62%|██████▏   | 2009/3252 [1:23:24<55:17,  2.67s/it]
 62%|██████▏   | 2010/3252 [1:23:27<53:33,  2.59s/it]
                                                     
{'loss': 0.3625, 'grad_norm': 0.22072692215442657, 'learning_rate': 6.732174699577015e-05, 'lm_loss': 0.5390871167182922, 'epoch': 2.47}

 62%|██████▏   | 2010/3252 [1:23:27<53:33,  2.59s/it]
 62%|██████▏   | 2011/3252 [1:23:29<52:28,  2.54s/it]
 62%|██████▏   | 2012/3252 [1:23:31<51:36,  2.50s/it]
 62%|██████▏   | 2013/3252 [1:23:34<50:55,  2.47s/it]
 62%|██████▏   | 2014/3252 [1:23:36<50:25,  2.44s/it]
 62%|██████▏   | 2015/3252 [1:23:39<50:09,  2.43s/it]
 62%|██████▏   | 2016/3252 [1:23:41<49:52,  2.42s/it]
 62%|██████▏   | 2017/3252 [1:23:43<49:38,  2.41s/it]
 62%|██████▏   | 2018/3252 [1:23:46<49:28,  2.41s/it]
 62%|██████▏   | 2019/3252 [1:23:48<49:21,  2.40s/it]
 62%|██████▏   | 2020/3252 [1:23:51<50:41,  2.47s/it]
                                                     
{'loss': 0.361, 'grad_norm': 0.2666431963443756, 'learning_rate': 6.638140671077633e-05, 'lm_loss': 0.006521072704344988, 'epoch': 2.48}

 62%|██████▏   | 2020/3252 [1:23:51<50:41,  2.47s/it]
 62%|██████▏   | 2021/3252 [1:23:53<51:08,  2.49s/it]
 62%|██████▏   | 2022/3252 [1:23:56<50:29,  2.46s/it]
 62%|██████▏   | 2023/3252 [1:23:58<49:58,  2.44s/it]
 62%|██████▏   | 2024/3252 [1:24:01<49:33,  2.42s/it]
 62%|██████▏   | 2025/3252 [1:24:03<49:22,  2.41s/it]
 62%|██████▏   | 2026/3252 [1:24:05<49:13,  2.41s/it]
 62%|██████▏   | 2027/3252 [1:24:08<49:06,  2.40s/it]
 62%|██████▏   | 2028/3252 [1:24:10<49:33,  2.43s/it]
 62%|██████▏   | 2029/3252 [1:24:13<49:16,  2.42s/it]
 62%|██████▏   | 2030/3252 [1:24:15<49:05,  2.41s/it]
                                                     
{'loss': 0.346, 'grad_norm': 0.22221514582633972, 'learning_rate': 6.544440609358575e-05, 'lm_loss': 0.3870696425437927, 'epoch': 2.5}

 62%|██████▏   | 2030/3252 [1:24:15<49:05,  2.41s/it]
 62%|██████▏   | 2031/3252 [1:24:18<50:07,  2.46s/it]
 62%|██████▏   | 2032/3252 [1:24:20<49:42,  2.45s/it]
 63%|██████▎   | 2033/3252 [1:24:22<49:23,  2.43s/it]
 63%|██████▎   | 2034/3252 [1:24:25<49:08,  2.42s/it]
 63%|██████▎   | 2035/3252 [1:24:27<48:58,  2.41s/it]
 63%|██████▎   | 2036/3252 [1:24:30<48:50,  2.41s/it]
 63%|██████▎   | 2037/3252 [1:24:32<49:18,  2.43s/it]
 63%|██████▎   | 2038/3252 [1:24:34<49:03,  2.42s/it]
 63%|██████▎   | 2039/3252 [1:24:37<48:47,  2.41s/it]
 63%|██████▎   | 2040/3252 [1:24:39<48:36,  2.41s/it]
                                                     
{'loss': 0.3437, 'grad_norm': 0.20660680532455444, 'learning_rate': 6.451083822574906e-05, 'lm_loss': 0.006160872522741556, 'epoch': 2.51}

 63%|██████▎   | 2040/3252 [1:24:39<48:36,  2.41s/it]
 63%|██████▎   | 2041/3252 [1:24:42<49:12,  2.44s/it]
 63%|██████▎   | 2042/3252 [1:24:44<48:52,  2.42s/it]
 63%|██████▎   | 2043/3252 [1:24:47<48:40,  2.42s/it]
 63%|██████▎   | 2044/3252 [1:24:49<48:32,  2.41s/it]
 63%|██████▎   | 2045/3252 [1:24:51<48:23,  2.41s/it]
 63%|██████▎   | 2046/3252 [1:24:54<48:21,  2.41s/it]
 63%|██████▎   | 2047/3252 [1:24:56<48:13,  2.40s/it]
 63%|██████▎   | 2048/3252 [1:24:59<48:04,  2.40s/it]
 63%|██████▎   | 2049/3252 [1:25:01<48:02,  2.40s/it]
 63%|██████▎   | 2050/3252 [1:25:03<48:04,  2.40s/it]
                                                     
{'loss': 0.3376, 'grad_norm': 0.21111372113227844, 'learning_rate': 6.35807958478078e-05, 'lm_loss': 0.40997114777565, 'epoch': 2.52}

 63%|██████▎   | 2050/3252 [1:25:04<48:04,  2.40s/it]
 63%|██████▎   | 2051/3252 [1:25:06<49:01,  2.45s/it]
 63%|██████▎   | 2052/3252 [1:25:08<48:40,  2.43s/it]
 63%|██████▎   | 2053/3252 [1:25:11<48:24,  2.42s/it]
 63%|██████▎   | 2054/3252 [1:25:13<48:16,  2.42s/it]
 63%|██████▎   | 2055/3252 [1:25:16<48:05,  2.41s/it]
 63%|██████▎   | 2056/3252 [1:25:18<48:29,  2.43s/it]
 63%|██████▎   | 2057/3252 [1:25:21<49:24,  2.48s/it]
 63%|██████▎   | 2058/3252 [1:25:23<51:22,  2.58s/it]
 63%|██████▎   | 2059/3252 [1:25:26<50:15,  2.53s/it]
 63%|██████▎   | 2060/3252 [1:25:28<49:23,  2.49s/it]
                                                     
{'loss': 0.3527, 'grad_norm': 0.20052087306976318, 'learning_rate': 6.26543713500818e-05, 'lm_loss': 0.16763317584991455, 'epoch': 2.53}

 63%|██████▎   | 2060/3252 [1:25:28<49:23,  2.49s/it]
 63%|██████▎   | 2061/3252 [1:25:31<49:01,  2.47s/it]
 63%|██████▎   | 2062/3252 [1:25:33<48:34,  2.45s/it]
 63%|██████▎   | 2063/3252 [1:25:35<48:13,  2.43s/it]
 63%|██████▎   | 2064/3252 [1:25:38<47:56,  2.42s/it]
 63%|██████▎   | 2065/3252 [1:25:40<47:47,  2.42s/it]
 64%|██████▎   | 2066/3252 [1:25:43<47:38,  2.41s/it]
 64%|██████▎   | 2067/3252 [1:25:45<47:29,  2.40s/it]
 64%|██████▎   | 2068/3252 [1:25:47<47:25,  2.40s/it]
 64%|██████▎   | 2069/3252 [1:25:50<47:21,  2.40s/it]
 64%|██████▎   | 2070/3252 [1:25:52<47:17,  2.40s/it]
                                                     
{'loss': 0.3455, 'grad_norm': 0.22500871121883392, 'learning_rate': 6.173165676349103e-05, 'lm_loss': 0.20609742403030396, 'epoch': 2.55}

 64%|██████▎   | 2070/3252 [1:25:52<47:17,  2.40s/it]
 64%|██████▎   | 2071/3252 [1:25:55<47:58,  2.44s/it]
 64%|██████▎   | 2072/3252 [1:25:57<47:38,  2.42s/it]
 64%|██████▎   | 2073/3252 [1:26:00<47:24,  2.41s/it]
 64%|██████▍   | 2074/3252 [1:26:02<47:18,  2.41s/it]
 64%|██████▍   | 2075/3252 [1:26:04<47:07,  2.40s/it]
 64%|██████▍   | 2076/3252 [1:26:07<48:13,  2.46s/it]
 64%|██████▍   | 2077/3252 [1:26:09<48:26,  2.47s/it]
 64%|██████▍   | 2078/3252 [1:26:12<47:59,  2.45s/it]
 64%|██████▍   | 2079/3252 [1:26:14<47:36,  2.43s/it]
 64%|██████▍   | 2080/3252 [1:26:17<47:17,  2.42s/it]
                                                     
{'loss': 0.345, 'grad_norm': 0.2034519463777542, 'learning_rate': 6.081274375041317e-05, 'lm_loss': 0.3098556101322174, 'epoch': 2.56}

 64%|██████▍   | 2080/3252 [1:26:17<47:17,  2.42s/it]
 64%|██████▍   | 2081/3252 [1:26:19<47:16,  2.42s/it]
 64%|██████▍   | 2082/3252 [1:26:21<47:02,  2.41s/it]
 64%|██████▍   | 2083/3252 [1:26:24<46:51,  2.41s/it]
 64%|██████▍   | 2084/3252 [1:26:26<46:44,  2.40s/it]
 64%|██████▍   | 2085/3252 [1:26:29<46:39,  2.40s/it]
 64%|██████▍   | 2086/3252 [1:26:31<46:37,  2.40s/it]
 64%|██████▍   | 2087/3252 [1:26:33<46:32,  2.40s/it]
 64%|██████▍   | 2088/3252 [1:26:36<46:28,  2.40s/it]
 64%|██████▍   | 2089/3252 [1:26:38<46:25,  2.40s/it]
 64%|██████▍   | 2090/3252 [1:26:41<46:25,  2.40s/it]
                                                     
{'loss': 0.3456, 'grad_norm': 0.2175668478012085, 'learning_rate': 5.989772359557799e-05, 'lm_loss': 0.502048909664154, 'epoch': 2.57}

 64%|██████▍   | 2090/3252 [1:26:41<46:25,  2.40s/it]
 64%|██████▍   | 2091/3252 [1:26:43<46:26,  2.40s/it]
 64%|██████▍   | 2092/3252 [1:26:45<46:21,  2.40s/it]
 64%|██████▍   | 2093/3252 [1:26:48<46:15,  2.39s/it]
 64%|██████▍   | 2094/3252 [1:26:50<46:59,  2.43s/it]
 64%|██████▍   | 2095/3252 [1:26:53<46:46,  2.43s/it]
 64%|██████▍   | 2096/3252 [1:26:55<46:28,  2.41s/it]
 64%|██████▍   | 2097/3252 [1:26:58<47:17,  2.46s/it]
 65%|██████▍   | 2098/3252 [1:27:00<46:54,  2.44s/it]
 65%|██████▍   | 2099/3252 [1:27:02<46:33,  2.42s/it]
 65%|██████▍   | 2100/3252 [1:27:05<46:19,  2.41s/it]
                                                     
{'loss': 0.3443, 'grad_norm': 0.22375336289405823, 'learning_rate': 5.8986687196999135e-05, 'lm_loss': 0.4008207321166992, 'epoch': 2.58}

 65%|██████▍   | 2100/3252 [1:27:05<46:19,  2.41s/it]
 65%|██████▍   | 2101/3252 [1:27:07<46:26,  2.42s/it]
 65%|██████▍   | 2102/3252 [1:27:10<46:19,  2.42s/it]
 65%|██████▍   | 2103/3252 [1:27:12<46:06,  2.41s/it]
 65%|██████▍   | 2104/3252 [1:27:14<46:06,  2.41s/it]
 65%|██████▍   | 2105/3252 [1:27:17<46:08,  2.41s/it]
 65%|██████▍   | 2106/3252 [1:27:19<45:59,  2.41s/it]
 65%|██████▍   | 2107/3252 [1:27:22<45:50,  2.40s/it]
 65%|██████▍   | 2108/3252 [1:27:24<45:45,  2.40s/it]
 65%|██████▍   | 2109/3252 [1:27:26<45:38,  2.40s/it]
 65%|██████▍   | 2110/3252 [1:27:29<45:39,  2.40s/it]
                                                     
{'loss': 0.3458, 'grad_norm': 0.22888906300067902, 'learning_rate': 5.8079725056944244e-05, 'lm_loss': 0.43999597430229187, 'epoch': 2.6}

 65%|██████▍   | 2110/3252 [1:27:29<45:39,  2.40s/it]
 65%|██████▍   | 2111/3252 [1:27:31<46:16,  2.43s/it]
 65%|██████▍   | 2112/3252 [1:27:34<46:00,  2.42s/it]
 65%|██████▍   | 2113/3252 [1:27:36<46:00,  2.42s/it]
 65%|██████▌   | 2114/3252 [1:27:39<45:49,  2.42s/it]
 65%|██████▌   | 2115/3252 [1:27:41<45:39,  2.41s/it]
 65%|██████▌   | 2116/3252 [1:27:43<45:32,  2.41s/it]
 65%|██████▌   | 2117/3252 [1:27:46<45:28,  2.40s/it]
 65%|██████▌   | 2118/3252 [1:27:48<45:21,  2.40s/it]
 65%|██████▌   | 2119/3252 [1:27:51<45:16,  2.40s/it]
 65%|██████▌   | 2120/3252 [1:27:53<45:13,  2.40s/it]
                                                     
{'loss': 0.3493, 'grad_norm': 0.211945042014122, 'learning_rate': 5.717692727294453e-05, 'lm_loss': 0.44805219769477844, 'epoch': 2.61}

 65%|██████▌   | 2120/3252 [1:27:53<45:13,  2.40s/it]
 65%|██████▌   | 2121/3252 [1:27:55<45:17,  2.40s/it]
 65%|██████▌   | 2122/3252 [1:27:58<45:14,  2.40s/it]
 65%|██████▌   | 2123/3252 [1:28:00<45:13,  2.40s/it]
 65%|██████▌   | 2124/3252 [1:28:03<45:05,  2.40s/it]
 65%|██████▌   | 2125/3252 [1:28:05<45:01,  2.40s/it]
 65%|██████▌   | 2126/3252 [1:28:07<45:00,  2.40s/it]
 65%|██████▌   | 2127/3252 [1:28:10<44:57,  2.40s/it]
 65%|██████▌   | 2128/3252 [1:28:12<44:51,  2.39s/it]
 65%|██████▌   | 2129/3252 [1:28:15<44:53,  2.40s/it]
 65%|██████▌   | 2130/3252 [1:28:17<44:51,  2.40s/it]
                                                     
{'loss': 0.3426, 'grad_norm': 0.27004632353782654, 'learning_rate': 5.627838352884435e-05, 'lm_loss': 0.013378121890127659, 'epoch': 2.62}

 65%|██████▌   | 2130/3252 [1:28:17<44:51,  2.40s/it]
 66%|██████▌   | 2131/3252 [1:28:19<45:18,  2.43s/it]
 66%|██████▌   | 2132/3252 [1:28:22<47:23,  2.54s/it]
 66%|██████▌   | 2133/3252 [1:28:25<46:34,  2.50s/it]
 66%|██████▌   | 2134/3252 [1:28:27<45:58,  2.47s/it]
 66%|██████▌   | 2135/3252 [1:28:29<45:28,  2.44s/it]
 66%|██████▌   | 2136/3252 [1:28:32<45:09,  2.43s/it]
 66%|██████▌   | 2137/3252 [1:28:34<44:58,  2.42s/it]
 66%|██████▌   | 2138/3252 [1:28:37<45:15,  2.44s/it]
 66%|██████▌   | 2139/3252 [1:28:39<45:00,  2.43s/it]
 66%|██████▌   | 2140/3252 [1:28:41<44:45,  2.42s/it]
                                                     
{'loss': 0.3253, 'grad_norm': 0.20318396389484406, 'learning_rate': 5.538418308589246e-05, 'lm_loss': 0.311331182718277, 'epoch': 2.63}

 66%|██████▌   | 2140/3252 [1:28:42<44:45,  2.42s/it]
 66%|██████▌   | 2141/3252 [1:28:44<44:53,  2.42s/it]
 66%|██████▌   | 2142/3252 [1:28:46<44:47,  2.42s/it]
 66%|██████▌   | 2143/3252 [1:28:49<44:36,  2.41s/it]
 66%|██████▌   | 2144/3252 [1:28:51<44:29,  2.41s/it]
 66%|██████▌   | 2145/3252 [1:28:53<44:19,  2.40s/it]
 66%|██████▌   | 2146/3252 [1:28:56<44:14,  2.40s/it]
 66%|██████▌   | 2147/3252 [1:28:58<44:10,  2.40s/it]
 66%|██████▌   | 2148/3252 [1:29:01<44:05,  2.40s/it]
 66%|██████▌   | 2149/3252 [1:29:03<44:00,  2.39s/it]
 66%|██████▌   | 2150/3252 [1:29:06<44:15,  2.41s/it]
                                                     
{'loss': 0.3365, 'grad_norm': 0.21796080470085144, 'learning_rate': 5.44944147738742e-05, 'lm_loss': 0.002209671074524522, 'epoch': 2.64}

 66%|██████▌   | 2150/3252 [1:29:06<44:15,  2.41s/it]
 66%|██████▌   | 2151/3252 [1:29:08<45:26,  2.48s/it]
 66%|██████▌   | 2152/3252 [1:29:11<45:03,  2.46s/it]
 66%|██████▌   | 2153/3252 [1:29:13<45:08,  2.46s/it]
 66%|██████▌   | 2154/3252 [1:29:15<44:44,  2.45s/it]
 66%|██████▋   | 2155/3252 [1:29:18<44:34,  2.44s/it]
 66%|██████▋   | 2156/3252 [1:29:20<44:21,  2.43s/it]
 66%|██████▋   | 2157/3252 [1:29:23<44:11,  2.42s/it]
 66%|██████▋   | 2158/3252 [1:29:25<44:00,  2.41s/it]
 66%|██████▋   | 2159/3252 [1:29:27<43:55,  2.41s/it]
 66%|██████▋   | 2160/3252 [1:29:30<43:51,  2.41s/it]
                                                     
{'loss': 0.3337, 'grad_norm': 0.20214977860450745, 'learning_rate': 5.360916698228755e-05, 'lm_loss': 0.15300677716732025, 'epoch': 2.66}

 66%|██████▋   | 2160/3252 [1:29:30<43:51,  2.41s/it]
 66%|██████▋   | 2161/3252 [1:29:32<44:56,  2.47s/it]
 66%|██████▋   | 2162/3252 [1:29:35<44:25,  2.45s/it]
 67%|██████▋   | 2163/3252 [1:29:37<44:09,  2.43s/it]
 67%|██████▋   | 2164/3252 [1:29:40<43:53,  2.42s/it]
 67%|██████▋   | 2165/3252 [1:29:42<43:44,  2.41s/it]
 67%|██████▋   | 2166/3252 [1:29:44<43:35,  2.41s/it]
 67%|██████▋   | 2167/3252 [1:29:47<43:29,  2.40s/it]
 67%|██████▋   | 2168/3252 [1:29:49<44:06,  2.44s/it]
 67%|██████▋   | 2169/3252 [1:29:52<43:48,  2.43s/it]
 67%|██████▋   | 2170/3252 [1:29:54<43:37,  2.42s/it]
                                                     
{'loss': 0.3378, 'grad_norm': 0.21231919527053833, 'learning_rate': 5.272852765156248e-05, 'lm_loss': 0.5005823969841003, 'epoch': 2.67}

 67%|██████▋   | 2170/3252 [1:29:54<43:37,  2.42s/it]
 67%|██████▋   | 2171/3252 [1:29:57<44:21,  2.46s/it]
 67%|██████▋   | 2172/3252 [1:29:59<43:53,  2.44s/it]
 67%|██████▋   | 2173/3252 [1:30:02<43:37,  2.43s/it]
 67%|██████▋   | 2174/3252 [1:30:04<43:24,  2.42s/it]
 67%|██████▋   | 2175/3252 [1:30:06<43:13,  2.41s/it]
 67%|██████▋   | 2176/3252 [1:30:09<43:08,  2.41s/it]
 67%|██████▋   | 2177/3252 [1:30:11<43:02,  2.40s/it]
 67%|██████▋   | 2178/3252 [1:30:13<42:52,  2.40s/it]
 67%|██████▋   | 2179/3252 [1:30:16<42:48,  2.39s/it]
 67%|██████▋   | 2180/3252 [1:30:18<42:44,  2.39s/it]
                                                     
{'loss': 0.3285, 'grad_norm': 0.20175392925739288, 'learning_rate': 5.1852584264324866e-05, 'lm_loss': 0.4928959906101227, 'epoch': 2.68}

 67%|██████▋   | 2180/3252 [1:30:18<42:44,  2.39s/it]
 67%|██████▋   | 2181/3252 [1:30:21<43:40,  2.45s/it]
 67%|██████▋   | 2182/3252 [1:30:23<43:20,  2.43s/it]
 67%|██████▋   | 2183/3252 [1:30:26<43:07,  2.42s/it]
 67%|██████▋   | 2184/3252 [1:30:28<43:00,  2.42s/it]
 67%|██████▋   | 2185/3252 [1:30:31<43:25,  2.44s/it]
 67%|██████▋   | 2186/3252 [1:30:33<43:10,  2.43s/it]
 67%|██████▋   | 2187/3252 [1:30:35<42:57,  2.42s/it]
 67%|██████▋   | 2188/3252 [1:30:38<42:47,  2.41s/it]
 67%|██████▋   | 2189/3252 [1:30:40<42:37,  2.41s/it]
 67%|██████▋   | 2190/3252 [1:30:43<42:32,  2.40s/it]
                                                     
{'loss': 0.3303, 'grad_norm': 0.21337538957595825, 'learning_rate': 5.0981423836706e-05, 'lm_loss': 0.6438872814178467, 'epoch': 2.69}

 67%|██████▋   | 2190/3252 [1:30:43<42:32,  2.40s/it]
 67%|██████▋   | 2191/3252 [1:30:45<43:03,  2.44s/it]
 67%|██████▋   | 2192/3252 [1:30:47<42:51,  2.43s/it]
 67%|██████▋   | 2193/3252 [1:30:50<42:42,  2.42s/it]
 67%|██████▋   | 2194/3252 [1:30:52<42:32,  2.41s/it]
 67%|██████▋   | 2195/3252 [1:30:55<42:24,  2.41s/it]
 68%|██████▊   | 2196/3252 [1:30:57<42:19,  2.41s/it]
 68%|██████▊   | 2197/3252 [1:30:59<42:12,  2.40s/it]
 68%|██████▊   | 2198/3252 [1:31:02<42:08,  2.40s/it]
 68%|██████▊   | 2199/3252 [1:31:04<42:04,  2.40s/it]
 68%|██████▊   | 2200/3252 [1:31:07<42:02,  2.40s/it]
                                                     
{'loss': 0.3312, 'grad_norm': 0.23339852690696716, 'learning_rate': 5.0115132909698394e-05, 'lm_loss': 0.22359822690486908, 'epoch': 2.71}

 68%|██████▊   | 2200/3252 [1:31:07<42:02,  2.40s/it]
 68%|██████▊   | 2201/3252 [1:31:09<42:09,  2.41s/it]
 68%|██████▊   | 2202/3252 [1:31:11<42:01,  2.40s/it]
 68%|██████▊   | 2203/3252 [1:31:14<42:24,  2.43s/it]
 68%|██████▊   | 2204/3252 [1:31:16<42:15,  2.42s/it]
 68%|██████▊   | 2205/3252 [1:31:19<42:22,  2.43s/it]
 68%|██████▊   | 2206/3252 [1:31:22<43:59,  2.52s/it]
 68%|██████▊   | 2207/3252 [1:31:24<43:17,  2.49s/it]
 68%|██████▊   | 2208/3252 [1:31:26<42:45,  2.46s/it]
 68%|██████▊   | 2209/3252 [1:31:29<42:22,  2.44s/it]
 68%|██████▊   | 2210/3252 [1:31:31<42:07,  2.43s/it]
                                                     
{'loss': 0.3417, 'grad_norm': 0.2867107391357422, 'learning_rate': 4.925379754055891e-05, 'lm_loss': 0.20340491831302643, 'epoch': 2.72}

 68%|██████▊   | 2210/3252 [1:31:31<42:07,  2.43s/it]
 68%|██████▊   | 2211/3252 [1:31:34<42:13,  2.43s/it]
 68%|██████▊   | 2212/3252 [1:31:36<42:00,  2.42s/it]
 68%|██████▊   | 2213/3252 [1:31:38<41:48,  2.41s/it]
 68%|██████▊   | 2214/3252 [1:31:41<41:40,  2.41s/it]
 68%|██████▊   | 2215/3252 [1:31:43<41:31,  2.40s/it]
 68%|██████▊   | 2216/3252 [1:31:46<41:25,  2.40s/it]
 68%|██████▊   | 2217/3252 [1:31:48<41:21,  2.40s/it]
 68%|██████▊   | 2218/3252 [1:31:50<41:19,  2.40s/it]
 68%|██████▊   | 2219/3252 [1:31:53<41:18,  2.40s/it]
 68%|██████▊   | 2220/3252 [1:31:55<41:16,  2.40s/it]
                                                     
{'loss': 0.3358, 'grad_norm': 0.24211230874061584, 'learning_rate': 4.8397503294259795e-05, 'lm_loss': 0.3286496698856354, 'epoch': 2.73}

 68%|██████▊   | 2220/3252 [1:31:55<41:16,  2.40s/it]
 68%|██████▊   | 2221/3252 [1:31:58<42:18,  2.46s/it]
 68%|██████▊   | 2222/3252 [1:32:00<41:53,  2.44s/it]
 68%|██████▊   | 2223/3252 [1:32:02<41:38,  2.43s/it]
 68%|██████▊   | 2224/3252 [1:32:05<41:55,  2.45s/it]
 68%|██████▊   | 2225/3252 [1:32:07<41:44,  2.44s/it]
 68%|██████▊   | 2226/3252 [1:32:10<41:28,  2.43s/it]
 68%|██████▊   | 2227/3252 [1:32:12<41:16,  2.42s/it]
 69%|██████▊   | 2228/3252 [1:32:15<41:08,  2.41s/it]
 69%|██████▊   | 2229/3252 [1:32:17<41:00,  2.41s/it]
 69%|██████▊   | 2230/3252 [1:32:19<40:56,  2.40s/it]
                                                     
{'loss': 0.3226, 'grad_norm': 0.24631580710411072, 'learning_rate': 4.754633523498851e-05, 'lm_loss': 0.27053970098495483, 'epoch': 2.74}

 69%|██████▊   | 2230/3252 [1:32:19<40:56,  2.40s/it]
 69%|██████▊   | 2231/3252 [1:32:22<41:04,  2.41s/it]
 69%|██████▊   | 2232/3252 [1:32:24<40:57,  2.41s/it]
 69%|██████▊   | 2233/3252 [1:32:27<40:48,  2.40s/it]
 69%|██████▊   | 2234/3252 [1:32:29<40:41,  2.40s/it]
 69%|██████▊   | 2235/3252 [1:32:31<40:40,  2.40s/it]
 69%|██████▉   | 2236/3252 [1:32:34<40:37,  2.40s/it]
 69%|██████▉   | 2237/3252 [1:32:36<40:36,  2.40s/it]
 69%|██████▉   | 2238/3252 [1:32:39<40:30,  2.40s/it]
 69%|██████▉   | 2239/3252 [1:32:41<40:27,  2.40s/it]
 69%|██████▉   | 2240/3252 [1:32:43<40:25,  2.40s/it]
                                                     
{'loss': 0.3399, 'grad_norm': 0.21038581430912018, 'learning_rate': 4.670037791769772e-05, 'lm_loss': 0.015245793387293816, 'epoch': 2.76}

 69%|██████▉   | 2240/3252 [1:32:44<40:25,  2.40s/it]
 69%|██████▉   | 2241/3252 [1:32:46<41:33,  2.47s/it]
 69%|██████▉   | 2242/3252 [1:32:49<41:43,  2.48s/it]
 69%|██████▉   | 2243/3252 [1:32:51<41:15,  2.45s/it]
 69%|██████▉   | 2244/3252 [1:32:53<40:53,  2.43s/it]
 69%|██████▉   | 2245/3252 [1:32:56<41:23,  2.47s/it]
 69%|██████▉   | 2246/3252 [1:32:58<41:01,  2.45s/it]
 69%|██████▉   | 2247/3252 [1:33:01<40:43,  2.43s/it]
 69%|██████▉   | 2248/3252 [1:33:03<40:26,  2.42s/it]
 69%|██████▉   | 2249/3252 [1:33:05<40:16,  2.41s/it]
 69%|██████▉   | 2250/3252 [1:33:08<40:08,  2.40s/it]
                                                     
{'loss': 0.3386, 'grad_norm': 0.23383326828479767, 'learning_rate': 4.585971537970547e-05, 'lm_loss': 0.40948039293289185, 'epoch': 2.77}

 69%|██████▉   | 2250/3252 [1:33:08<40:08,  2.40s/it]
 69%|██████▉   | 2251/3252 [1:33:10<40:54,  2.45s/it]
 69%|██████▉   | 2252/3252 [1:33:13<40:33,  2.43s/it]
 69%|██████▉   | 2253/3252 [1:33:15<40:17,  2.42s/it]
 69%|██████▉   | 2254/3252 [1:33:18<40:06,  2.41s/it]
 69%|██████▉   | 2255/3252 [1:33:20<40:03,  2.41s/it]
 69%|██████▉   | 2256/3252 [1:33:22<39:56,  2.41s/it]
 69%|██████▉   | 2257/3252 [1:33:25<39:56,  2.41s/it]
 69%|██████▉   | 2258/3252 [1:33:27<39:52,  2.41s/it]
 69%|██████▉   | 2259/3252 [1:33:30<40:39,  2.46s/it]
 69%|██████▉   | 2260/3252 [1:33:32<40:19,  2.44s/it]
                                                     
{'loss': 0.3315, 'grad_norm': 0.29276248812675476, 'learning_rate': 4.502443113234688e-05, 'lm_loss': 0.01793058030307293, 'epoch': 2.78}

 69%|██████▉   | 2260/3252 [1:33:32<40:19,  2.44s/it]
 70%|██████▉   | 2261/3252 [1:33:35<40:39,  2.46s/it]
 70%|██████▉   | 2262/3252 [1:33:37<40:17,  2.44s/it]
 70%|██████▉   | 2263/3252 [1:33:39<40:03,  2.43s/it]
 70%|██████▉   | 2264/3252 [1:33:42<39:49,  2.42s/it]
 70%|██████▉   | 2265/3252 [1:33:44<39:41,  2.41s/it]
 70%|██████▉   | 2266/3252 [1:33:47<39:42,  2.42s/it]
 70%|██████▉   | 2267/3252 [1:33:49<39:39,  2.42s/it]
 70%|██████▉   | 2268/3252 [1:33:51<39:28,  2.41s/it]
 70%|██████▉   | 2269/3252 [1:33:54<39:21,  2.40s/it]
 70%|██████▉   | 2270/3252 [1:33:56<39:17,  2.40s/it]
                                                     
{'loss': 0.3229, 'grad_norm': 0.2475883513689041, 'learning_rate': 4.419460815267818e-05, 'lm_loss': 0.24858522415161133, 'epoch': 2.79}

 70%|██████▉   | 2270/3252 [1:33:56<39:17,  2.40s/it]
 70%|██████▉   | 2271/3252 [1:33:59<39:43,  2.43s/it]
 70%|██████▉   | 2272/3252 [1:34:01<39:29,  2.42s/it]
 70%|██████▉   | 2273/3252 [1:34:04<39:20,  2.41s/it]
 70%|██████▉   | 2274/3252 [1:34:06<39:19,  2.41s/it]
 70%|██████▉   | 2275/3252 [1:34:08<39:12,  2.41s/it]
 70%|██████▉   | 2276/3252 [1:34:11<39:02,  2.40s/it]
 70%|███████   | 2277/3252 [1:34:13<38:58,  2.40s/it]
 70%|███████   | 2278/3252 [1:34:16<39:02,  2.41s/it]
 70%|███████   | 2279/3252 [1:34:18<39:23,  2.43s/it]
 70%|███████   | 2280/3252 [1:34:21<40:54,  2.52s/it]
                                                     
{'loss': 0.3213, 'grad_norm': 0.20107312500476837, 'learning_rate': 4.33703288752339e-05, 'lm_loss': 0.3717556595802307, 'epoch': 2.8}

 70%|███████   | 2280/3252 [1:34:21<40:54,  2.52s/it]
 70%|███████   | 2281/3252 [1:34:23<40:21,  2.49s/it]
 70%|███████   | 2282/3252 [1:34:26<39:48,  2.46s/it]
 70%|███████   | 2283/3252 [1:34:28<39:27,  2.44s/it]
 70%|███████   | 2284/3252 [1:34:30<39:13,  2.43s/it]
 70%|███████   | 2285/3252 [1:34:33<39:06,  2.43s/it]
 70%|███████   | 2286/3252 [1:34:35<38:55,  2.42s/it]
 70%|███████   | 2287/3252 [1:34:38<38:47,  2.41s/it]
 70%|███████   | 2288/3252 [1:34:40<38:41,  2.41s/it]
 70%|███████   | 2289/3252 [1:34:42<38:34,  2.40s/it]
 70%|███████   | 2290/3252 [1:34:45<38:31,  2.40s/it]
                                                     
{'loss': 0.3319, 'grad_norm': 0.18761609494686127, 'learning_rate': 4.25516751838377e-05, 'lm_loss': 0.37396931648254395, 'epoch': 2.82}

 70%|███████   | 2290/3252 [1:34:45<38:31,  2.40s/it]
 70%|███████   | 2291/3252 [1:34:47<38:44,  2.42s/it]
 70%|███████   | 2292/3252 [1:34:50<38:36,  2.41s/it]
 71%|███████   | 2293/3252 [1:34:52<38:28,  2.41s/it]
 71%|███████   | 2294/3252 [1:34:54<38:23,  2.40s/it]
 71%|███████   | 2295/3252 [1:34:57<38:16,  2.40s/it]
 71%|███████   | 2296/3252 [1:34:59<38:11,  2.40s/it]
 71%|███████   | 2297/3252 [1:35:02<38:07,  2.40s/it]
 71%|███████   | 2298/3252 [1:35:04<38:26,  2.42s/it]
 71%|███████   | 2299/3252 [1:35:07<38:51,  2.45s/it]
 71%|███████   | 2300/3252 [1:35:09<38:32,  2.43s/it]
                                                     
{'loss': 0.3289, 'grad_norm': 0.20234645903110504, 'learning_rate': 4.1738728403467974e-05, 'lm_loss': 0.0042279609479010105, 'epoch': 2.83}

 71%|███████   | 2300/3252 [1:35:09<38:32,  2.43s/it]
 71%|███████   | 2301/3252 [1:35:12<39:15,  2.48s/it]
 71%|███████   | 2302/3252 [1:35:14<38:49,  2.45s/it]
 71%|███████   | 2303/3252 [1:35:16<38:30,  2.43s/it]
 71%|███████   | 2304/3252 [1:35:19<38:21,  2.43s/it]
 71%|███████   | 2305/3252 [1:35:21<38:12,  2.42s/it]
 71%|███████   | 2306/3252 [1:35:24<38:05,  2.42s/it]
 71%|███████   | 2307/3252 [1:35:26<38:06,  2.42s/it]
 71%|███████   | 2308/3252 [1:35:28<37:57,  2.41s/it]
 71%|███████   | 2309/3252 [1:35:31<37:50,  2.41s/it]
 71%|███████   | 2310/3252 [1:35:33<37:44,  2.40s/it]
                                                     
{'loss': 0.3195, 'grad_norm': 0.23914796113967896, 'learning_rate': 4.093156929217929e-05, 'lm_loss': 0.31252244114875793, 'epoch': 2.84}

 71%|███████   | 2310/3252 [1:35:33<37:44,  2.40s/it]
 71%|███████   | 2311/3252 [1:35:36<37:51,  2.41s/it]
 71%|███████   | 2312/3252 [1:35:38<37:42,  2.41s/it]
 71%|███████   | 2313/3252 [1:35:40<37:36,  2.40s/it]
 71%|███████   | 2314/3252 [1:35:43<37:32,  2.40s/it]
 71%|███████   | 2315/3252 [1:35:45<37:26,  2.40s/it]
 71%|███████   | 2316/3252 [1:35:48<38:04,  2.44s/it]
 71%|███████   | 2317/3252 [1:35:50<37:52,  2.43s/it]
 71%|███████▏  | 2318/3252 [1:35:53<37:41,  2.42s/it]
 71%|███████▏  | 2319/3252 [1:35:55<38:17,  2.46s/it]
 71%|███████▏  | 2320/3252 [1:35:58<37:55,  2.44s/it]
                                                     
{'loss': 0.3372, 'grad_norm': 0.2221924364566803, 'learning_rate': 4.013027803307951e-05, 'lm_loss': 0.49251559376716614, 'epoch': 2.85}

 71%|███████▏  | 2320/3252 [1:35:58<37:55,  2.44s/it]
 71%|███████▏  | 2321/3252 [1:36:00<38:01,  2.45s/it]
 71%|███████▏  | 2322/3252 [1:36:02<37:44,  2.43s/it]
 71%|███████▏  | 2323/3252 [1:36:05<37:32,  2.42s/it]
 71%|███████▏  | 2324/3252 [1:36:07<37:21,  2.42s/it]
 71%|███████▏  | 2325/3252 [1:36:10<37:12,  2.41s/it]
 72%|███████▏  | 2326/3252 [1:36:12<37:08,  2.41s/it]
 72%|███████▏  | 2327/3252 [1:36:14<37:04,  2.40s/it]
 72%|███████▏  | 2328/3252 [1:36:17<36:59,  2.40s/it]
 72%|███████▏  | 2329/3252 [1:36:19<36:56,  2.40s/it]
 72%|███████▏  | 2330/3252 [1:36:22<36:51,  2.40s/it]
                                                     
{'loss': 0.3172, 'grad_norm': 0.20624208450317383, 'learning_rate': 3.933493422636483e-05, 'lm_loss': 0.37159696221351624, 'epoch': 2.87}

 72%|███████▏  | 2330/3252 [1:36:22<36:51,  2.40s/it]
 72%|███████▏  | 2331/3252 [1:36:24<37:18,  2.43s/it]
 72%|███████▏  | 2332/3252 [1:36:26<37:04,  2.42s/it]
 72%|███████▏  | 2333/3252 [1:36:29<37:26,  2.44s/it]
 72%|███████▏  | 2334/3252 [1:36:31<37:09,  2.43s/it]
 72%|███████▏  | 2335/3252 [1:36:34<36:57,  2.42s/it]
 72%|███████▏  | 2336/3252 [1:36:36<36:46,  2.41s/it]
 72%|███████▏  | 2337/3252 [1:36:39<36:40,  2.41s/it]
 72%|███████▏  | 2338/3252 [1:36:41<36:36,  2.40s/it]
 72%|███████▏  | 2339/3252 [1:36:43<36:31,  2.40s/it]
 72%|███████▏  | 2340/3252 [1:36:46<36:26,  2.40s/it]
                                                     
{'loss': 0.3312, 'grad_norm': 0.22666673362255096, 'learning_rate': 3.854561688141205e-05, 'lm_loss': 0.3433045446872711, 'epoch': 2.88}

 72%|███████▏  | 2340/3252 [1:36:46<36:26,  2.40s/it]
 72%|███████▏  | 2341/3252 [1:36:48<36:33,  2.41s/it]
 72%|███████▏  | 2342/3252 [1:36:51<36:28,  2.41s/it]
 72%|███████▏  | 2343/3252 [1:36:53<36:22,  2.40s/it]
 72%|███████▏  | 2344/3252 [1:36:55<36:16,  2.40s/it]
 72%|███████▏  | 2345/3252 [1:36:58<36:16,  2.40s/it]
 72%|███████▏  | 2346/3252 [1:37:00<36:15,  2.40s/it]
 72%|███████▏  | 2347/3252 [1:37:03<36:10,  2.40s/it]
 72%|███████▏  | 2348/3252 [1:37:05<36:08,  2.40s/it]
 72%|███████▏  | 2349/3252 [1:37:07<36:05,  2.40s/it]
 72%|███████▏  | 2350/3252 [1:37:10<36:02,  2.40s/it]
                                                     
{'loss': 0.3085, 'grad_norm': 0.22225916385650635, 'learning_rate': 3.776240440892973e-05, 'lm_loss': 0.5552940964698792, 'epoch': 2.89}

 72%|███████▏  | 2350/3252 [1:37:10<36:02,  2.40s/it]
 72%|███████▏  | 2351/3252 [1:37:12<36:37,  2.44s/it]
 72%|███████▏  | 2352/3252 [1:37:15<36:21,  2.42s/it]
 72%|███████▏  | 2353/3252 [1:37:17<36:25,  2.43s/it]
 72%|███████▏  | 2354/3252 [1:37:20<38:09,  2.55s/it]
 72%|███████▏  | 2355/3252 [1:37:22<37:24,  2.50s/it]
 72%|███████▏  | 2356/3252 [1:37:25<36:53,  2.47s/it]
 72%|███████▏  | 2357/3252 [1:37:27<36:31,  2.45s/it]
 73%|███████▎  | 2358/3252 [1:37:29<36:15,  2.43s/it]
 73%|███████▎  | 2359/3252 [1:37:32<36:02,  2.42s/it]
 73%|███████▎  | 2360/3252 [1:37:34<35:55,  2.42s/it]
                                                     
{'loss': 0.326, 'grad_norm': 0.23195017874240875, 'learning_rate': 3.6985374613169165e-05, 'lm_loss': 0.4995444416999817, 'epoch': 2.9}

 73%|███████▎  | 2360/3252 [1:37:35<35:55,  2.42s/it]
 73%|███████▎  | 2361/3252 [1:37:37<36:39,  2.47s/it]
 73%|███████▎  | 2362/3252 [1:37:39<36:20,  2.45s/it]
 73%|███████▎  | 2363/3252 [1:37:42<36:02,  2.43s/it]
 73%|███████▎  | 2364/3252 [1:37:44<35:51,  2.42s/it]
 73%|███████▎  | 2365/3252 [1:37:46<35:41,  2.41s/it]
 73%|███████▎  | 2366/3252 [1:37:49<35:35,  2.41s/it]
 73%|███████▎  | 2367/3252 [1:37:51<35:30,  2.41s/it]
 73%|███████▎  | 2368/3252 [1:37:54<35:25,  2.40s/it]
 73%|███████▎  | 2369/3252 [1:37:56<35:20,  2.40s/it]
 73%|███████▎  | 2370/3252 [1:37:58<35:18,  2.40s/it]
                                                     
{'loss': 0.3231, 'grad_norm': 0.21929782629013062, 'learning_rate': 3.621460468419498e-05, 'lm_loss': 0.013740750961005688, 'epoch': 2.92}

 73%|███████▎  | 2370/3252 [1:37:59<35:18,  2.40s/it]
 73%|███████▎  | 2371/3252 [1:38:01<35:39,  2.43s/it]
 73%|███████▎  | 2372/3252 [1:38:04<36:15,  2.47s/it]
 73%|███████▎  | 2373/3252 [1:38:06<36:20,  2.48s/it]
 73%|███████▎  | 2374/3252 [1:38:08<35:57,  2.46s/it]
 73%|███████▎  | 2375/3252 [1:38:11<35:37,  2.44s/it]
 73%|███████▎  | 2376/3252 [1:38:13<35:23,  2.42s/it]
 73%|███████▎  | 2377/3252 [1:38:16<35:29,  2.43s/it]
 73%|███████▎  | 2378/3252 [1:38:18<35:19,  2.42s/it]
 73%|███████▎  | 2379/3252 [1:38:20<35:07,  2.41s/it]
 73%|███████▎  | 2380/3252 [1:38:23<35:03,  2.41s/it]
                                                     
{'loss': 0.3272, 'grad_norm': 0.2427763193845749, 'learning_rate': 3.5450171190217364e-05, 'lm_loss': 0.3741510212421417, 'epoch': 2.93}

 73%|███████▎  | 2380/3252 [1:38:23<35:03,  2.41s/it]
 73%|███████▎  | 2381/3252 [1:38:25<35:12,  2.43s/it]
 73%|███████▎  | 2382/3252 [1:38:28<35:01,  2.42s/it]
 73%|███████▎  | 2383/3252 [1:38:30<34:56,  2.41s/it]
 73%|███████▎  | 2384/3252 [1:38:33<34:50,  2.41s/it]
 73%|███████▎  | 2385/3252 [1:38:35<34:43,  2.40s/it]
 73%|███████▎  | 2386/3252 [1:38:37<34:38,  2.40s/it]
 73%|███████▎  | 2387/3252 [1:38:40<34:35,  2.40s/it]
 73%|███████▎  | 2388/3252 [1:38:42<35:53,  2.49s/it]
 73%|███████▎  | 2389/3252 [1:38:45<35:24,  2.46s/it]
 73%|███████▎  | 2390/3252 [1:38:47<35:29,  2.47s/it]
                                                     
{'loss': 0.3182, 'grad_norm': 0.22147271037101746, 'learning_rate': 3.4692150069985565e-05, 'lm_loss': 0.6816649436950684, 'epoch': 2.94}

 73%|███████▎  | 2390/3252 [1:38:47<35:29,  2.47s/it]
 74%|███████▎  | 2391/3252 [1:38:50<35:11,  2.45s/it]
 74%|███████▎  | 2392/3252 [1:38:52<34:56,  2.44s/it]
 74%|███████▎  | 2393/3252 [1:38:55<35:20,  2.47s/it]
 74%|███████▎  | 2394/3252 [1:38:57<34:59,  2.45s/it]
 74%|███████▎  | 2395/3252 [1:38:59<34:48,  2.44s/it]
 74%|███████▎  | 2396/3252 [1:39:02<34:34,  2.42s/it]
 74%|███████▎  | 2397/3252 [1:39:04<34:28,  2.42s/it]
 74%|███████▎  | 2398/3252 [1:39:07<34:20,  2.41s/it]
 74%|███████▍  | 2399/3252 [1:39:09<34:13,  2.41s/it]
 74%|███████▍  | 2400/3252 [1:39:11<34:07,  2.40s/it]
                                                     
{'loss': 0.3174, 'grad_norm': 0.205271914601326, 'learning_rate': 3.394061662524432e-05, 'lm_loss': 0.1729678064584732, 'epoch': 2.95}

 74%|███████▍  | 2400/3252 [1:39:12<34:07,  2.40s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.17it/s]
                                                     

                                             
{'eval_sthsthv2/narration_val_loss': 0.797164797782898, 'eval_sthsthv2/narration_val_runtime': 3.5435, 'eval_sthsthv2/narration_val_samples_per_second': 188.512, 'eval_sthsthv2/narration_val_steps_per_second': 0.564, 'epoch': 2.95}

 74%|███████▍  | 2400/3252 [1:39:15<34:07,  2.40s/it]

100%|██████████| 2/2 [00:00<00:00,  2.17it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.44it/s]
                                                     

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6359807252883911, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.2016, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 184.909, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.625, 'epoch': 2.95}

 74%|███████▍  | 2400/3252 [1:39:18<34:07,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.44it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.78it/s]
                                                     

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.5870580673217773, 'eval_holoassist/dialog-klg-sum_val_runtime': 3.1463, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 225.344, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.636, 'epoch': 2.95}

 74%|███████▍  | 2400/3252 [1:39:22<34:07,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.78it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                     

                                     
{'eval_ego4d/narration_val_loss': 0.512168288230896, 'eval_ego4d/narration_val_runtime': 2.1296, 'eval_ego4d/narration_val_samples_per_second': 104.714, 'eval_ego4d/narration_val_steps_per_second': 0.47, 'epoch': 2.95}

 74%|███████▍  | 2400/3252 [1:39:24<34:07,  2.40s/it]

100%|██████████| 1/1 [00:00<00:00,  2.45it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.44it/s]
                                                     

                                             
{'eval_ego4d/summary_val_loss': 1.135745882987976, 'eval_ego4d/summary_val_runtime': 2.9825, 'eval_ego4d/summary_val_samples_per_second': 189.101, 'eval_ego4d/summary_val_steps_per_second': 0.671, 'epoch': 2.95}

 74%|███████▍  | 2400/3252 [1:39:27<34:07,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.44it/s]

                                             
 74%|███████▍  | 2401/3252 [1:39:29<1:39:51,  7.04s/it]
 74%|███████▍  | 2402/3252 [1:39:32<1:19:58,  5.65s/it]
 74%|███████▍  | 2403/3252 [1:39:34<1:06:04,  4.67s/it]
 74%|███████▍  | 2404/3252 [1:39:36<56:20,  3.99s/it]  
 74%|███████▍  | 2405/3252 [1:39:39<49:30,  3.51s/it]
 74%|███████▍  | 2406/3252 [1:39:41<44:44,  3.17s/it]
 74%|███████▍  | 2407/3252 [1:39:44<41:52,  2.97s/it]
 74%|███████▍  | 2408/3252 [1:39:46<39:23,  2.80s/it]
 74%|███████▍  | 2409/3252 [1:39:49<37:39,  2.68s/it]
 74%|███████▍  | 2410/3252 [1:39:51<36:25,  2.60s/it]
                                                     
{'loss': 0.3118, 'grad_norm': 0.2207953780889511, 'learning_rate': 3.319564551325331e-05, 'lm_loss': 0.5619734525680542, 'epoch': 2.96}

 74%|███████▍  | 2410/3252 [1:39:51<36:25,  2.60s/it]
 74%|███████▍  | 2411/3252 [1:39:54<36:32,  2.61s/it]
 74%|███████▍  | 2412/3252 [1:39:56<35:38,  2.55s/it]
 74%|███████▍  | 2413/3252 [1:39:58<34:58,  2.50s/it]
 74%|███████▍  | 2414/3252 [1:40:01<34:30,  2.47s/it]
 74%|███████▍  | 2415/3252 [1:40:03<34:09,  2.45s/it]
 74%|███████▍  | 2416/3252 [1:40:06<33:51,  2.43s/it]
 74%|███████▍  | 2417/3252 [1:40:08<33:38,  2.42s/it]
 74%|███████▍  | 2418/3252 [1:40:10<33:28,  2.41s/it]
 74%|███████▍  | 2419/3252 [1:40:13<33:20,  2.40s/it]
 74%|███████▍  | 2420/3252 [1:40:15<33:14,  2.40s/it]
                                                     
{'loss': 0.3254, 'grad_norm': 0.20841309428215027, 'learning_rate': 3.245731073937068e-05, 'lm_loss': 0.012085404247045517, 'epoch': 2.98}

 74%|███████▍  | 2420/3252 [1:40:15<33:14,  2.40s/it]
 74%|███████▍  | 2421/3252 [1:40:18<33:38,  2.43s/it]
 74%|███████▍  | 2422/3252 [1:40:20<33:25,  2.42s/it]
 75%|███████▍  | 2423/3252 [1:40:22<33:15,  2.41s/it]
 75%|███████▍  | 2424/3252 [1:40:25<33:16,  2.41s/it]
 75%|███████▍  | 2425/3252 [1:40:27<33:09,  2.41s/it]
 75%|███████▍  | 2426/3252 [1:40:30<33:04,  2.40s/it]
 75%|███████▍  | 2427/3252 [1:40:32<32:58,  2.40s/it]
 75%|███████▍  | 2428/3252 [1:40:34<32:59,  2.40s/it]
 75%|███████▍  | 2429/3252 [1:40:37<32:55,  2.40s/it]
 75%|███████▍  | 2430/3252 [1:40:39<32:50,  2.40s/it]
                                                     
{'loss': 0.3196, 'grad_norm': 0.22362670302391052, 'learning_rate': 3.1725685649701354e-05, 'lm_loss': 0.48738574981689453, 'epoch': 2.99}

 75%|███████▍  | 2430/3252 [1:40:39<32:50,  2.40s/it]
 75%|███████▍  | 2431/3252 [1:40:42<33:39,  2.46s/it]
 75%|███████▍  | 2432/3252 [1:40:44<33:21,  2.44s/it]
 75%|███████▍  | 2433/3252 [1:40:47<33:06,  2.43s/it]
 75%|███████▍  | 2434/3252 [1:40:49<33:04,  2.43s/it]
 75%|███████▍  | 2435/3252 [1:40:51<32:56,  2.42s/it]
 75%|███████▍  | 2436/3252 [1:40:54<32:47,  2.41s/it]
 75%|███████▍  | 2437/3252 [1:40:56<32:39,  2.40s/it]
 75%|███████▍  | 2438/3252 [1:40:59<32:34,  2.40s/it]
 75%|███████▌  | 2439/3252 [1:41:01<33:24,  2.47s/it]
 75%|███████▌  | 2440/3252 [1:41:07<46:10,  3.41s/it]
                                                     
{'loss': 0.3116, 'grad_norm': 0.5974604487419128, 'learning_rate': 3.1000842923810946e-05, 'lm_loss': 0.521206796169281, 'epoch': 3.0}

 75%|███████▌  | 2440/3252 [1:41:07<46:10,  3.41s/it]
 75%|███████▌  | 2441/3252 [1:41:09<42:27,  3.14s/it]
 75%|███████▌  | 2442/3252 [1:41:12<39:21,  2.92s/it]
 75%|███████▌  | 2443/3252 [1:41:14<37:10,  2.76s/it]
 75%|███████▌  | 2444/3252 [1:41:17<36:42,  2.73s/it]
 75%|███████▌  | 2445/3252 [1:41:19<35:34,  2.64s/it]
 75%|███████▌  | 2446/3252 [1:41:22<34:52,  2.60s/it]
 75%|███████▌  | 2447/3252 [1:41:24<33:59,  2.53s/it]
 75%|███████▌  | 2448/3252 [1:41:27<33:25,  2.49s/it]
 75%|███████▌  | 2449/3252 [1:41:29<33:01,  2.47s/it]
 75%|███████▌  | 2450/3252 [1:41:31<32:43,  2.45s/it]
                                                     
{'loss': 0.2913, 'grad_norm': 0.24171565473079681, 'learning_rate': 3.0282854567505737e-05, 'lm_loss': 0.35258013010025024, 'epoch': 3.01}

 75%|███████▌  | 2450/3252 [1:41:31<32:43,  2.45s/it]
 75%|███████▌  | 2451/3252 [1:41:34<32:40,  2.45s/it]
 75%|███████▌  | 2452/3252 [1:41:36<32:25,  2.43s/it]
 75%|███████▌  | 2453/3252 [1:41:39<32:20,  2.43s/it]
 75%|███████▌  | 2454/3252 [1:41:41<32:10,  2.42s/it]
 75%|███████▌  | 2455/3252 [1:41:43<31:59,  2.41s/it]
 76%|███████▌  | 2456/3252 [1:41:46<31:56,  2.41s/it]
 76%|███████▌  | 2457/3252 [1:41:48<31:48,  2.40s/it]
 76%|███████▌  | 2458/3252 [1:41:51<31:41,  2.39s/it]
 76%|███████▌  | 2459/3252 [1:41:53<31:40,  2.40s/it]
 76%|███████▌  | 2460/3252 [1:41:55<31:36,  2.39s/it]
                                                     
{'loss': 0.2978, 'grad_norm': 0.24949206411838531, 'learning_rate': 2.9571791905679468e-05, 'lm_loss': 0.5585914850234985, 'epoch': 3.03}

 76%|███████▌  | 2460/3252 [1:41:55<31:36,  2.39s/it]
 76%|███████▌  | 2461/3252 [1:41:58<31:36,  2.40s/it]
 76%|███████▌  | 2462/3252 [1:42:00<31:32,  2.40s/it]
 76%|███████▌  | 2463/3252 [1:42:03<31:38,  2.41s/it]
 76%|███████▌  | 2464/3252 [1:42:05<32:32,  2.48s/it]
 76%|███████▌  | 2465/3252 [1:42:08<32:37,  2.49s/it]
 76%|███████▌  | 2466/3252 [1:42:10<32:15,  2.46s/it]
 76%|███████▌  | 2467/3252 [1:42:13<32:37,  2.49s/it]
 76%|███████▌  | 2468/3252 [1:42:15<32:12,  2.46s/it]
 76%|███████▌  | 2469/3252 [1:42:17<31:52,  2.44s/it]
 76%|███████▌  | 2470/3252 [1:42:20<31:40,  2.43s/it]
                                                     
{'loss': 0.2907, 'grad_norm': 0.25244140625, 'learning_rate': 2.8867725575228165e-05, 'lm_loss': 0.4139080047607422, 'epoch': 3.04}

 76%|███████▌  | 2470/3252 [1:42:20<31:40,  2.43s/it]
 76%|███████▌  | 2471/3252 [1:42:23<32:28,  2.50s/it]
 76%|███████▌  | 2472/3252 [1:42:25<32:04,  2.47s/it]
 76%|███████▌  | 2473/3252 [1:42:27<31:45,  2.45s/it]
 76%|███████▌  | 2474/3252 [1:42:30<31:31,  2.43s/it]
 76%|███████▌  | 2475/3252 [1:42:32<31:20,  2.42s/it]
 76%|███████▌  | 2476/3252 [1:42:35<31:20,  2.42s/it]
 76%|███████▌  | 2477/3252 [1:42:37<31:12,  2.42s/it]
 76%|███████▌  | 2478/3252 [1:42:39<31:04,  2.41s/it]
 76%|███████▌  | 2479/3252 [1:42:42<30:57,  2.40s/it]
 76%|███████▋  | 2480/3252 [1:42:44<30:59,  2.41s/it]
                                                     
{'loss': 0.289, 'grad_norm': 0.2522548735141754, 'learning_rate': 2.817072551803298e-05, 'lm_loss': 0.30655616521835327, 'epoch': 3.05}

 76%|███████▋  | 2480/3252 [1:42:44<30:59,  2.41s/it]
 76%|███████▋  | 2481/3252 [1:42:47<31:35,  2.46s/it]
 76%|███████▋  | 2482/3252 [1:42:49<31:17,  2.44s/it]
 76%|███████▋  | 2483/3252 [1:42:51<31:04,  2.42s/it]
 76%|███████▋  | 2484/3252 [1:42:54<30:52,  2.41s/it]
 76%|███████▋  | 2485/3252 [1:42:56<31:13,  2.44s/it]
 76%|███████▋  | 2486/3252 [1:42:59<31:00,  2.43s/it]
 76%|███████▋  | 2487/3252 [1:43:01<30:52,  2.42s/it]
 77%|███████▋  | 2488/3252 [1:43:04<30:43,  2.41s/it]
 77%|███████▋  | 2489/3252 [1:43:06<30:43,  2.42s/it]
 77%|███████▋  | 2490/3252 [1:43:08<30:38,  2.41s/it]
                                                     
{'loss': 0.2755, 'grad_norm': 0.20454613864421844, 'learning_rate': 2.74808609740121e-05, 'lm_loss': 0.000409410975407809, 'epoch': 3.06}

 77%|███████▋  | 2490/3252 [1:43:09<30:38,  2.41s/it]
 77%|███████▋  | 2491/3252 [1:43:11<31:06,  2.45s/it]
 77%|███████▋  | 2492/3252 [1:43:13<30:50,  2.44s/it]
 77%|███████▋  | 2493/3252 [1:43:16<30:39,  2.42s/it]
 77%|███████▋  | 2494/3252 [1:43:18<30:30,  2.41s/it]
 77%|███████▋  | 2495/3252 [1:43:21<30:22,  2.41s/it]
 77%|███████▋  | 2496/3252 [1:43:23<30:17,  2.40s/it]
 77%|███████▋  | 2497/3252 [1:43:25<30:13,  2.40s/it]
 77%|███████▋  | 2498/3252 [1:43:28<30:40,  2.44s/it]
 77%|███████▋  | 2499/3252 [1:43:30<30:30,  2.43s/it]
 77%|███████▋  | 2500/3252 [1:43:33<30:19,  2.42s/it]
                                                     
{'loss': 0.284, 'grad_norm': 0.25672852993011475, 'learning_rate': 2.679820047424253e-05, 'lm_loss': 0.26931536197662354, 'epoch': 3.08}

 77%|███████▋  | 2500/3252 [1:43:33<30:19,  2.42s/it]
 77%|███████▋  | 2501/3252 [1:43:35<30:48,  2.46s/it]
 77%|███████▋  | 2502/3252 [1:43:38<30:33,  2.45s/it]
 77%|███████▋  | 2503/3252 [1:43:40<30:21,  2.43s/it]
 77%|███████▋  | 2504/3252 [1:43:42<30:10,  2.42s/it]
 77%|███████▋  | 2505/3252 [1:43:45<30:01,  2.41s/it]
 77%|███████▋  | 2506/3252 [1:43:47<29:54,  2.41s/it]
 77%|███████▋  | 2507/3252 [1:43:50<29:50,  2.40s/it]
 77%|███████▋  | 2508/3252 [1:43:52<29:45,  2.40s/it]
 77%|███████▋  | 2509/3252 [1:43:54<29:40,  2.40s/it]
 77%|███████▋  | 2510/3252 [1:43:57<29:38,  2.40s/it]
                                                     
{'loss': 0.2877, 'grad_norm': 0.2318192720413208, 'learning_rate': 2.612281183415225e-05, 'lm_loss': 1.032917857170105, 'epoch': 3.09}

 77%|███████▋  | 2510/3252 [1:43:57<29:38,  2.40s/it]
 77%|███████▋  | 2511/3252 [1:43:59<30:01,  2.43s/it]
 77%|███████▋  | 2512/3252 [1:44:02<29:49,  2.42s/it]
 77%|███████▋  | 2513/3252 [1:44:04<29:41,  2.41s/it]
 77%|███████▋  | 2514/3252 [1:44:06<29:38,  2.41s/it]
 77%|███████▋  | 2515/3252 [1:44:09<29:32,  2.41s/it]
 77%|███████▋  | 2516/3252 [1:44:11<29:31,  2.41s/it]
 77%|███████▋  | 2517/3252 [1:44:14<29:26,  2.40s/it]
 77%|███████▋  | 2518/3252 [1:44:16<29:49,  2.44s/it]
 77%|███████▋  | 2519/3252 [1:44:19<29:38,  2.43s/it]
 77%|███████▋  | 2520/3252 [1:44:21<29:29,  2.42s/it]
                                                     
{'loss': 0.2849, 'grad_norm': 0.24927888810634613, 'learning_rate': 2.545476214678344e-05, 'lm_loss': 0.0024431466590613127, 'epoch': 3.1}

 77%|███████▋  | 2520/3252 [1:44:21<29:29,  2.42s/it]
 78%|███████▊  | 2521/3252 [1:44:24<30:07,  2.47s/it]
 78%|███████▊  | 2522/3252 [1:44:26<29:48,  2.45s/it]
 78%|███████▊  | 2523/3252 [1:44:28<29:40,  2.44s/it]
 78%|███████▊  | 2524/3252 [1:44:31<29:28,  2.43s/it]
 78%|███████▊  | 2525/3252 [1:44:33<29:17,  2.42s/it]
 78%|███████▊  | 2526/3252 [1:44:36<29:08,  2.41s/it]
 78%|███████▊  | 2527/3252 [1:44:38<29:02,  2.40s/it]
 78%|███████▊  | 2528/3252 [1:44:40<28:59,  2.40s/it]
 78%|███████▊  | 2529/3252 [1:44:43<28:55,  2.40s/it]
 78%|███████▊  | 2530/3252 [1:44:45<28:51,  2.40s/it]
                                                     
{'loss': 0.2877, 'grad_norm': 0.24561016261577606, 'learning_rate': 2.4794117776127335e-05, 'lm_loss': 0.42003723978996277, 'epoch': 3.11}

 78%|███████▊  | 2530/3252 [1:44:45<28:51,  2.40s/it]
 78%|███████▊  | 2531/3252 [1:44:48<28:55,  2.41s/it]
 78%|███████▊  | 2532/3252 [1:44:50<28:49,  2.40s/it]
 78%|███████▊  | 2533/3252 [1:44:52<28:47,  2.40s/it]
 78%|███████▊  | 2534/3252 [1:44:55<28:46,  2.40s/it]
 78%|███████▊  | 2535/3252 [1:44:57<28:42,  2.40s/it]
 78%|███████▊  | 2536/3252 [1:45:00<28:36,  2.40s/it]
 78%|███████▊  | 2537/3252 [1:45:02<28:33,  2.40s/it]
 78%|███████▊  | 2538/3252 [1:45:05<29:22,  2.47s/it]
 78%|███████▊  | 2539/3252 [1:45:07<29:24,  2.47s/it]
 78%|███████▊  | 2540/3252 [1:45:10<29:04,  2.45s/it]
                                                     
{'loss': 0.2892, 'grad_norm': 0.24293185770511627, 'learning_rate': 2.4140944350531712e-05, 'lm_loss': 0.32725948095321655, 'epoch': 3.12}

 78%|███████▊  | 2540/3252 [1:45:10<29:04,  2.45s/it]
 78%|███████▊  | 2541/3252 [1:45:12<29:31,  2.49s/it]
 78%|███████▊  | 2542/3252 [1:45:14<29:11,  2.47s/it]
 78%|███████▊  | 2543/3252 [1:45:17<28:53,  2.45s/it]
 78%|███████▊  | 2544/3252 [1:45:19<28:40,  2.43s/it]
 78%|███████▊  | 2545/3252 [1:45:22<28:37,  2.43s/it]
 78%|███████▊  | 2546/3252 [1:45:24<28:26,  2.42s/it]
 78%|███████▊  | 2547/3252 [1:45:27<28:20,  2.41s/it]
 78%|███████▊  | 2548/3252 [1:45:29<28:15,  2.41s/it]
 78%|███████▊  | 2549/3252 [1:45:31<28:12,  2.41s/it]
 78%|███████▊  | 2550/3252 [1:45:34<28:06,  2.40s/it]
                                                     
{'loss': 0.2818, 'grad_norm': 0.24916379153728485, 'learning_rate': 2.3495306756181577e-05, 'lm_loss': 0.44974300265312195, 'epoch': 3.14}

 78%|███████▊  | 2550/3252 [1:45:34<28:06,  2.40s/it]
 78%|███████▊  | 2551/3252 [1:45:36<28:49,  2.47s/it]
 78%|███████▊  | 2552/3252 [1:45:39<28:36,  2.45s/it]
 79%|███████▊  | 2553/3252 [1:45:41<28:21,  2.43s/it]
 79%|███████▊  | 2554/3252 [1:45:44<28:11,  2.42s/it]
 79%|███████▊  | 2555/3252 [1:45:46<28:31,  2.46s/it]
 79%|███████▊  | 2556/3252 [1:45:48<28:16,  2.44s/it]
 79%|███████▊  | 2557/3252 [1:45:51<28:01,  2.42s/it]
 79%|███████▊  | 2558/3252 [1:45:53<27:54,  2.41s/it]
 79%|███████▊  | 2559/3252 [1:45:56<28:12,  2.44s/it]
 79%|███████▊  | 2560/3252 [1:45:58<28:01,  2.43s/it]
                                                     
{'loss': 0.2837, 'grad_norm': 0.24889081716537476, 'learning_rate': 2.285726913065298e-05, 'lm_loss': 0.0003567894746083766, 'epoch': 3.15}

 79%|███████▊  | 2560/3252 [1:45:58<28:01,  2.43s/it]
 79%|███████▉  | 2561/3252 [1:46:01<27:59,  2.43s/it]
 79%|███████▉  | 2562/3252 [1:46:03<27:52,  2.42s/it]
 79%|███████▉  | 2563/3252 [1:46:05<27:43,  2.41s/it]
 79%|███████▉  | 2564/3252 [1:46:08<27:37,  2.41s/it]
 79%|███████▉  | 2565/3252 [1:46:10<27:32,  2.41s/it]
 79%|███████▉  | 2566/3252 [1:46:13<27:28,  2.40s/it]
 79%|███████▉  | 2567/3252 [1:46:15<27:24,  2.40s/it]
 79%|███████▉  | 2568/3252 [1:46:17<27:19,  2.40s/it]
 79%|███████▉  | 2569/3252 [1:46:20<27:15,  2.40s/it]
 79%|███████▉  | 2570/3252 [1:46:22<27:12,  2.39s/it]
                                                     
{'loss': 0.2815, 'grad_norm': 0.23499910533428192, 'learning_rate': 2.222689485654187e-05, 'lm_loss': 0.4358348548412323, 'epoch': 3.16}

 79%|███████▉  | 2570/3252 [1:46:22<27:12,  2.39s/it]
 79%|███████▉  | 2571/3252 [1:46:25<27:23,  2.41s/it]
 79%|███████▉  | 2572/3252 [1:46:27<27:31,  2.43s/it]
 79%|███████▉  | 2573/3252 [1:46:29<27:20,  2.42s/it]
 79%|███████▉  | 2574/3252 [1:46:32<27:13,  2.41s/it]
 79%|███████▉  | 2575/3252 [1:46:34<27:08,  2.40s/it]
 79%|███████▉  | 2576/3252 [1:46:37<27:16,  2.42s/it]
 79%|███████▉  | 2577/3252 [1:46:39<27:11,  2.42s/it]
 79%|███████▉  | 2578/3252 [1:46:41<27:06,  2.41s/it]
 79%|███████▉  | 2579/3252 [1:46:44<26:59,  2.41s/it]
 79%|███████▉  | 2580/3252 [1:46:46<26:53,  2.40s/it]
                                                     
{'loss': 0.2776, 'grad_norm': 0.22046653926372528, 'learning_rate': 2.1604246555167638e-05, 'lm_loss': 0.5003271102905273, 'epoch': 3.17}

 79%|███████▉  | 2580/3252 [1:46:47<26:53,  2.40s/it]
 79%|███████▉  | 2581/3252 [1:46:49<27:34,  2.47s/it]
 79%|███████▉  | 2582/3252 [1:46:51<27:17,  2.44s/it]
 79%|███████▉  | 2583/3252 [1:46:54<27:06,  2.43s/it]
 79%|███████▉  | 2584/3252 [1:46:56<26:56,  2.42s/it]
 79%|███████▉  | 2585/3252 [1:46:58<26:48,  2.41s/it]
 80%|███████▉  | 2586/3252 [1:47:01<26:43,  2.41s/it]
 80%|███████▉  | 2587/3252 [1:47:03<26:38,  2.40s/it]
 80%|███████▉  | 2588/3252 [1:47:06<26:34,  2.40s/it]
 80%|███████▉  | 2589/3252 [1:47:08<26:32,  2.40s/it]
 80%|███████▉  | 2590/3252 [1:47:10<26:28,  2.40s/it]
                                                     
{'loss': 0.2769, 'grad_norm': 0.3285812437534332, 'learning_rate': 2.09893860803523e-05, 'lm_loss': 0.3734745383262634, 'epoch': 3.19}

 80%|███████▉  | 2590/3252 [1:47:11<26:28,  2.40s/it]
 80%|███████▉  | 2591/3252 [1:47:13<26:30,  2.41s/it]
 80%|███████▉  | 2592/3252 [1:47:15<26:48,  2.44s/it]
 80%|███████▉  | 2593/3252 [1:47:18<26:36,  2.42s/it]
 80%|███████▉  | 2594/3252 [1:47:20<26:29,  2.42s/it]
 80%|███████▉  | 2595/3252 [1:47:23<26:22,  2.41s/it]
 80%|███████▉  | 2596/3252 [1:47:25<26:15,  2.40s/it]
 80%|███████▉  | 2597/3252 [1:47:27<26:11,  2.40s/it]
 80%|███████▉  | 2598/3252 [1:47:30<26:14,  2.41s/it]
 80%|███████▉  | 2599/3252 [1:47:32<26:10,  2.40s/it]
 80%|███████▉  | 2600/3252 [1:47:35<26:05,  2.40s/it]
                                                     
{'loss': 0.2904, 'grad_norm': 0.23421913385391235, 'learning_rate': 2.0382374512275893e-05, 'lm_loss': 0.5799974799156189, 'epoch': 3.2}

 80%|███████▉  | 2600/3252 [1:47:35<26:05,  2.40s/it]
 80%|███████▉  | 2601/3252 [1:47:37<26:35,  2.45s/it]
 80%|████████  | 2602/3252 [1:47:40<26:22,  2.43s/it]
 80%|████████  | 2603/3252 [1:47:42<26:12,  2.42s/it]
 80%|████████  | 2604/3252 [1:47:44<26:05,  2.42s/it]
 80%|████████  | 2605/3252 [1:47:47<25:59,  2.41s/it]
 80%|████████  | 2606/3252 [1:47:49<25:56,  2.41s/it]
 80%|████████  | 2607/3252 [1:47:52<25:48,  2.40s/it]
 80%|████████  | 2608/3252 [1:47:54<25:44,  2.40s/it]
 80%|████████  | 2609/3252 [1:47:56<25:41,  2.40s/it]
 80%|████████  | 2610/3252 [1:47:59<25:39,  2.40s/it]
                                                     
{'loss': 0.2903, 'grad_norm': 0.27281901240348816, 'learning_rate': 1.9783272151408804e-05, 'lm_loss': 0.018561435863375664, 'epoch': 3.21}

 80%|████████  | 2610/3252 [1:47:59<25:39,  2.40s/it]
 80%|████████  | 2611/3252 [1:48:01<25:42,  2.41s/it]
 80%|████████  | 2612/3252 [1:48:04<26:36,  2.49s/it]
 80%|████████  | 2613/3252 [1:48:06<26:29,  2.49s/it]
 80%|████████  | 2614/3252 [1:48:09<26:08,  2.46s/it]
 80%|████████  | 2615/3252 [1:48:11<26:27,  2.49s/it]
 80%|████████  | 2616/3252 [1:48:14<26:06,  2.46s/it]
 80%|████████  | 2617/3252 [1:48:16<25:50,  2.44s/it]
 81%|████████  | 2618/3252 [1:48:18<25:38,  2.43s/it]
 81%|████████  | 2619/3252 [1:48:21<25:28,  2.42s/it]
 81%|████████  | 2620/3252 [1:48:23<25:25,  2.41s/it]
                                                     
{'loss': 0.2745, 'grad_norm': 0.225446879863739, 'learning_rate': 1.9192138512521584e-05, 'lm_loss': 0.3000793755054474, 'epoch': 3.22}

 81%|████████  | 2620/3252 [1:48:23<25:25,  2.41s/it]
 81%|████████  | 2621/3252 [1:48:26<25:22,  2.41s/it]
 81%|████████  | 2622/3252 [1:48:28<25:16,  2.41s/it]
 81%|████████  | 2623/3252 [1:48:30<25:12,  2.40s/it]
 81%|████████  | 2624/3252 [1:48:33<25:07,  2.40s/it]
 81%|████████  | 2625/3252 [1:48:35<25:02,  2.40s/it]
 81%|████████  | 2626/3252 [1:48:38<25:01,  2.40s/it]
 81%|████████  | 2627/3252 [1:48:40<24:56,  2.40s/it]
 81%|████████  | 2628/3252 [1:48:42<24:56,  2.40s/it]
 81%|████████  | 2629/3252 [1:48:45<25:34,  2.46s/it]
 81%|████████  | 2630/3252 [1:48:47<25:17,  2.44s/it]
                                                     
{'loss': 0.276, 'grad_norm': 0.842960000038147, 'learning_rate': 1.8609032318772734e-05, 'lm_loss': 0.01048329845070839, 'epoch': 3.23}

 81%|████████  | 2630/3252 [1:48:48<25:17,  2.44s/it]
 81%|████████  | 2631/3252 [1:48:50<25:26,  2.46s/it]
 81%|████████  | 2632/3252 [1:48:52<25:14,  2.44s/it]
 81%|████████  | 2633/3252 [1:48:55<25:32,  2.48s/it]
 81%|████████  | 2634/3252 [1:48:57<25:15,  2.45s/it]
 81%|████████  | 2635/3252 [1:49:00<25:02,  2.43s/it]
 81%|████████  | 2636/3252 [1:49:02<24:54,  2.43s/it]
 81%|████████  | 2637/3252 [1:49:04<24:46,  2.42s/it]
 81%|████████  | 2638/3252 [1:49:07<24:40,  2.41s/it]
 81%|████████  | 2639/3252 [1:49:09<24:36,  2.41s/it]
 81%|████████  | 2640/3252 [1:49:12<24:33,  2.41s/it]
                                                     
{'loss': 0.2803, 'grad_norm': 0.25414907932281494, 'learning_rate': 1.803401149587508e-05, 'lm_loss': 0.28476935625076294, 'epoch': 3.25}

 81%|████████  | 2640/3252 [1:49:12<24:33,  2.41s/it]
 81%|████████  | 2641/3252 [1:49:14<24:34,  2.41s/it]
 81%|████████  | 2642/3252 [1:49:16<24:31,  2.41s/it]
 81%|████████▏ | 2643/3252 [1:49:19<24:26,  2.41s/it]
 81%|████████▏ | 2644/3252 [1:49:21<24:24,  2.41s/it]
 81%|████████▏ | 2645/3252 [1:49:24<24:18,  2.40s/it]
 81%|████████▏ | 2646/3252 [1:49:26<24:52,  2.46s/it]
 81%|████████▏ | 2647/3252 [1:49:29<24:37,  2.44s/it]
 81%|████████▏ | 2648/3252 [1:49:31<24:26,  2.43s/it]
 81%|████████▏ | 2649/3252 [1:49:33<24:17,  2.42s/it]
 81%|████████▏ | 2650/3252 [1:49:36<24:10,  2.41s/it]
                                                     
{'loss': 0.271, 'grad_norm': 0.2555672824382782, 'learning_rate': 1.7467133166341453e-05, 'lm_loss': 0.22742848098278046, 'epoch': 3.26}

 81%|████████▏ | 2650/3252 [1:49:36<24:10,  2.41s/it]
 82%|████████▏ | 2651/3252 [1:49:38<24:27,  2.44s/it]
 82%|████████▏ | 2652/3252 [1:49:41<24:16,  2.43s/it]
 82%|████████▏ | 2653/3252 [1:49:43<24:07,  2.42s/it]
 82%|████████▏ | 2654/3252 [1:49:46<24:03,  2.41s/it]
 82%|████████▏ | 2655/3252 [1:49:48<23:57,  2.41s/it]
 82%|████████▏ | 2656/3252 [1:49:50<23:52,  2.40s/it]
 82%|████████▏ | 2657/3252 [1:49:53<23:50,  2.40s/it]
 82%|████████▏ | 2658/3252 [1:49:55<23:45,  2.40s/it]
 82%|████████▏ | 2659/3252 [1:49:58<23:41,  2.40s/it]
 82%|████████▏ | 2660/3252 [1:50:00<23:40,  2.40s/it]
                                                     
{'loss': 0.2799, 'grad_norm': 0.25009992718696594, 'learning_rate': 1.690845364381034e-05, 'lm_loss': 0.0004972154856659472, 'epoch': 3.27}

 82%|████████▏ | 2660/3252 [1:50:00<23:40,  2.40s/it]
 82%|████████▏ | 2661/3252 [1:50:02<24:00,  2.44s/it]
 82%|████████▏ | 2662/3252 [1:50:05<23:50,  2.43s/it]
 82%|████████▏ | 2663/3252 [1:50:07<23:43,  2.42s/it]
 82%|████████▏ | 2664/3252 [1:50:10<23:38,  2.41s/it]
 82%|████████▏ | 2665/3252 [1:50:12<23:33,  2.41s/it]
 82%|████████▏ | 2666/3252 [1:50:15<23:54,  2.45s/it]
 82%|████████▏ | 2667/3252 [1:50:17<23:49,  2.44s/it]
 82%|████████▏ | 2668/3252 [1:50:19<23:37,  2.43s/it]
 82%|████████▏ | 2669/3252 [1:50:22<23:29,  2.42s/it]
 82%|████████▏ | 2670/3252 [1:50:24<23:25,  2.41s/it]
                                                     
{'loss': 0.2722, 'grad_norm': 0.23934796452522278, 'learning_rate': 1.6358028427451377e-05, 'lm_loss': 0.4390437602996826, 'epoch': 3.28}

 82%|████████▏ | 2670/3252 [1:50:24<23:25,  2.41s/it]
 82%|████████▏ | 2671/3252 [1:50:27<23:25,  2.42s/it]
 82%|████████▏ | 2672/3252 [1:50:29<23:19,  2.41s/it]
 82%|████████▏ | 2673/3252 [1:50:31<23:13,  2.41s/it]
 82%|████████▏ | 2674/3252 [1:50:34<23:10,  2.41s/it]
 82%|████████▏ | 2675/3252 [1:50:36<23:05,  2.40s/it]
 82%|████████▏ | 2676/3252 [1:50:39<23:04,  2.40s/it]
 82%|████████▏ | 2677/3252 [1:50:41<23:01,  2.40s/it]
 82%|████████▏ | 2678/3252 [1:50:43<22:57,  2.40s/it]
 82%|████████▏ | 2679/3252 [1:50:46<22:53,  2.40s/it]
 82%|████████▏ | 2680/3252 [1:50:48<22:49,  2.40s/it]
                                                     
{'loss': 0.2756, 'grad_norm': 0.2333541363477707, 'learning_rate': 1.5815912196452265e-05, 'lm_loss': 0.4070286154747009, 'epoch': 3.3}

 82%|████████▏ | 2680/3252 [1:50:48<22:49,  2.40s/it]
 82%|████████▏ | 2681/3252 [1:50:51<23:09,  2.43s/it]
 82%|████████▏ | 2682/3252 [1:50:53<22:59,  2.42s/it]
 83%|████████▎ | 2683/3252 [1:50:56<22:52,  2.41s/it]
 83%|████████▎ | 2684/3252 [1:50:58<22:47,  2.41s/it]
 83%|████████▎ | 2685/3252 [1:51:00<22:43,  2.41s/it]
 83%|████████▎ | 2686/3252 [1:51:03<23:30,  2.49s/it]
 83%|████████▎ | 2687/3252 [1:51:06<23:26,  2.49s/it]
 83%|████████▎ | 2688/3252 [1:51:08<23:07,  2.46s/it]
 83%|████████▎ | 2689/3252 [1:51:10<23:24,  2.49s/it]
 83%|████████▎ | 2690/3252 [1:51:13<23:04,  2.46s/it]
                                                     
{'loss': 0.2788, 'grad_norm': 0.2377316951751709, 'learning_rate': 1.5282158804586933e-05, 'lm_loss': 0.7464975714683533, 'epoch': 3.31}

 83%|████████▎ | 2690/3252 [1:51:13<23:04,  2.46s/it]
 83%|████████▎ | 2691/3252 [1:51:15<23:21,  2.50s/it]
 83%|████████▎ | 2692/3252 [1:51:18<23:01,  2.47s/it]
 83%|████████▎ | 2693/3252 [1:51:20<22:47,  2.45s/it]
 83%|████████▎ | 2694/3252 [1:51:23<22:36,  2.43s/it]
 83%|████████▎ | 2695/3252 [1:51:25<22:28,  2.42s/it]
 83%|████████▎ | 2696/3252 [1:51:27<22:23,  2.42s/it]
 83%|████████▎ | 2697/3252 [1:51:30<22:23,  2.42s/it]
 83%|████████▎ | 2698/3252 [1:51:32<22:18,  2.42s/it]
 83%|████████▎ | 2699/3252 [1:51:35<22:12,  2.41s/it]
 83%|████████▎ | 2700/3252 [1:51:37<22:09,  2.41s/it]
                                                     
{'loss': 0.2817, 'grad_norm': 0.40853920578956604, 'learning_rate': 1.4756821274865695e-05, 'lm_loss': 0.34282809495925903, 'epoch': 3.32}

 83%|████████▎ | 2700/3252 [1:51:37<22:09,  2.41s/it]
 83%|████████▎ | 2701/3252 [1:51:40<22:08,  2.41s/it]
 83%|████████▎ | 2702/3252 [1:51:42<22:01,  2.40s/it]
 83%|████████▎ | 2703/3252 [1:51:44<22:22,  2.44s/it]
 83%|████████▎ | 2704/3252 [1:51:47<22:12,  2.43s/it]
 83%|████████▎ | 2705/3252 [1:51:49<22:03,  2.42s/it]
 83%|████████▎ | 2706/3252 [1:51:52<21:56,  2.41s/it]
 83%|████████▎ | 2707/3252 [1:51:54<22:09,  2.44s/it]
 83%|████████▎ | 2708/3252 [1:51:57<22:05,  2.44s/it]
 83%|████████▎ | 2709/3252 [1:51:59<21:57,  2.43s/it]
 83%|████████▎ | 2710/3252 [1:52:01<21:51,  2.42s/it]
                                                     
{'loss': 0.278, 'grad_norm': 0.24172692000865936, 'learning_rate': 1.4239951794267847e-05, 'lm_loss': 0.5130422711372375, 'epoch': 3.33}

 83%|████████▎ | 2710/3252 [1:52:02<21:51,  2.42s/it]
 83%|████████▎ | 2711/3252 [1:52:04<22:11,  2.46s/it]
 83%|████████▎ | 2712/3252 [1:52:06<21:57,  2.44s/it]
 83%|████████▎ | 2713/3252 [1:52:09<21:47,  2.43s/it]
 83%|████████▎ | 2714/3252 [1:52:11<21:41,  2.42s/it]
 83%|████████▎ | 2715/3252 [1:52:14<21:36,  2.41s/it]
 84%|████████▎ | 2716/3252 [1:52:16<21:31,  2.41s/it]
 84%|████████▎ | 2717/3252 [1:52:18<21:26,  2.40s/it]
 84%|████████▎ | 2718/3252 [1:52:21<21:21,  2.40s/it]
 84%|████████▎ | 2719/3252 [1:52:23<21:18,  2.40s/it]
 84%|████████▎ | 2720/3252 [1:52:26<21:30,  2.43s/it]
                                                     
{'loss': 0.2732, 'grad_norm': 0.2280878871679306, 'learning_rate': 1.3731601708557606e-05, 'lm_loss': 0.1481008380651474, 'epoch': 3.35}

 84%|████████▎ | 2720/3252 [1:52:26<21:30,  2.43s/it]
 84%|████████▎ | 2721/3252 [1:52:28<21:40,  2.45s/it]
 84%|████████▎ | 2722/3252 [1:52:30<21:28,  2.43s/it]
 84%|████████▎ | 2723/3252 [1:52:33<21:20,  2.42s/it]
 84%|████████▍ | 2724/3252 [1:52:35<21:14,  2.41s/it]
 84%|████████▍ | 2725/3252 [1:52:38<21:10,  2.41s/it]
 84%|████████▍ | 2726/3252 [1:52:40<21:04,  2.40s/it]
 84%|████████▍ | 2727/3252 [1:52:42<21:00,  2.40s/it]
 84%|████████▍ | 2728/3252 [1:52:45<20:55,  2.40s/it]
 84%|████████▍ | 2729/3252 [1:52:47<20:54,  2.40s/it]
 84%|████████▍ | 2730/3252 [1:52:50<20:55,  2.41s/it]
                                                     
{'loss': 0.2793, 'grad_norm': 0.23931239545345306, 'learning_rate': 1.323182151718324e-05, 'lm_loss': 0.34719547629356384, 'epoch': 3.36}

 84%|████████▍ | 2730/3252 [1:52:50<20:55,  2.41s/it]
 84%|████████▍ | 2731/3252 [1:52:52<21:36,  2.49s/it]
 84%|████████▍ | 2732/3252 [1:52:55<21:19,  2.46s/it]
 84%|████████▍ | 2733/3252 [1:52:57<21:06,  2.44s/it]
 84%|████████▍ | 2734/3252 [1:53:00<20:57,  2.43s/it]
 84%|████████▍ | 2735/3252 [1:53:02<20:51,  2.42s/it]
 84%|████████▍ | 2736/3252 [1:53:04<20:45,  2.41s/it]
 84%|████████▍ | 2737/3252 [1:53:07<20:39,  2.41s/it]
 84%|████████▍ | 2738/3252 [1:53:09<20:33,  2.40s/it]
 84%|████████▍ | 2739/3252 [1:53:12<20:32,  2.40s/it]
 84%|████████▍ | 2740/3252 [1:53:14<20:53,  2.45s/it]
                                                     
{'loss': 0.2658, 'grad_norm': 0.24372921884059906, 'learning_rate': 1.2740660868260633e-05, 'lm_loss': 0.23781433701515198, 'epoch': 3.37}

 84%|████████▍ | 2740/3252 [1:53:14<20:53,  2.45s/it]
 84%|████████▍ | 2741/3252 [1:53:17<21:19,  2.50s/it]
 84%|████████▍ | 2742/3252 [1:53:19<20:59,  2.47s/it]
 84%|████████▍ | 2743/3252 [1:53:21<20:45,  2.45s/it]
 84%|████████▍ | 2744/3252 [1:53:24<20:36,  2.43s/it]
 84%|████████▍ | 2745/3252 [1:53:26<20:26,  2.42s/it]
 84%|████████▍ | 2746/3252 [1:53:29<20:20,  2.41s/it]
 84%|████████▍ | 2747/3252 [1:53:31<20:15,  2.41s/it]
 85%|████████▍ | 2748/3252 [1:53:33<20:10,  2.40s/it]
 85%|████████▍ | 2749/3252 [1:53:36<20:06,  2.40s/it]
 85%|████████▍ | 2750/3252 [1:53:38<20:04,  2.40s/it]
                                                     
{'loss': 0.2769, 'grad_norm': 0.2538334131240845, 'learning_rate': 1.2258168553641058e-05, 'lm_loss': 0.002106061205267906, 'epoch': 3.38}

 85%|████████▍ | 2750/3252 [1:53:38<20:04,  2.40s/it]
 85%|████████▍ | 2751/3252 [1:53:41<20:19,  2.43s/it]
 85%|████████▍ | 2752/3252 [1:53:43<20:11,  2.42s/it]
 85%|████████▍ | 2753/3252 [1:53:46<20:06,  2.42s/it]
 85%|████████▍ | 2754/3252 [1:53:48<19:59,  2.41s/it]
 85%|████████▍ | 2755/3252 [1:53:50<19:56,  2.41s/it]
 85%|████████▍ | 2756/3252 [1:53:53<19:51,  2.40s/it]
 85%|████████▍ | 2757/3252 [1:53:55<19:48,  2.40s/it]
 85%|████████▍ | 2758/3252 [1:53:58<19:45,  2.40s/it]
 85%|████████▍ | 2759/3252 [1:54:00<19:40,  2.40s/it]
 85%|████████▍ | 2760/3252 [1:54:03<20:22,  2.48s/it]
                                                     
{'loss': 0.2762, 'grad_norm': 0.21485857665538788, 'learning_rate': 1.1784392504064379e-05, 'lm_loss': 0.0037351991049945354, 'epoch': 3.39}

 85%|████████▍ | 2760/3252 [1:54:03<20:22,  2.48s/it]
 85%|████████▍ | 2761/3252 [1:54:05<20:39,  2.53s/it]
 85%|████████▍ | 2762/3252 [1:54:08<20:17,  2.49s/it]
 85%|████████▍ | 2763/3252 [1:54:10<20:22,  2.50s/it]
 85%|████████▍ | 2764/3252 [1:54:13<20:05,  2.47s/it]
 85%|████████▌ | 2765/3252 [1:54:15<19:51,  2.45s/it]
 85%|████████▌ | 2766/3252 [1:54:17<19:41,  2.43s/it]
 85%|████████▌ | 2767/3252 [1:54:20<19:35,  2.42s/it]
 85%|████████▌ | 2768/3252 [1:54:22<19:28,  2.41s/it]
 85%|████████▌ | 2769/3252 [1:54:25<19:23,  2.41s/it]
 85%|████████▌ | 2770/3252 [1:54:27<19:18,  2.40s/it]
                                                     
{'loss': 0.2781, 'grad_norm': 0.2696593701839447, 'learning_rate': 1.13193797843975e-05, 'lm_loss': 0.008214179426431656, 'epoch': 3.41}

 85%|████████▌ | 2770/3252 [1:54:27<19:18,  2.40s/it]
 85%|████████▌ | 2771/3252 [1:54:29<19:25,  2.42s/it]
 85%|████████▌ | 2772/3252 [1:54:32<19:18,  2.41s/it]
 85%|████████▌ | 2773/3252 [1:54:34<19:14,  2.41s/it]
 85%|████████▌ | 2774/3252 [1:54:37<19:08,  2.40s/it]
 85%|████████▌ | 2775/3252 [1:54:39<19:04,  2.40s/it]
 85%|████████▌ | 2776/3252 [1:54:41<19:00,  2.40s/it]
 85%|████████▌ | 2777/3252 [1:54:44<19:31,  2.47s/it]
 85%|████████▌ | 2778/3252 [1:54:46<19:18,  2.44s/it]
 85%|████████▌ | 2779/3252 [1:54:49<19:09,  2.43s/it]
 85%|████████▌ | 2780/3252 [1:54:51<19:04,  2.43s/it]
                                                     
{'loss': 0.2659, 'grad_norm': 0.22649990022182465, 'learning_rate': 1.0863176588958956e-05, 'lm_loss': 0.34741172194480896, 'epoch': 3.42}

 85%|████████▌ | 2780/3252 [1:54:51<19:04,  2.43s/it]
 86%|████████▌ | 2781/3252 [1:54:54<19:17,  2.46s/it]
 86%|████████▌ | 2782/3252 [1:54:56<19:06,  2.44s/it]
 86%|████████▌ | 2783/3252 [1:54:59<18:59,  2.43s/it]
 86%|████████▌ | 2784/3252 [1:55:01<18:54,  2.42s/it]
 86%|████████▌ | 2785/3252 [1:55:03<19:00,  2.44s/it]
 86%|████████▌ | 2786/3252 [1:55:06<18:54,  2.44s/it]
 86%|████████▌ | 2787/3252 [1:55:08<18:46,  2.42s/it]
 86%|████████▌ | 2788/3252 [1:55:11<18:41,  2.42s/it]
 86%|████████▌ | 2789/3252 [1:55:13<18:35,  2.41s/it]
 86%|████████▌ | 2790/3252 [1:55:15<18:31,  2.41s/it]
                                                     
{'loss': 0.284, 'grad_norm': 0.24250134825706482, 'learning_rate': 1.0415828236930058e-05, 'lm_loss': 0.5389891266822815, 'epoch': 3.43}

 86%|████████▌ | 2790/3252 [1:55:16<18:31,  2.41s/it]
 86%|████████▌ | 2791/3252 [1:55:18<18:41,  2.43s/it]
 86%|████████▌ | 2792/3252 [1:55:20<18:32,  2.42s/it]
 86%|████████▌ | 2793/3252 [1:55:23<18:27,  2.41s/it]
 86%|████████▌ | 2794/3252 [1:55:25<18:39,  2.45s/it]
 86%|████████▌ | 2795/3252 [1:55:28<18:28,  2.43s/it]
 86%|████████▌ | 2796/3252 [1:55:30<18:22,  2.42s/it]
 86%|████████▌ | 2797/3252 [1:55:32<18:16,  2.41s/it]
 86%|████████▌ | 2798/3252 [1:55:35<18:11,  2.40s/it]
 86%|████████▌ | 2799/3252 [1:55:37<18:07,  2.40s/it]
 86%|████████▌ | 2800/3252 [1:55:40<18:03,  2.40s/it]
                                                     
{'loss': 0.274, 'grad_norm': 0.26676544547080994, 'learning_rate': 9.977379167852763e-06, 'lm_loss': 0.4435025155544281, 'epoch': 3.44}

 86%|████████▌ | 2800/3252 [1:55:40<18:03,  2.40s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.40it/s]
                                                     

                                             
{'eval_sthsthv2/narration_val_loss': 0.7912942171096802, 'eval_sthsthv2/narration_val_runtime': 3.4678, 'eval_sthsthv2/narration_val_samples_per_second': 192.628, 'eval_sthsthv2/narration_val_steps_per_second': 0.577, 'epoch': 3.44}

 86%|████████▌ | 2800/3252 [1:55:43<18:03,  2.40s/it]

100%|██████████| 2/2 [00:00<00:00,  2.40it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.67it/s]
                                                     

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6429004073143005, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.197, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 185.176, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.626, 'epoch': 3.44}

 86%|████████▌ | 2800/3252 [1:55:47<18:03,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.67it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.56it/s]
                                                     

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.5907538533210754, 'eval_holoassist/dialog-klg-sum_val_runtime': 2.9553, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 239.905, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.677, 'epoch': 3.44}

 86%|████████▌ | 2800/3252 [1:55:50<18:03,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.56it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                     

                                     
{'eval_ego4d/narration_val_loss': 0.5181609392166138, 'eval_ego4d/narration_val_runtime': 2.2456, 'eval_ego4d/narration_val_samples_per_second': 99.307, 'eval_ego4d/narration_val_steps_per_second': 0.445, 'epoch': 3.44}

 86%|████████▌ | 2800/3252 [1:55:52<18:03,  2.40s/it]

100%|██████████| 1/1 [00:00<00:00,  3.61it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.58it/s]
                                                     

                                             
{'eval_ego4d/summary_val_loss': 1.143816590309143, 'eval_ego4d/summary_val_runtime': 2.8533, 'eval_ego4d/summary_val_samples_per_second': 197.669, 'eval_ego4d/summary_val_steps_per_second': 0.701, 'epoch': 3.44}

 86%|████████▌ | 2800/3252 [1:55:55<18:03,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.58it/s]

                                             
 86%|████████▌ | 2801/3252 [1:55:57<52:23,  6.97s/it]
 86%|████████▌ | 2802/3252 [1:56:00<41:58,  5.60s/it]
 86%|████████▌ | 2803/3252 [1:56:02<34:41,  4.64s/it]
 86%|████████▌ | 2804/3252 [1:56:04<29:36,  3.96s/it]
 86%|████████▋ | 2805/3252 [1:56:07<26:01,  3.49s/it]
 86%|████████▋ | 2806/3252 [1:56:09<23:30,  3.16s/it]
 86%|████████▋ | 2807/3252 [1:56:12<21:45,  2.93s/it]
 86%|████████▋ | 2808/3252 [1:56:14<20:30,  2.77s/it]
 86%|████████▋ | 2809/3252 [1:56:16<19:39,  2.66s/it]
 86%|████████▋ | 2810/3252 [1:56:19<19:02,  2.58s/it]
                                                     
{'loss': 0.2637, 'grad_norm': 0.21658803522586823, 'learning_rate': 9.547872937215185e-06, 'lm_loss': 0.34525051712989807, 'epoch': 3.46}

 86%|████████▋ | 2810/3252 [1:56:19<19:02,  2.58s/it]
 86%|████████▋ | 2811/3252 [1:56:21<18:56,  2.58s/it]
 86%|████████▋ | 2812/3252 [1:56:24<18:30,  2.52s/it]
 87%|████████▋ | 2813/3252 [1:56:27<19:08,  2.62s/it]
 87%|████████▋ | 2814/3252 [1:56:29<18:37,  2.55s/it]
 87%|████████▋ | 2815/3252 [1:56:31<18:14,  2.50s/it]
 87%|████████▋ | 2816/3252 [1:56:34<18:06,  2.49s/it]
 87%|████████▋ | 2817/3252 [1:56:36<17:50,  2.46s/it]
 87%|████████▋ | 2818/3252 [1:56:39<17:38,  2.44s/it]
 87%|████████▋ | 2819/3252 [1:56:41<17:30,  2.43s/it]
 87%|████████▋ | 2820/3252 [1:56:43<17:23,  2.41s/it]
                                                     
{'loss': 0.275, 'grad_norm': 0.24701812863349915, 'learning_rate': 9.127352212124662e-06, 'lm_loss': 0.4970829486846924, 'epoch': 3.47}

 87%|████████▋ | 2820/3252 [1:56:43<17:23,  2.41s/it]
 87%|████████▋ | 2821/3252 [1:56:46<17:23,  2.42s/it]
 87%|████████▋ | 2822/3252 [1:56:48<17:19,  2.42s/it]
 87%|████████▋ | 2823/3252 [1:56:51<17:13,  2.41s/it]
 87%|████████▋ | 2824/3252 [1:56:53<17:09,  2.40s/it]
 87%|████████▋ | 2825/3252 [1:56:55<17:06,  2.40s/it]
 87%|████████▋ | 2826/3252 [1:56:58<17:05,  2.41s/it]
 87%|████████▋ | 2827/3252 [1:57:00<17:01,  2.40s/it]
 87%|████████▋ | 2828/3252 [1:57:03<16:57,  2.40s/it]
 87%|████████▋ | 2829/3252 [1:57:05<16:54,  2.40s/it]
 87%|████████▋ | 2830/3252 [1:57:07<16:51,  2.40s/it]
                                                     
{'loss': 0.2797, 'grad_norm': 0.24823924899101257, 'learning_rate': 8.715858767069374e-06, 'lm_loss': 0.27129065990448, 'epoch': 3.48}

 87%|████████▋ | 2830/3252 [1:57:08<16:51,  2.40s/it]
 87%|████████▋ | 2831/3252 [1:57:10<17:08,  2.44s/it]
 87%|████████▋ | 2832/3252 [1:57:13<17:22,  2.48s/it]
 87%|████████▋ | 2833/3252 [1:57:15<17:09,  2.46s/it]
 87%|████████▋ | 2834/3252 [1:57:17<17:01,  2.44s/it]
 87%|████████▋ | 2835/3252 [1:57:20<16:56,  2.44s/it]
 87%|████████▋ | 2836/3252 [1:57:22<16:50,  2.43s/it]
 87%|████████▋ | 2837/3252 [1:57:25<16:42,  2.42s/it]
 87%|████████▋ | 2838/3252 [1:57:27<16:37,  2.41s/it]
 87%|████████▋ | 2839/3252 [1:57:29<16:33,  2.41s/it]
 87%|████████▋ | 2840/3252 [1:57:32<16:29,  2.40s/it]
                                                     
{'loss': 0.2719, 'grad_norm': 0.24856092035770416, 'learning_rate': 8.313433479768263e-06, 'lm_loss': 0.5285996794700623, 'epoch': 3.49}

 87%|████████▋ | 2840/3252 [1:57:32<16:29,  2.40s/it]
 87%|████████▋ | 2841/3252 [1:57:34<16:43,  2.44s/it]
 87%|████████▋ | 2842/3252 [1:57:37<16:34,  2.43s/it]
 87%|████████▋ | 2843/3252 [1:57:39<16:28,  2.42s/it]
 87%|████████▋ | 2844/3252 [1:57:41<16:23,  2.41s/it]
 87%|████████▋ | 2845/3252 [1:57:44<16:18,  2.40s/it]
 88%|████████▊ | 2846/3252 [1:57:46<16:13,  2.40s/it]
 88%|████████▊ | 2847/3252 [1:57:49<16:11,  2.40s/it]
 88%|████████▊ | 2848/3252 [1:57:51<16:08,  2.40s/it]
 88%|████████▊ | 2849/3252 [1:57:53<16:05,  2.40s/it]
 88%|████████▊ | 2850/3252 [1:57:56<16:02,  2.39s/it]
                                                     
{'loss': 0.2781, 'grad_norm': 0.23478907346725464, 'learning_rate': 7.920116327110449e-06, 'lm_loss': 0.43295639753341675, 'epoch': 3.51}

 88%|████████▊ | 2850/3252 [1:57:56<16:02,  2.39s/it]
 88%|████████▊ | 2851/3252 [1:57:58<16:14,  2.43s/it]
 88%|████████▊ | 2852/3252 [1:58:01<16:47,  2.52s/it]
 88%|████████▊ | 2853/3252 [1:58:04<16:44,  2.52s/it]
 88%|████████▊ | 2854/3252 [1:58:06<16:27,  2.48s/it]
 88%|████████▊ | 2855/3252 [1:58:09<16:33,  2.50s/it]
 88%|████████▊ | 2856/3252 [1:58:11<16:19,  2.47s/it]
 88%|████████▊ | 2857/3252 [1:58:13<16:07,  2.45s/it]
 88%|████████▊ | 2858/3252 [1:58:16<16:00,  2.44s/it]
 88%|████████▊ | 2859/3252 [1:58:18<15:53,  2.43s/it]
 88%|████████▊ | 2860/3252 [1:58:21<15:47,  2.42s/it]
                                                     
{'loss': 0.2776, 'grad_norm': 0.27008718252182007, 'learning_rate': 7.535946381183812e-06, 'lm_loss': 0.418768048286438, 'epoch': 3.52}

 88%|████████▊ | 2860/3252 [1:58:21<15:47,  2.42s/it]
 88%|████████▊ | 2861/3252 [1:58:23<16:01,  2.46s/it]
 88%|████████▊ | 2862/3252 [1:58:26<15:54,  2.45s/it]
 88%|████████▊ | 2863/3252 [1:58:28<15:46,  2.43s/it]
 88%|████████▊ | 2864/3252 [1:58:30<15:39,  2.42s/it]
 88%|████████▊ | 2865/3252 [1:58:33<15:33,  2.41s/it]
 88%|████████▊ | 2866/3252 [1:58:35<15:28,  2.41s/it]
 88%|████████▊ | 2867/3252 [1:58:37<15:23,  2.40s/it]
 88%|████████▊ | 2868/3252 [1:58:40<15:20,  2.40s/it]
 88%|████████▊ | 2869/3252 [1:58:42<15:27,  2.42s/it]
 88%|████████▊ | 2870/3252 [1:58:45<15:21,  2.41s/it]
                                                     
{'loss': 0.2677, 'grad_norm': 0.2499087005853653, 'learning_rate': 7.160961805393574e-06, 'lm_loss': 0.1824418306350708, 'epoch': 3.53}

 88%|████████▊ | 2870/3252 [1:58:45<15:21,  2.41s/it]
 88%|████████▊ | 2871/3252 [1:58:47<15:18,  2.41s/it]
 88%|████████▊ | 2872/3252 [1:58:50<15:12,  2.40s/it]
 88%|████████▊ | 2873/3252 [1:58:52<15:21,  2.43s/it]
 88%|████████▊ | 2874/3252 [1:58:54<15:15,  2.42s/it]
 88%|████████▊ | 2875/3252 [1:58:57<15:09,  2.41s/it]
 88%|████████▊ | 2876/3252 [1:58:59<15:07,  2.41s/it]
 88%|████████▊ | 2877/3252 [1:59:02<15:03,  2.41s/it]
 88%|████████▊ | 2878/3252 [1:59:04<14:59,  2.40s/it]
 89%|████████▊ | 2879/3252 [1:59:06<14:56,  2.40s/it]
 89%|████████▊ | 2880/3252 [1:59:09<14:52,  2.40s/it]
                                                     
{'loss': 0.2747, 'grad_norm': 0.21596089005470276, 'learning_rate': 6.79519985067123e-06, 'lm_loss': 0.5955700874328613, 'epoch': 3.54}

 89%|████████▊ | 2880/3252 [1:59:09<14:52,  2.40s/it]
 89%|████████▊ | 2881/3252 [1:59:11<14:51,  2.40s/it]
 89%|████████▊ | 2882/3252 [1:59:14<14:48,  2.40s/it]
 89%|████████▊ | 2883/3252 [1:59:16<14:44,  2.40s/it]
 89%|████████▊ | 2884/3252 [1:59:18<14:42,  2.40s/it]
 89%|████████▊ | 2885/3252 [1:59:21<14:41,  2.40s/it]
 89%|████████▊ | 2886/3252 [1:59:23<15:00,  2.46s/it]
 89%|████████▉ | 2887/3252 [1:59:26<14:51,  2.44s/it]
 89%|████████▉ | 2888/3252 [1:59:28<14:44,  2.43s/it]
 89%|████████▉ | 2889/3252 [1:59:31<14:38,  2.42s/it]
 89%|████████▉ | 2890/3252 [1:59:33<14:33,  2.41s/it]
                                                     
{'loss': 0.2638, 'grad_norm': 0.24689841270446777, 'learning_rate': 6.4386968517739065e-06, 'lm_loss': 0.22983774542808533, 'epoch': 3.55}

 89%|████████▉ | 2890/3252 [1:59:33<14:33,  2.41s/it]
 89%|████████▉ | 2891/3252 [1:59:36<14:48,  2.46s/it]
 89%|████████▉ | 2892/3252 [1:59:38<14:37,  2.44s/it]
 89%|████████▉ | 2893/3252 [1:59:40<14:30,  2.42s/it]
 89%|████████▉ | 2894/3252 [1:59:43<14:24,  2.41s/it]
 89%|████████▉ | 2895/3252 [1:59:45<14:20,  2.41s/it]
 89%|████████▉ | 2896/3252 [1:59:48<14:17,  2.41s/it]
 89%|████████▉ | 2897/3252 [1:59:50<14:14,  2.41s/it]
 89%|████████▉ | 2898/3252 [1:59:52<14:11,  2.41s/it]
 89%|████████▉ | 2899/3252 [1:59:55<14:07,  2.40s/it]
 89%|████████▉ | 2900/3252 [1:59:57<14:04,  2.40s/it]
                                                     
{'loss': 0.2749, 'grad_norm': 0.23166438937187195, 'learning_rate': 6.091488223675057e-06, 'lm_loss': 0.4197535216808319, 'epoch': 3.57}

 89%|████████▉ | 2900/3252 [1:59:57<14:04,  2.40s/it]
 89%|████████▉ | 2901/3252 [2:00:00<14:08,  2.42s/it]
 89%|████████▉ | 2902/3252 [2:00:02<14:03,  2.41s/it]
 89%|████████▉ | 2903/3252 [2:00:04<13:59,  2.40s/it]
 89%|████████▉ | 2904/3252 [2:00:07<13:56,  2.40s/it]
 89%|████████▉ | 2905/3252 [2:00:09<13:52,  2.40s/it]
 89%|████████▉ | 2906/3252 [2:00:12<14:07,  2.45s/it]
 89%|████████▉ | 2907/3252 [2:00:14<13:59,  2.43s/it]
 89%|████████▉ | 2908/3252 [2:00:17<13:52,  2.42s/it]
 89%|████████▉ | 2909/3252 [2:00:19<13:46,  2.41s/it]
 89%|████████▉ | 2910/3252 [2:00:21<13:43,  2.41s/it]
                                                     
{'loss': 0.2629, 'grad_norm': 0.2455013394355774, 'learning_rate': 5.753608458046145e-06, 'lm_loss': 0.01642519421875477, 'epoch': 3.58}

 89%|████████▉ | 2910/3252 [2:00:21<13:43,  2.41s/it]
 90%|████████▉ | 2911/3252 [2:00:24<13:45,  2.42s/it]
 90%|████████▉ | 2912/3252 [2:00:26<13:40,  2.41s/it]
 90%|████████▉ | 2913/3252 [2:00:29<13:37,  2.41s/it]
 90%|████████▉ | 2914/3252 [2:00:31<13:33,  2.41s/it]
 90%|████████▉ | 2915/3252 [2:00:33<13:29,  2.40s/it]
 90%|████████▉ | 2916/3252 [2:00:36<13:26,  2.40s/it]
 90%|████████▉ | 2917/3252 [2:00:38<13:23,  2.40s/it]
 90%|████████▉ | 2918/3252 [2:00:41<13:23,  2.41s/it]
 90%|████████▉ | 2919/3252 [2:00:43<13:19,  2.40s/it]
 90%|████████▉ | 2920/3252 [2:00:45<13:17,  2.40s/it]
                                                     
{'loss': 0.2667, 'grad_norm': 0.22232580184936523, 'learning_rate': 5.425091119830416e-06, 'lm_loss': 0.40061134099960327, 'epoch': 3.59}

 90%|████████▉ | 2920/3252 [2:00:46<13:17,  2.40s/it]
 90%|████████▉ | 2921/3252 [2:00:48<13:37,  2.47s/it]
 90%|████████▉ | 2922/3252 [2:00:50<13:27,  2.45s/it]
 90%|████████▉ | 2923/3252 [2:00:53<13:19,  2.43s/it]
 90%|████████▉ | 2924/3252 [2:00:55<13:14,  2.42s/it]
 90%|████████▉ | 2925/3252 [2:00:58<13:09,  2.42s/it]
 90%|████████▉ | 2926/3252 [2:01:00<13:33,  2.50s/it]
 90%|█████████ | 2927/3252 [2:01:03<13:32,  2.50s/it]
 90%|█████████ | 2928/3252 [2:01:05<13:20,  2.47s/it]
 90%|█████████ | 2929/3252 [2:01:08<13:24,  2.49s/it]
 90%|█████████ | 2930/3252 [2:01:10<13:13,  2.46s/it]
                                                     
{'loss': 0.2729, 'grad_norm': 0.2575761079788208, 'learning_rate': 5.105968843908448e-06, 'lm_loss': 0.4494326114654541, 'epoch': 3.6}

 90%|█████████ | 2930/3252 [2:01:10<13:13,  2.46s/it]
 90%|█████████ | 2931/3252 [2:01:13<13:06,  2.45s/it]
 90%|█████████ | 2932/3252 [2:01:15<12:59,  2.43s/it]
 90%|█████████ | 2933/3252 [2:01:17<12:56,  2.43s/it]
 90%|█████████ | 2934/3252 [2:01:20<12:50,  2.42s/it]
 90%|█████████ | 2935/3252 [2:01:22<12:45,  2.41s/it]
 90%|█████████ | 2936/3252 [2:01:25<12:41,  2.41s/it]
 90%|█████████ | 2937/3252 [2:01:27<12:37,  2.41s/it]
 90%|█████████ | 2938/3252 [2:01:29<12:35,  2.40s/it]
 90%|█████████ | 2939/3252 [2:01:32<12:32,  2.40s/it]
 90%|█████████ | 2940/3252 [2:01:34<12:29,  2.40s/it]
                                                     
{'loss': 0.2791, 'grad_norm': 0.2470293939113617, 'learning_rate': 4.796273331856227e-06, 'lm_loss': 0.09627572447061539, 'epoch': 3.62}

 90%|█████████ | 2940/3252 [2:01:34<12:29,  2.40s/it]
 90%|█████████ | 2941/3252 [2:01:37<12:40,  2.45s/it]
 90%|█████████ | 2942/3252 [2:01:39<12:32,  2.43s/it]
 90%|█████████ | 2943/3252 [2:01:42<12:35,  2.45s/it]
 91%|█████████ | 2944/3252 [2:01:44<12:27,  2.43s/it]
 91%|█████████ | 2945/3252 [2:01:46<12:22,  2.42s/it]
 91%|█████████ | 2946/3252 [2:01:49<12:18,  2.41s/it]
 91%|█████████ | 2947/3252 [2:01:51<12:25,  2.44s/it]
 91%|█████████ | 2948/3252 [2:01:54<12:17,  2.43s/it]
 91%|█████████ | 2949/3252 [2:01:56<12:13,  2.42s/it]
 91%|█████████ | 2950/3252 [2:01:58<12:07,  2.41s/it]
                                                     
{'loss': 0.272, 'grad_norm': 0.2393941730260849, 'learning_rate': 4.496035348795935e-06, 'lm_loss': 0.4748915731906891, 'epoch': 3.63}

 91%|█████████ | 2950/3252 [2:01:59<12:07,  2.41s/it]
 91%|█████████ | 2951/3252 [2:02:01<12:21,  2.46s/it]
 91%|█████████ | 2952/3252 [2:02:03<12:17,  2.46s/it]
 91%|█████████ | 2953/3252 [2:02:06<12:09,  2.44s/it]
 91%|█████████ | 2954/3252 [2:02:08<12:03,  2.43s/it]
 91%|█████████ | 2955/3252 [2:02:11<11:59,  2.42s/it]
 91%|█████████ | 2956/3252 [2:02:13<11:55,  2.42s/it]
 91%|█████████ | 2957/3252 [2:02:16<11:51,  2.41s/it]
 91%|█████████ | 2958/3252 [2:02:18<11:47,  2.41s/it]
 91%|█████████ | 2959/3252 [2:02:20<11:43,  2.40s/it]
 91%|█████████ | 2960/3252 [2:02:23<11:49,  2.43s/it]
                                                     
{'loss': 0.2755, 'grad_norm': 0.24173906445503235, 'learning_rate': 4.205284720339764e-06, 'lm_loss': 0.44986408948898315, 'epoch': 3.64}

 91%|█████████ | 2960/3252 [2:02:23<11:49,  2.43s/it]
 91%|█████████ | 2961/3252 [2:02:25<11:52,  2.45s/it]
 91%|█████████ | 2962/3252 [2:02:28<11:45,  2.43s/it]
 91%|█████████ | 2963/3252 [2:02:30<11:39,  2.42s/it]
 91%|█████████ | 2964/3252 [2:02:32<11:34,  2.41s/it]
 91%|█████████ | 2965/3252 [2:02:35<11:31,  2.41s/it]
 91%|█████████ | 2966/3252 [2:02:37<11:28,  2.41s/it]
 91%|█████████ | 2967/3252 [2:02:40<11:25,  2.40s/it]
 91%|█████████▏| 2968/3252 [2:02:42<11:21,  2.40s/it]
 91%|█████████▏| 2969/3252 [2:02:44<11:18,  2.40s/it]
 91%|█████████▏| 2970/3252 [2:02:47<11:15,  2.40s/it]
                                                     
{'loss': 0.264, 'grad_norm': 0.25045934319496155, 'learning_rate': 3.924050329627005e-06, 'lm_loss': 0.19378171861171722, 'epoch': 3.65}

 91%|█████████▏| 2970/3252 [2:02:47<11:15,  2.40s/it]
 91%|█████████▏| 2971/3252 [2:02:49<11:32,  2.46s/it]
 91%|█████████▏| 2972/3252 [2:02:52<11:23,  2.44s/it]
 91%|█████████▏| 2973/3252 [2:02:54<11:17,  2.43s/it]
 91%|█████████▏| 2974/3252 [2:02:57<11:12,  2.42s/it]
 91%|█████████▏| 2975/3252 [2:02:59<11:08,  2.41s/it]
 92%|█████████▏| 2976/3252 [2:03:01<11:04,  2.41s/it]
 92%|█████████▏| 2977/3252 [2:03:04<11:01,  2.41s/it]
 92%|█████████▏| 2978/3252 [2:03:06<10:59,  2.41s/it]
 92%|█████████▏| 2979/3252 [2:03:09<10:55,  2.40s/it]
 92%|█████████▏| 2980/3252 [2:03:11<10:57,  2.42s/it]
                                                     
{'loss': 0.2646, 'grad_norm': 0.23744182288646698, 'learning_rate': 3.6523601144548003e-06, 'lm_loss': 0.33511972427368164, 'epoch': 3.67}

 92%|█████████▏| 2980/3252 [2:03:11<10:57,  2.42s/it]
 92%|█████████▏| 2981/3252 [2:03:14<10:55,  2.42s/it]
 92%|█████████▏| 2982/3252 [2:03:16<10:51,  2.41s/it]
 92%|█████████▏| 2983/3252 [2:03:18<10:47,  2.41s/it]
 92%|█████████▏| 2984/3252 [2:03:21<10:45,  2.41s/it]
 92%|█████████▏| 2985/3252 [2:03:23<10:41,  2.40s/it]
 92%|█████████▏| 2986/3252 [2:03:25<10:38,  2.40s/it]
 92%|█████████▏| 2987/3252 [2:03:28<10:35,  2.40s/it]
 92%|█████████▏| 2988/3252 [2:03:30<10:32,  2.40s/it]
 92%|█████████▏| 2989/3252 [2:03:33<10:29,  2.39s/it]
 92%|█████████▏| 2990/3252 [2:03:35<10:27,  2.40s/it]
                                                     
{'loss': 0.2705, 'grad_norm': 0.24717333912849426, 'learning_rate': 3.390241064502819e-06, 'lm_loss': 0.002338010584935546, 'epoch': 3.68}

 92%|█████████▏| 2990/3252 [2:03:35<10:27,  2.40s/it]
 92%|█████████▏| 2991/3252 [2:03:38<10:35,  2.43s/it]
 92%|█████████▏| 2992/3252 [2:03:40<10:29,  2.42s/it]
 92%|█████████▏| 2993/3252 [2:03:42<10:25,  2.41s/it]
 92%|█████████▏| 2994/3252 [2:03:45<10:21,  2.41s/it]
 92%|█████████▏| 2995/3252 [2:03:47<10:19,  2.41s/it]
 92%|█████████▏| 2996/3252 [2:03:50<10:17,  2.41s/it]
 92%|█████████▏| 2997/3252 [2:03:52<10:14,  2.41s/it]
 92%|█████████▏| 2998/3252 [2:03:54<10:12,  2.41s/it]
 92%|█████████▏| 2999/3252 [2:03:57<10:08,  2.40s/it]
 92%|█████████▏| 3000/3252 [2:04:00<10:29,  2.50s/it]
                                                     
{'loss': 0.2767, 'grad_norm': 0.24774053692817688, 'learning_rate': 3.1377192186521354e-06, 'lm_loss': 0.40670278668403625, 'epoch': 3.69}

 92%|█████████▏| 3000/3252 [2:04:00<10:29,  2.50s/it]
 92%|█████████▏| 3001/3252 [2:04:02<10:37,  2.54s/it]
 92%|█████████▏| 3002/3252 [2:04:05<10:23,  2.50s/it]
 92%|█████████▏| 3003/3252 [2:04:07<10:24,  2.51s/it]
 92%|█████████▏| 3004/3252 [2:04:09<10:13,  2.47s/it]
 92%|█████████▏| 3005/3252 [2:04:12<10:04,  2.45s/it]
 92%|█████████▏| 3006/3252 [2:04:14<09:58,  2.43s/it]
 92%|█████████▏| 3007/3252 [2:04:17<09:52,  2.42s/it]
 92%|█████████▏| 3008/3252 [2:04:19<09:48,  2.41s/it]
 93%|█████████▎| 3009/3252 [2:04:21<09:44,  2.41s/it]
 93%|█████████▎| 3010/3252 [2:04:24<09:41,  2.40s/it]
                                                     
{'loss': 0.2688, 'grad_norm': 0.22164671123027802, 'learning_rate': 2.8948196623984623e-06, 'lm_loss': 0.5157948136329651, 'epoch': 3.7}

 93%|█████████▎| 3010/3252 [2:04:24<09:41,  2.40s/it]
 93%|█████████▎| 3011/3252 [2:04:26<09:42,  2.42s/it]
 93%|█████████▎| 3012/3252 [2:04:29<09:38,  2.41s/it]
 93%|█████████▎| 3013/3252 [2:04:31<09:35,  2.41s/it]
 93%|█████████▎| 3014/3252 [2:04:33<09:32,  2.40s/it]
 93%|█████████▎| 3015/3252 [2:04:36<09:30,  2.41s/it]
 93%|█████████▎| 3016/3252 [2:04:38<09:27,  2.40s/it]
 93%|█████████▎| 3017/3252 [2:04:41<09:25,  2.40s/it]
 93%|█████████▎| 3018/3252 [2:04:43<09:22,  2.40s/it]
 93%|█████████▎| 3019/3252 [2:04:45<09:19,  2.40s/it]
 93%|█████████▎| 3020/3252 [2:04:48<09:16,  2.40s/it]
                                                     
{'loss': 0.2698, 'grad_norm': 0.3645094633102417, 'learning_rate': 2.661566525360193e-06, 'lm_loss': 0.4789891541004181, 'epoch': 3.71}

 93%|█████████▎| 3020/3252 [2:04:48<09:16,  2.40s/it]
 93%|█████████▎| 3021/3252 [2:04:50<09:22,  2.43s/it]
 93%|█████████▎| 3022/3252 [2:04:53<09:16,  2.42s/it]
 93%|█████████▎| 3023/3252 [2:04:55<09:12,  2.41s/it]
 93%|█████████▎| 3024/3252 [2:04:58<09:08,  2.41s/it]
 93%|█████████▎| 3025/3252 [2:05:00<09:05,  2.40s/it]
 93%|█████████▎| 3026/3252 [2:05:02<09:01,  2.40s/it]
 93%|█████████▎| 3027/3252 [2:05:05<08:59,  2.40s/it]
 93%|█████████▎| 3028/3252 [2:05:07<08:56,  2.39s/it]
 93%|█████████▎| 3029/3252 [2:05:10<08:53,  2.39s/it]
 93%|█████████▎| 3030/3252 [2:05:12<08:51,  2.40s/it]
                                                     
{'loss': 0.2589, 'grad_norm': 0.23957160115242004, 'learning_rate': 2.4379829788813815e-06, 'lm_loss': 0.5246862769126892, 'epoch': 3.73}

 93%|█████████▎| 3030/3252 [2:05:12<08:51,  2.40s/it]
 93%|█████████▎| 3031/3252 [2:05:14<08:50,  2.40s/it]
 93%|█████████▎| 3032/3252 [2:05:17<08:48,  2.40s/it]
 93%|█████████▎| 3033/3252 [2:05:19<08:45,  2.40s/it]
 93%|█████████▎| 3034/3252 [2:05:22<08:48,  2.43s/it]
 93%|█████████▎| 3035/3252 [2:05:24<08:44,  2.42s/it]
 93%|█████████▎| 3036/3252 [2:05:26<08:40,  2.41s/it]
 93%|█████████▎| 3037/3252 [2:05:29<08:37,  2.41s/it]
 93%|█████████▎| 3038/3252 [2:05:31<08:34,  2.41s/it]
 93%|█████████▎| 3039/3252 [2:05:34<08:31,  2.40s/it]
 93%|█████████▎| 3040/3252 [2:05:36<08:29,  2.40s/it]
                                                     
{'loss': 0.2626, 'grad_norm': 0.23417112231254578, 'learning_rate': 2.22409123372993e-06, 'lm_loss': 0.0004370030073914677, 'epoch': 3.74}

 93%|█████████▎| 3040/3252 [2:05:36<08:29,  2.40s/it]
 94%|█████████▎| 3041/3252 [2:05:38<08:27,  2.40s/it]
 94%|█████████▎| 3042/3252 [2:05:41<08:24,  2.40s/it]
 94%|█████████▎| 3043/3252 [2:05:43<08:21,  2.40s/it]
 94%|█████████▎| 3044/3252 [2:05:46<08:18,  2.40s/it]
 94%|█████████▎| 3045/3252 [2:05:48<08:16,  2.40s/it]
 94%|█████████▎| 3046/3252 [2:05:50<08:13,  2.40s/it]
 94%|█████████▎| 3047/3252 [2:05:53<08:10,  2.39s/it]
 94%|█████████▎| 3048/3252 [2:05:55<08:08,  2.39s/it]
 94%|█████████▍| 3049/3252 [2:05:58<08:06,  2.39s/it]
 94%|█████████▍| 3050/3252 [2:06:00<08:03,  2.39s/it]
                                                     
{'loss': 0.2711, 'grad_norm': 0.2439100742340088, 'learning_rate': 2.0199125378910668e-06, 'lm_loss': 0.06448531150817871, 'epoch': 3.75}

 94%|█████████▍| 3050/3252 [2:06:00<08:03,  2.39s/it]
 94%|█████████▍| 3051/3252 [2:06:02<08:08,  2.43s/it]
 94%|█████████▍| 3052/3252 [2:06:05<08:03,  2.42s/it]
 94%|█████████▍| 3053/3252 [2:06:07<08:00,  2.41s/it]
 94%|█████████▍| 3054/3252 [2:06:10<08:00,  2.43s/it]
 94%|█████████▍| 3055/3252 [2:06:12<07:55,  2.42s/it]
 94%|█████████▍| 3056/3252 [2:06:15<07:51,  2.41s/it]
 94%|█████████▍| 3057/3252 [2:06:17<07:48,  2.40s/it]
 94%|█████████▍| 3058/3252 [2:06:19<07:45,  2.40s/it]
 94%|█████████▍| 3059/3252 [2:06:22<07:42,  2.40s/it]
 94%|█████████▍| 3060/3252 [2:06:24<07:39,  2.40s/it]
                                                     
{'loss': 0.255, 'grad_norm': 0.2323949635028839, 'learning_rate': 1.825467174456652e-06, 'lm_loss': 0.4853764474391937, 'epoch': 3.76}

 94%|█████████▍| 3060/3252 [2:06:24<07:39,  2.40s/it]
 94%|█████████▍| 3061/3252 [2:06:27<07:40,  2.41s/it]
 94%|█████████▍| 3062/3252 [2:06:29<07:36,  2.40s/it]
 94%|█████████▍| 3063/3252 [2:06:31<07:33,  2.40s/it]
 94%|█████████▍| 3064/3252 [2:06:34<07:31,  2.40s/it]
 94%|█████████▍| 3065/3252 [2:06:36<07:28,  2.40s/it]
 94%|█████████▍| 3066/3252 [2:06:39<07:27,  2.40s/it]
 94%|█████████▍| 3067/3252 [2:06:41<07:24,  2.40s/it]
 94%|█████████▍| 3068/3252 [2:06:43<07:22,  2.40s/it]
 94%|█████████▍| 3069/3252 [2:06:46<07:19,  2.40s/it]
 94%|█████████▍| 3070/3252 [2:06:48<07:16,  2.40s/it]
                                                     
{'loss': 0.2823, 'grad_norm': 0.24211806058883667, 'learning_rate': 1.6407744596103281e-06, 'lm_loss': 0.34452250599861145, 'epoch': 3.78}

 94%|█████████▍| 3070/3252 [2:06:48<07:16,  2.40s/it]
 94%|█████████▍| 3071/3252 [2:06:51<07:14,  2.40s/it]
 94%|█████████▍| 3072/3252 [2:06:53<07:11,  2.40s/it]
 94%|█████████▍| 3073/3252 [2:06:55<07:09,  2.40s/it]
 95%|█████████▍| 3074/3252 [2:06:58<07:15,  2.45s/it]
 95%|█████████▍| 3075/3252 [2:07:00<07:15,  2.46s/it]
 95%|█████████▍| 3076/3252 [2:07:03<07:09,  2.44s/it]
 95%|█████████▍| 3077/3252 [2:07:05<07:13,  2.47s/it]
 95%|█████████▍| 3078/3252 [2:07:08<07:06,  2.45s/it]
 95%|█████████▍| 3079/3252 [2:07:10<07:01,  2.43s/it]
 95%|█████████▍| 3080/3252 [2:07:12<06:57,  2.42s/it]
                                                     
{'loss': 0.2748, 'grad_norm': 0.23925240337848663, 'learning_rate': 1.4658527407084733e-06, 'lm_loss': 0.0037044312339276075, 'epoch': 3.79}

 95%|█████████▍| 3080/3252 [2:07:13<06:57,  2.42s/it]
 95%|█████████▍| 3081/3252 [2:07:15<06:55,  2.43s/it]
 95%|█████████▍| 3082/3252 [2:07:17<06:51,  2.42s/it]
 95%|█████████▍| 3083/3252 [2:07:20<06:47,  2.41s/it]
 95%|█████████▍| 3084/3252 [2:07:22<06:44,  2.41s/it]
 95%|█████████▍| 3085/3252 [2:07:25<06:41,  2.41s/it]
 95%|█████████▍| 3086/3252 [2:07:27<06:38,  2.40s/it]
 95%|█████████▍| 3087/3252 [2:07:29<06:36,  2.40s/it]
 95%|█████████▍| 3088/3252 [2:07:32<06:33,  2.40s/it]
 95%|█████████▍| 3089/3252 [2:07:34<06:31,  2.40s/it]
 95%|█████████▌| 3090/3252 [2:07:37<06:28,  2.40s/it]
                                                     
{'loss': 0.2634, 'grad_norm': 0.2354845553636551, 'learning_rate': 1.3007193944577056e-06, 'lm_loss': 0.6369805932044983, 'epoch': 3.8}

 95%|█████████▌| 3090/3252 [2:07:37<06:28,  2.40s/it]
 95%|█████████▌| 3091/3252 [2:07:39<06:32,  2.44s/it]
 95%|█████████▌| 3092/3252 [2:07:41<06:27,  2.42s/it]
 95%|█████████▌| 3093/3252 [2:07:44<06:23,  2.42s/it]
 95%|█████████▌| 3094/3252 [2:07:46<06:20,  2.41s/it]
 95%|█████████▌| 3095/3252 [2:07:49<06:23,  2.44s/it]
 95%|█████████▌| 3096/3252 [2:07:51<06:19,  2.43s/it]
 95%|█████████▌| 3097/3252 [2:07:54<06:15,  2.42s/it]
 95%|█████████▌| 3098/3252 [2:07:56<06:11,  2.41s/it]
 95%|█████████▌| 3099/3252 [2:07:58<06:08,  2.41s/it]
 95%|█████████▌| 3100/3252 [2:08:01<06:05,  2.41s/it]
                                                     
{'loss': 0.2667, 'grad_norm': 0.24435867369174957, 'learning_rate': 1.1453908251886636e-06, 'lm_loss': 0.026858188211917877, 'epoch': 3.81}

 95%|█████████▌| 3100/3252 [2:08:01<06:05,  2.41s/it]
 95%|█████████▌| 3101/3252 [2:08:03<06:03,  2.41s/it]
 95%|█████████▌| 3102/3252 [2:08:06<06:00,  2.41s/it]
 95%|█████████▌| 3103/3252 [2:08:08<05:58,  2.41s/it]
 95%|█████████▌| 3104/3252 [2:08:10<05:55,  2.40s/it]
 95%|█████████▌| 3105/3252 [2:08:13<05:53,  2.40s/it]
 96%|█████████▌| 3106/3252 [2:08:15<05:50,  2.40s/it]
 96%|█████████▌| 3107/3252 [2:08:18<05:47,  2.40s/it]
 96%|█████████▌| 3108/3252 [2:08:20<05:48,  2.42s/it]
 96%|█████████▌| 3109/3252 [2:08:22<05:44,  2.41s/it]
 96%|█████████▌| 3110/3252 [2:08:25<05:41,  2.41s/it]
                                                     
{'loss': 0.2791, 'grad_norm': 0.23908250033855438, 'learning_rate': 9.998824632263981e-07, 'lm_loss': 0.0005383649840950966, 'epoch': 3.83}

 96%|█████████▌| 3110/3252 [2:08:25<05:41,  2.41s/it]
 96%|█████████▌| 3111/3252 [2:08:27<05:45,  2.45s/it]
 96%|█████████▌| 3112/3252 [2:08:30<05:40,  2.43s/it]
 96%|█████████▌| 3113/3252 [2:08:32<05:36,  2.42s/it]
 96%|█████████▌| 3114/3252 [2:08:35<05:33,  2.42s/it]
 96%|█████████▌| 3115/3252 [2:08:37<05:30,  2.42s/it]
 96%|█████████▌| 3116/3252 [2:08:39<05:27,  2.41s/it]
 96%|█████████▌| 3117/3252 [2:08:42<05:24,  2.41s/it]
 96%|█████████▌| 3118/3252 [2:08:44<05:21,  2.40s/it]
 96%|█████████▌| 3119/3252 [2:08:47<05:20,  2.41s/it]
 96%|█████████▌| 3120/3252 [2:08:49<05:16,  2.40s/it]
                                                     
{'loss': 0.2719, 'grad_norm': 0.3875643312931061, 'learning_rate': 8.64208763357488e-07, 'lm_loss': 0.40408530831336975, 'epoch': 3.84}

 96%|█████████▌| 3120/3252 [2:08:49<05:16,  2.40s/it]
 96%|█████████▌| 3121/3252 [2:08:51<05:15,  2.41s/it]
 96%|█████████▌| 3122/3252 [2:08:54<05:12,  2.40s/it]
 96%|█████████▌| 3123/3252 [2:08:56<05:09,  2.40s/it]
 96%|█████████▌| 3124/3252 [2:08:59<05:06,  2.40s/it]
 96%|█████████▌| 3125/3252 [2:09:01<05:04,  2.40s/it]
 96%|█████████▌| 3126/3252 [2:09:03<05:02,  2.40s/it]
 96%|█████████▌| 3127/3252 [2:09:06<04:59,  2.40s/it]
 96%|█████████▌| 3128/3252 [2:09:08<05:02,  2.44s/it]
 96%|█████████▌| 3129/3252 [2:09:11<04:58,  2.43s/it]
 96%|█████████▌| 3130/3252 [2:09:13<04:54,  2.42s/it]
                                                     
{'loss': 0.2639, 'grad_norm': 0.2575129270553589, 'learning_rate': 7.383832033941551e-07, 'lm_loss': 0.36554184556007385, 'epoch': 3.85}

 96%|█████████▌| 3130/3252 [2:09:13<04:54,  2.42s/it]
 96%|█████████▋| 3131/3252 [2:09:16<04:56,  2.45s/it]
 96%|█████████▋| 3132/3252 [2:09:18<04:52,  2.44s/it]
 96%|█████████▋| 3133/3252 [2:09:20<04:48,  2.42s/it]
 96%|█████████▋| 3134/3252 [2:09:23<04:45,  2.42s/it]
 96%|█████████▋| 3135/3252 [2:09:25<04:42,  2.41s/it]
 96%|█████████▋| 3136/3252 [2:09:28<04:38,  2.41s/it]
 96%|█████████▋| 3137/3252 [2:09:30<04:36,  2.40s/it]
 96%|█████████▋| 3138/3252 [2:09:32<04:33,  2.40s/it]
 97%|█████████▋| 3139/3252 [2:09:35<04:31,  2.40s/it]
 97%|█████████▋| 3140/3252 [2:09:37<04:28,  2.40s/it]
                                                     
{'loss': 0.2624, 'grad_norm': 0.25785592198371887, 'learning_rate': 6.224182828353242e-07, 'lm_loss': 0.31942135095596313, 'epoch': 3.86}

 97%|█████████▋| 3140/3252 [2:09:37<04:28,  2.40s/it]
 97%|█████████▋| 3141/3252 [2:09:40<04:29,  2.43s/it]
 97%|█████████▋| 3142/3252 [2:09:42<04:25,  2.42s/it]
 97%|█████████▋| 3143/3252 [2:09:44<04:22,  2.41s/it]
 97%|█████████▋| 3144/3252 [2:09:47<04:19,  2.40s/it]
 97%|█████████▋| 3145/3252 [2:09:49<04:17,  2.40s/it]
 97%|█████████▋| 3146/3252 [2:09:52<04:14,  2.40s/it]
 97%|█████████▋| 3147/3252 [2:09:54<04:13,  2.42s/it]
 97%|█████████▋| 3148/3252 [2:09:57<04:19,  2.50s/it]
 97%|█████████▋| 3149/3252 [2:09:59<04:17,  2.50s/it]
 97%|█████████▋| 3150/3252 [2:10:02<04:11,  2.47s/it]
                                                     
{'loss': 0.2665, 'grad_norm': 0.22496092319488525, 'learning_rate': 5.163255216250163e-07, 'lm_loss': 0.15444932878017426, 'epoch': 3.87}

 97%|█████████▋| 3150/3252 [2:10:02<04:11,  2.47s/it]
 97%|█████████▋| 3151/3252 [2:10:04<04:12,  2.50s/it]
 97%|█████████▋| 3152/3252 [2:10:07<04:07,  2.48s/it]
 97%|█████████▋| 3153/3252 [2:10:09<04:02,  2.45s/it]
 97%|█████████▋| 3154/3252 [2:10:11<03:58,  2.44s/it]
 97%|█████████▋| 3155/3252 [2:10:14<03:55,  2.43s/it]
 97%|█████████▋| 3156/3252 [2:10:16<03:52,  2.42s/it]
 97%|█████████▋| 3157/3252 [2:10:19<03:49,  2.42s/it]
 97%|█████████▋| 3158/3252 [2:10:21<03:46,  2.41s/it]
 97%|█████████▋| 3159/3252 [2:10:23<03:43,  2.40s/it]
 97%|█████████▋| 3160/3252 [2:10:26<03:41,  2.40s/it]
                                                     
{'loss': 0.2654, 'grad_norm': 0.2237294763326645, 'learning_rate': 4.2011545900781936e-07, 'lm_loss': 0.0006178503390401602, 'epoch': 3.89}

 97%|█████████▋| 3160/3252 [2:10:26<03:41,  2.40s/it]
 97%|█████████▋| 3161/3252 [2:10:28<03:41,  2.43s/it]
 97%|█████████▋| 3162/3252 [2:10:31<03:37,  2.42s/it]
 97%|█████████▋| 3163/3252 [2:10:33<03:35,  2.42s/it]
 97%|█████████▋| 3164/3252 [2:10:36<03:32,  2.41s/it]
 97%|█████████▋| 3165/3252 [2:10:38<03:34,  2.46s/it]
 97%|█████████▋| 3166/3252 [2:10:41<03:29,  2.44s/it]
 97%|█████████▋| 3167/3252 [2:10:43<03:26,  2.43s/it]
 97%|█████████▋| 3168/3252 [2:10:45<03:23,  2.42s/it]
 97%|█████████▋| 3169/3252 [2:10:48<03:23,  2.45s/it]
 97%|█████████▋| 3170/3252 [2:10:50<03:19,  2.43s/it]
                                                     
{'loss': 0.2777, 'grad_norm': 0.24541829526424408, 'learning_rate': 3.3379765248207074e-07, 'lm_loss': 0.675696611404419, 'epoch': 3.9}

 97%|█████████▋| 3170/3252 [2:10:50<03:19,  2.43s/it]
 98%|█████████▊| 3171/3252 [2:10:53<03:17,  2.44s/it]
 98%|█████████▊| 3172/3252 [2:10:55<03:14,  2.43s/it]
 98%|█████████▊| 3173/3252 [2:10:58<03:11,  2.42s/it]
 98%|█████████▊| 3174/3252 [2:11:00<03:08,  2.41s/it]
 98%|█████████▊| 3175/3252 [2:11:02<03:05,  2.41s/it]
 98%|█████████▊| 3176/3252 [2:11:05<03:02,  2.40s/it]
 98%|█████████▊| 3177/3252 [2:11:07<03:00,  2.41s/it]
 98%|█████████▊| 3178/3252 [2:11:09<02:57,  2.40s/it]
 98%|█████████▊| 3179/3252 [2:11:12<02:54,  2.39s/it]
 98%|█████████▊| 3180/3252 [2:11:14<02:52,  2.39s/it]
                                                     
{'loss': 0.268, 'grad_norm': 0.2388031929731369, 'learning_rate': 2.573806768503162e-07, 'lm_loss': 0.4999023377895355, 'epoch': 3.91}

 98%|█████████▊| 3180/3252 [2:11:15<02:52,  2.39s/it]
 98%|█████████▊| 3181/3252 [2:11:17<02:54,  2.45s/it]
 98%|█████████▊| 3182/3252 [2:11:19<02:52,  2.46s/it]
 98%|█████████▊| 3183/3252 [2:11:22<02:48,  2.44s/it]
 98%|█████████▊| 3184/3252 [2:11:24<02:45,  2.43s/it]
 98%|█████████▊| 3185/3252 [2:11:27<02:41,  2.42s/it]
 98%|█████████▊| 3186/3252 [2:11:29<02:39,  2.41s/it]
 98%|█████████▊| 3187/3252 [2:11:31<02:36,  2.41s/it]
 98%|█████████▊| 3188/3252 [2:11:34<02:34,  2.41s/it]
 98%|█████████▊| 3189/3252 [2:11:36<02:31,  2.41s/it]
 98%|█████████▊| 3190/3252 [2:11:39<02:28,  2.40s/it]
                                                     
{'loss': 0.2815, 'grad_norm': 0.21918249130249023, 'learning_rate': 1.908721233675026e-07, 'lm_loss': 0.08862845599651337, 'epoch': 3.92}

 98%|█████████▊| 3190/3252 [2:11:39<02:28,  2.40s/it]
 98%|█████████▊| 3191/3252 [2:11:41<02:29,  2.45s/it]
 98%|█████████▊| 3192/3252 [2:11:43<02:25,  2.43s/it]
 98%|█████████▊| 3193/3252 [2:11:46<02:22,  2.42s/it]
 98%|█████████▊| 3194/3252 [2:11:48<02:19,  2.41s/it]
 98%|█████████▊| 3195/3252 [2:11:51<02:16,  2.40s/it]
 98%|█████████▊| 3196/3252 [2:11:53<02:14,  2.40s/it]
 98%|█████████▊| 3197/3252 [2:11:55<02:12,  2.41s/it]
 98%|█████████▊| 3198/3252 [2:11:58<02:09,  2.41s/it]
 98%|█████████▊| 3199/3252 [2:12:00<02:07,  2.40s/it]
 98%|█████████▊| 3200/3252 [2:12:03<02:04,  2.40s/it]
                                                     
{'loss': 0.281, 'grad_norm': 0.25757116079330444, 'learning_rate': 1.342785989868922e-07, 'lm_loss': 0.0005358088528737426, 'epoch': 3.94}

 98%|█████████▊| 3200/3252 [2:12:03<02:04,  2.40s/it]

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.59it/s]
                                                     

                                             
{'eval_sthsthv2/narration_val_loss': 0.7790353894233704, 'eval_sthsthv2/narration_val_runtime': 3.7612, 'eval_sthsthv2/narration_val_samples_per_second': 177.605, 'eval_sthsthv2/narration_val_steps_per_second': 0.532, 'epoch': 3.94}

 98%|█████████▊| 3200/3252 [2:12:06<02:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.59it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.52it/s]
                                                     

                                             
{'eval_ego4d/dialog-klg-sum_val_loss': 0.6423233151435852, 'eval_ego4d/dialog-klg-sum_val_runtime': 3.1951, 'eval_ego4d/dialog-klg-sum_val_samples_per_second': 185.282, 'eval_ego4d/dialog-klg-sum_val_steps_per_second': 0.626, 'epoch': 3.94}

 98%|█████████▊| 3200/3252 [2:12:10<02:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.52it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:00<00:00,  2.09it/s]
                                                     

                                             
{'eval_holoassist/dialog-klg-sum_val_loss': 0.5899379849433899, 'eval_holoassist/dialog-klg-sum_val_runtime': 2.9833, 'eval_holoassist/dialog-klg-sum_val_samples_per_second': 237.654, 'eval_holoassist/dialog-klg-sum_val_steps_per_second': 0.67, 'epoch': 3.94}

 98%|█████████▊| 3200/3252 [2:12:13<02:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  2.09it/s]

                                             

  0%|          | 0/1 [00:00<?, ?it/s]
                                                     

                                     
{'eval_ego4d/narration_val_loss': 0.5158975124359131, 'eval_ego4d/narration_val_runtime': 2.1717, 'eval_ego4d/narration_val_samples_per_second': 102.685, 'eval_ego4d/narration_val_steps_per_second': 0.46, 'epoch': 3.94}

 98%|█████████▊| 3200/3252 [2:12:15<02:04,  2.40s/it]

100%|██████████| 1/1 [00:00<00:00,  5.07it/s]

                                             

  0%|          | 0/2 [00:00<?, ?it/s]

100%|██████████| 2/2 [00:01<00:00,  1.49it/s]
                                                     

                                             
{'eval_ego4d/summary_val_loss': 1.1419909000396729, 'eval_ego4d/summary_val_runtime': 2.8928, 'eval_ego4d/summary_val_samples_per_second': 194.965, 'eval_ego4d/summary_val_steps_per_second': 0.691, 'epoch': 3.94}

 98%|█████████▊| 3200/3252 [2:12:18<02:04,  2.40s/it]

100%|██████████| 2/2 [00:01<00:00,  1.49it/s]

                                             
 98%|█████████▊| 3201/3252 [2:12:20<05:55,  6.97s/it]
 98%|█████████▊| 3202/3252 [2:12:23<04:42,  5.65s/it]
 98%|█████████▊| 3203/3252 [2:12:25<03:48,  4.67s/it]
 99%|█████████▊| 3204/3252 [2:12:28<03:11,  3.99s/it]
 99%|█████████▊| 3205/3252 [2:12:30<02:44,  3.51s/it]
 99%|█████████▊| 3206/3252 [2:12:32<02:26,  3.18s/it]
 99%|█████████▊| 3207/3252 [2:12:35<02:12,  2.94s/it]
 99%|█████████▊| 3208/3252 [2:12:37<02:02,  2.78s/it]
 99%|█████████▊| 3209/3252 [2:12:40<01:54,  2.66s/it]
 99%|█████████▊| 3210/3252 [2:12:42<01:48,  2.58s/it]
                                                     
{'loss': 0.2708, 'grad_norm': 0.25546175241470337, 'learning_rate': 8.760572570373216e-08, 'lm_loss': 0.5509198904037476, 'epoch': 3.95}

 99%|█████████▊| 3210/3252 [2:12:42<01:48,  2.58s/it]
 99%|█████████▊| 3211/3252 [2:12:45<01:47,  2.61s/it]
 99%|█████████▉| 3212/3252 [2:12:47<01:41,  2.55s/it]
 99%|█████████▉| 3213/3252 [2:12:49<01:37,  2.50s/it]
 99%|█████████▉| 3214/3252 [2:12:52<01:34,  2.48s/it]
 99%|█████████▉| 3215/3252 [2:12:54<01:30,  2.45s/it]
 99%|█████████▉| 3216/3252 [2:12:57<01:27,  2.43s/it]
 99%|█████████▉| 3217/3252 [2:12:59<01:24,  2.42s/it]
 99%|█████████▉| 3218/3252 [2:13:01<01:22,  2.41s/it]
 99%|█████████▉| 3219/3252 [2:13:04<01:19,  2.41s/it]
 99%|█████████▉| 3220/3252 [2:13:06<01:16,  2.40s/it]
                                                     
{'loss': 0.2677, 'grad_norm': 0.23680241405963898, 'learning_rate': 5.08581399967345e-08, 'lm_loss': 0.00033050638739950955, 'epoch': 3.96}

 99%|█████████▉| 3220/3252 [2:13:06<01:16,  2.40s/it]
 99%|█████████▉| 3221/3252 [2:13:09<01:14,  2.41s/it]
 99%|█████████▉| 3222/3252 [2:13:11<01:13,  2.46s/it]
 99%|█████████▉| 3223/3252 [2:13:14<01:11,  2.45s/it]
 99%|█████████▉| 3224/3252 [2:13:16<01:08,  2.43s/it]
 99%|█████████▉| 3225/3252 [2:13:19<01:06,  2.48s/it]
 99%|█████████▉| 3226/3252 [2:13:21<01:03,  2.45s/it]
 99%|█████████▉| 3227/3252 [2:13:23<01:00,  2.43s/it]
 99%|█████████▉| 3228/3252 [2:13:26<00:58,  2.42s/it]
 99%|█████████▉| 3229/3252 [2:13:28<00:55,  2.42s/it]
 99%|█████████▉| 3230/3252 [2:13:31<00:53,  2.42s/it]
                                                     
{'loss': 0.2662, 'grad_norm': 0.2335221767425537, 'learning_rate': 2.4039492367500248e-08, 'lm_loss': 0.006043595261871815, 'epoch': 3.97}

 99%|█████████▉| 3230/3252 [2:13:31<00:53,  2.42s/it]
 99%|█████████▉| 3231/3252 [2:13:33<00:51,  2.46s/it]
 99%|█████████▉| 3232/3252 [2:13:36<00:48,  2.44s/it]
 99%|█████████▉| 3233/3252 [2:13:38<00:46,  2.43s/it]
 99%|█████████▉| 3234/3252 [2:13:40<00:43,  2.42s/it]
 99%|█████████▉| 3235/3252 [2:13:43<00:40,  2.41s/it]
100%|█████████▉| 3236/3252 [2:13:45<00:38,  2.40s/it]
100%|█████████▉| 3237/3252 [2:13:48<00:36,  2.40s/it]
100%|█████████▉| 3238/3252 [2:13:50<00:33,  2.40s/it]
100%|█████████▉| 3239/3252 [2:13:52<00:31,  2.42s/it]
100%|█████████▉| 3240/3252 [2:13:55<00:28,  2.42s/it]
                                                     
{'loss': 0.2771, 'grad_norm': 0.2395607978105545, 'learning_rate': 7.152446977898297e-09, 'lm_loss': 0.2598412334918976, 'epoch': 3.99}

100%|█████████▉| 3240/3252 [2:13:55<00:28,  2.42s/it]
100%|█████████▉| 3241/3252 [2:13:57<00:26,  2.43s/it]
100%|█████████▉| 3242/3252 [2:14:00<00:24,  2.42s/it]
100%|█████████▉| 3243/3252 [2:14:02<00:21,  2.44s/it]
100%|█████████▉| 3244/3252 [2:14:05<00:19,  2.43s/it]
100%|█████████▉| 3245/3252 [2:14:07<00:16,  2.41s/it]
100%|█████████▉| 3246/3252 [2:14:09<00:14,  2.41s/it]
100%|█████████▉| 3247/3252 [2:14:12<00:12,  2.41s/it]
100%|█████████▉| 3248/3252 [2:14:14<00:09,  2.40s/it]
100%|█████████▉| 3249/3252 [2:14:17<00:07,  2.40s/it]
100%|█████████▉| 3250/3252 [2:14:19<00:04,  2.40s/it]
                                                     
{'loss': 0.2765, 'grad_norm': 0.2267940193414688, 'learning_rate': 1.9868138537715297e-10, 'lm_loss': 0.6943933367729187, 'epoch': 4.0}

100%|█████████▉| 3250/3252 [2:14:19<00:04,  2.40s/it]
100%|█████████▉| 3251/3252 [2:14:21<00:02,  2.41s/it]
100%|██████████| 3252/3252 [2:14:24<00:00,  2.47s/it]
                                                     
{'train_runtime': 8064.8498, 'train_samples_per_second': 206.391, 'train_steps_per_second': 0.403, 'train_loss': 0.4584020487414295, 'lm_loss': 0.1839885711669922, 'epoch': 4.0}

100%|██████████| 3252/3252 [2:14:24<00:00,  2.47s/it]
100%|██████████| 3252/3252 [2:14:24<00:00,  2.48s/it]
Training finished!
wandb: - 0.029 MB of 0.029 MB uploaded
wandb: \ 0.029 MB of 0.029 MB uploaded
wandb: | 0.029 MB of 0.029 MB uploaded
wandb: / 0.029 MB of 0.029 MB uploaded
wandb: - 0.029 MB of 0.029 MB uploaded
wandb: \ 0.090 MB of 0.147 MB uploaded (0.005 MB deduped)
wandb: | 0.147 MB of 0.147 MB uploaded (0.005 MB deduped)
wandb: 
wandb: Run history:
wandb:                    eval/ego4d/dialog-klg-sum_val_loss ▆▁▃▂▅▅██
wandb:                 eval/ego4d/dialog-klg-sum_val_runtime ▁█▇▂▂▅▄▄
wandb:      eval/ego4d/dialog-klg-sum_val_samples_per_second █▁▂▇▇▄▅▅
wandb:        eval/ego4d/dialog-klg-sum_val_steps_per_second █▁▂▇▇▅▅▅
wandb:                         eval/ego4d/narration_val_loss █▅▄▂▂▁▂▁
wandb:                      eval/ego4d/narration_val_runtime ▅▅▄▂█▁▄▂
wandb:           eval/ego4d/narration_val_samples_per_second ▄▄▅▇▁█▅▇
wandb:             eval/ego4d/narration_val_steps_per_second ▄▄▅▇▁█▅▇
wandb:                           eval/ego4d/summary_val_loss █▃▇▁▅▄▇▆
wandb:                        eval/ego4d/summary_val_runtime ▅█▇▁▅█▃▄
wandb:             eval/ego4d/summary_val_samples_per_second ▄▁▂█▄▁▆▅
wandb:               eval/ego4d/summary_val_steps_per_second ▄▁▂█▄▁▆▄
wandb:               eval/holoassist/dialog-klg-sum_val_loss █▄▃▁▁▁▂▂
wandb:            eval/holoassist/dialog-klg-sum_val_runtime ▃▆▄▄▃█▁▂
wandb: eval/holoassist/dialog-klg-sum_val_samples_per_second ▆▃▅▅▅▁█▇
wandb:   eval/holoassist/dialog-klg-sum_val_steps_per_second ▆▃▅▅▅▁█▇
wandb:                      eval/sthsthv2/narration_val_loss ▁▅▆█▆▇▆▆
wandb:                   eval/sthsthv2/narration_val_runtime ▅▆▁▅█▃▂█
wandb:        eval/sthsthv2/narration_val_samples_per_second ▄▃█▄▁▆▇▁
wandb:          eval/sthsthv2/narration_val_steps_per_second ▄▃█▄▁▆▇▂
wandb:                                           train/epoch ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇████
wandb:                                     train/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇████
wandb:                                       train/grad_norm ██▁▁▂▁▁▂▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁
wandb:                                   train/learning_rate ▄███████▇▇▇▇▇▆▆▆▆▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁▁
wandb:                                         train/lm_loss ▅▇▄▄▄▂▂▄▂▃▄▅█▁▄▅▁▁▁▄▄▁▁▁▇▃▂▁▄▂▃▁▃▂▃▄▃▁▃▂
wandb:                                            train/loss █▆▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁
wandb: 
wandb: Run summary:
wandb:                    eval/ego4d/dialog-klg-sum_val_loss 0.64232
wandb:                 eval/ego4d/dialog-klg-sum_val_runtime 3.1951
wandb:      eval/ego4d/dialog-klg-sum_val_samples_per_second 185.282
wandb:        eval/ego4d/dialog-klg-sum_val_steps_per_second 0.626
wandb:                         eval/ego4d/narration_val_loss 0.5159
wandb:                      eval/ego4d/narration_val_runtime 2.1717
wandb:           eval/ego4d/narration_val_samples_per_second 102.685
wandb:             eval/ego4d/narration_val_steps_per_second 0.46
wandb:                           eval/ego4d/summary_val_loss 1.14199
wandb:                        eval/ego4d/summary_val_runtime 2.8928
wandb:             eval/ego4d/summary_val_samples_per_second 194.965
wandb:               eval/ego4d/summary_val_steps_per_second 0.691
wandb:               eval/holoassist/dialog-klg-sum_val_loss 0.58994
wandb:            eval/holoassist/dialog-klg-sum_val_runtime 2.9833
wandb: eval/holoassist/dialog-klg-sum_val_samples_per_second 237.654
wandb:   eval/holoassist/dialog-klg-sum_val_steps_per_second 0.67
wandb:                      eval/sthsthv2/narration_val_loss 0.77904
wandb:                   eval/sthsthv2/narration_val_runtime 3.7612
wandb:        eval/sthsthv2/narration_val_samples_per_second 177.605
wandb:          eval/sthsthv2/narration_val_steps_per_second 0.532
wandb:                                            total_flos 3.162989674395295e+20
wandb:                                           train/epoch 4.0
wandb:                                     train/global_step 3252
wandb:                                       train/grad_norm 0.22679
wandb:                                   train/learning_rate 0.0
wandb:                                         train/lm_loss 0.18399
wandb:                                            train/loss 0.2765
wandb:                                            train_loss 0.4584
wandb:                                         train_runtime 8064.8498
wandb:                              train_samples_per_second 206.391
wandb:                                train_steps_per_second 0.403
wandb: 
wandb: 🚀 View run 20240821-L4096-I10-ep4-NOSEP-nr0.1-klgmix-1s-lora-bs512 at: https://wandb.ai/meta-intern-yichi/ProAct/runs/099a2e32-77a1-438d-8762-5f5fbff7af2c
wandb: ⭐️ View project at: https://wandb.ai/meta-intern-yichi/ProAct
wandb: Synced 5 W&B file(s), 0 media file(s), 3 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240821_102007-099a2e32-77a1-438d-8762-5f5fbff7af2c/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.