error_type: ValueError

error_message: optimizer got an empty parameter list

traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 851, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 515, in main
    optimizer = torch.optim.AdamW(
                ^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/optim/adamw.py", line 37, in __init__
    super().__init__(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/optim/adam.py", line 101, in __init__
    super().__init__(params, defaults)
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/optim/optimizer.py", line 396, in __init__
    raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list


traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 854, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 561, in main
    optimizer = torch.optim.AdamW(
                ^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/optim/adamw.py", line 37, in __init__
    super().__init__(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/optim/adam.py", line 101, in __init__
    super().__init__(params, defaults)
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/optim/optimizer.py", line 396, in __init__
    raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list


error_type: UnboundLocalError

traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 862, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 810, in main
    eval_loss = eval_metrics["eval_loss"]
                ^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'eval_metrics' where it is not associated with a value


error_type: OutOfMemoryError

traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 883, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 464, in main
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5048, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5468, in _load_pretrained_model
    _error_msgs, disk_offload_index = load_shard_file(args)
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 843, in load_shard_file
    disk_offload_index = _load_state_dict_into_meta_model(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 748, in _load_state_dict_into_meta_model
    param = param[...]
            ~~~~~^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.16 GiB. GPU 2 has a total capacity of 139.81 GiB of which 412.12 MiB is free. Process 911021 has 137.82 GiB memory in use. Including non-PyTorch memory, this process has 1.47 GiB memory in use. Of the allocated memory 512 bytes is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)


traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 883, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 464, in main
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5048, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5468, in _load_pretrained_model
    _error_msgs, disk_offload_index = load_shard_file(args)
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 843, in load_shard_file
    disk_offload_index = _load_state_dict_into_meta_model(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 748, in _load_state_dict_into_meta_model
    param = param[...]
            ~~~~~^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.16 GiB. GPU 1 has a total capacity of 139.81 GiB of which 496.12 MiB is free. Process 911020 has 137.85 GiB memory in use. Including non-PyTorch memory, this process has 1.47 GiB memory in use. Of the allocated memory 512 bytes is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)


traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 883, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 464, in main
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5048, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5468, in _load_pretrained_model
    _error_msgs, disk_offload_index = load_shard_file(args)
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 843, in load_shard_file
    disk_offload_index = _load_state_dict_into_meta_model(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/transformers/modeling_utils.py", line 748, in _load_state_dict_into_meta_model
    param = param[...]
            ~~~~~^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.16 GiB. GPU 3 has a total capacity of 139.81 GiB of which 20.12 MiB is free. Process 911022 has 138.21 GiB memory in use. Including non-PyTorch memory, this process has 1.47 GiB memory in use. Of the allocated memory 512 bytes is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)


traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 883, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 547, in main
    model = get_peft_model(model, lora_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/mapping_func.py", line 125, in get_peft_model
    return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/peft_model.py", line 1815, in __init__
    super().__init__(model, peft_config, adapter_name, **kwargs)
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/peft_model.py", line 130, in __init__
    self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 209, in __init__
    self.inject_adapter(self.model, adapter_name, low_cpu_mem_usage=low_cpu_mem_usage, state_dict=state_dict)
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 578, in inject_adapter
    self._create_and_replace(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/tuners/lora/model.py", line 259, in _create_and_replace
    new_module = self._create_new_module(lora_config, adapter_name, target, device_map=device_map, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/peft/tuners/lora/model.py", line 371, in _create_new_module
    raise ValueError(
ValueError: Target module Glm4MoeTopkRouter() is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv1d`, `torch.nn.Conv2d`, `torch.nn.Conv3d`, `transformers.pytorch_utils.Conv1D`, `torch.nn.MultiheadAttention.`.


traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 883, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 563, in main
    model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
                                                          ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1559, in prepare
    result = tuple(
             ^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1560, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1402, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1895, in prepare_model
    model = FSDP(model, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 468, in __init__
    _auto_wrap(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py", line 101, in _auto_wrap
    _recursive_wrap(**recursive_wrap_kwargs, **root_kwargs)  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 533, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
                                        ^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 533, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
                                        ^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 533, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
                                        ^^^^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 551, in _recursive_wrap
    return _wrap(module, wrapper_cls, **kwargs), nonwrapped_numel
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 480, in _wrap
    return wrapper_cls(module, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 494, in __init__
    _init_param_handle_from_module(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py", line 604, in _init_param_handle_from_module
    state.compute_device = _get_compute_device(
                           ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py", line 1068, in _get_compute_device
    raise ValueError(
ValueError: Inconsistent compute device and `device_id` on rank 2: cuda:0 vs cuda:2


traceback:
Traceback (most recent call last):
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 883, in <module>
    main()
  File "/workspace/Avinash/CPT_GLM/train_qlora_improved.py", line 563, in main
    model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
                                                          ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1559, in prepare
    result = tuple(
             ^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1560, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1402, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/accelerate/accelerator.py", line 1895, in prepare_model
    model = FSDP(model, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 468, in __init__
    _auto_wrap(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py", line 101, in _auto_wrap
    _recursive_wrap(**recursive_wrap_kwargs, **root_kwargs)  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 533, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
                                        ^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 533, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
                                        ^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 533, in _recursive_wrap
    wrapped_child, num_wrapped_params = _recursive_wrap(
                                        ^^^^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 551, in _recursive_wrap
    return _wrap(module, wrapper_cls, **kwargs), nonwrapped_numel
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py", line 480, in _wrap
    return wrapper_cls(module, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 494, in __init__
    _init_param_handle_from_module(
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py", line 604, in _init_param_handle_from_module
    state.compute_device = _get_compute_device(
                           ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/Avinash/ash_env/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py", line 1068, in _get_compute_device
    raise ValueError(
ValueError: Inconsistent compute device and `device_id` on rank 3: cuda:0 vs cuda:3