| /home/mshahidul/miniconda3/envs/verl/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. |
| import pynvml # type: ignore[import] |
| INFO 02-01 15:58:43 [__init__.py:216] Automatically detected platform cuda. |
| /home/mshahidul/miniconda3/envs/verl/lib/python3.12/site-packages/megatron/core/models/backends.py:21: UserWarning: Apex is not installed. Falling back to Torch Norm |
| warnings.warn("Apex is not installed. Falling back to Torch Norm") |
| /home/mshahidul/miniconda3/envs/verl/lib/python3.12/site-packages/megatron/core/optimizer/__init__.py:18: UserWarning: Transformer Engine and Apex are not installed. Falling back to Torch optimizers. |
| warnings.warn( |
| /home/mshahidul/miniconda3/envs/verl/lib/python3.12/site-packages/megatron/core/optimizer/optimizer.py:28: UserWarning: Transformer Engine and Apex are not installed. Falling back to local implementations of multi_tensor_applier and multi_tensor_scale |
| warnings.warn( |
| /home/mshahidul/miniconda3/envs/verl/lib/python3.12/site-packages/megatron/core/optimizer/clip_grads.py:29: UserWarning: Transformer Engine and Apex are not installed. Falling back to local implementations of multi_tensor_applier, multi_tensor_l2norm, and multi_tensor_scale |
| warnings.warn( |
| /home/mshahidul/miniconda3/envs/verl/lib/python3.12/site-packages/megatron/core/models/gpt/gpt_layer_specs.py:67: UserWarning: Apex is not installed. Falling back to Torch Norm |
| warnings.warn("Apex is not installed. Falling back to Torch Norm") |
| Could not override 'reward_model.reward_manager.name'. |
| To append to your config use +reward_model.reward_manager.name=base |
| Key 'name' is not in struct |
| full_key: reward_model.reward_manager.name |
| object_type=dict |
|
|
| Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. |
|
|
