autosync step 90
dcccba0 verified - huggingface autosync step 90
extra_state_world_size_8_rank_0.pt Detected Pickle imports (7)
- "torch.ByteStorage",
- "collections.OrderedDict",
- "numpy.dtype",
- "torch._utils._rebuild_tensor_v2",
- "numpy.ndarray",
- "numpy._core.multiarray._reconstruct",
- "_codecs.encode"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_1.pt Detected Pickle imports (7)
- "torch._utils._rebuild_tensor_v2",
- "torch.ByteStorage",
- "collections.OrderedDict",
- "numpy.dtype",
- "numpy._core.multiarray._reconstruct",
- "numpy.ndarray",
- "_codecs.encode"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_2.pt Detected Pickle imports (7)
- "torch._utils._rebuild_tensor_v2",
- "numpy._core.multiarray._reconstruct",
- "numpy.ndarray",
- "collections.OrderedDict",
- "torch.ByteStorage",
- "_codecs.encode",
- "numpy.dtype"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_3.pt Detected Pickle imports (7)
- "numpy._core.multiarray._reconstruct",
- "_codecs.encode",
- "collections.OrderedDict",
- "torch.ByteStorage",
- "torch._utils._rebuild_tensor_v2",
- "numpy.dtype",
- "numpy.ndarray"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_4.pt Detected Pickle imports (7)
- "collections.OrderedDict",
- "numpy.dtype",
- "torch.ByteStorage",
- "_codecs.encode",
- "numpy._core.multiarray._reconstruct",
- "torch._utils._rebuild_tensor_v2",
- "numpy.ndarray"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_5.pt Detected Pickle imports (7)
- "numpy._core.multiarray._reconstruct",
- "_codecs.encode",
- "collections.OrderedDict",
- "torch.ByteStorage",
- "torch._utils._rebuild_tensor_v2",
- "numpy.dtype",
- "numpy.ndarray"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_6.pt Detected Pickle imports (7)
- "numpy.ndarray",
- "numpy.dtype",
- "_codecs.encode",
- "numpy._core.multiarray._reconstruct",
- "torch._utils._rebuild_tensor_v2",
- "torch.ByteStorage",
- "collections.OrderedDict"
How to fix it?
15.1 kB autosync step 90 extra_state_world_size_8_rank_7.pt Detected Pickle imports (7)
- "torch.ByteStorage",
- "_codecs.encode",
- "torch._utils._rebuild_tensor_v2",
- "numpy.ndarray",
- "collections.OrderedDict",
- "numpy.dtype",
- "numpy._core.multiarray._reconstruct"
How to fix it?
15.1 kB autosync step 90 model_world_size_8_rank_0.pt Detected Pickle imports (15)
- "torch.device",
- "collections.OrderedDict",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch.float32",
- "torch.serialization._get_layout",
- "torch._utils._rebuild_tensor_v2",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch._tensor._rebuild_from_type_v2",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.Size",
- "torch.IntStorage",
- "torch.distributed.tensor.DTensor",
- "torch.FloatStorage",
- "torch.distributed.tensor.placement_types.Shard",
- "torch._utils._rebuild_wrapper_subclass"
How to fix it?
4.15 GB autosync step 90 model_world_size_8_rank_1.pt Detected Pickle imports (15)
- "torch.device",
- "collections.OrderedDict",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch.float32",
- "torch.serialization._get_layout",
- "torch._utils._rebuild_tensor_v2",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch._tensor._rebuild_from_type_v2",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.Size",
- "torch.IntStorage",
- "torch.distributed.tensor.DTensor",
- "torch.FloatStorage",
- "torch.distributed.tensor.placement_types.Shard",
- "torch._utils._rebuild_wrapper_subclass"
How to fix it?
4.15 GB autosync step 90 model_world_size_8_rank_2.pt Detected Pickle imports (15)
- "collections.OrderedDict",
- "torch.Size",
- "torch.float32",
- "torch._tensor._rebuild_from_type_v2",
- "torch.FloatStorage",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch._utils._rebuild_wrapper_subclass",
- "torch.serialization._get_layout",
- "torch.device",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch.IntStorage",
- "torch.distributed.tensor.DTensor",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
4.15 GB autosync step 90 - 4.15 GB autosync step 90
model_world_size_8_rank_4.pt Detected Pickle imports (15)
- "torch._tensor._rebuild_from_type_v2",
- "torch._utils._rebuild_wrapper_subclass",
- "collections.OrderedDict",
- "torch.device",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.float32",
- "torch.distributed.tensor.DTensor",
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "torch.serialization._get_layout",
- "torch.FloatStorage",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch.Size",
- "torch.distributed.device_mesh.DeviceMesh"
How to fix it?
4.15 GB autosync step 90 model_world_size_8_rank_5.pt Detected Pickle imports (15)
- "collections.OrderedDict",
- "torch.Size",
- "torch.float32",
- "torch._tensor._rebuild_from_type_v2",
- "torch.FloatStorage",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch._utils._rebuild_wrapper_subclass",
- "torch.serialization._get_layout",
- "torch.device",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch.IntStorage",
- "torch.distributed.tensor.DTensor",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
4.15 GB autosync step 90 model_world_size_8_rank_6.pt Detected Pickle imports (15)
- "collections.OrderedDict",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.serialization._get_layout",
- "torch.FloatStorage",
- "torch.distributed.tensor.DTensor",
- "torch.device",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.Size",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "torch.float32",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch._tensor._rebuild_from_type_v2",
- "torch._utils._rebuild_wrapper_subclass"
How to fix it?
4.15 GB autosync step 90 model_world_size_8_rank_7.pt Detected Pickle imports (15)
- "torch._tensor._rebuild_from_type_v2",
- "torch._utils._rebuild_wrapper_subclass",
- "collections.OrderedDict",
- "torch.device",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.float32",
- "torch.distributed.tensor.DTensor",
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "torch.serialization._get_layout",
- "torch.FloatStorage",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch.Size",
- "torch.distributed.device_mesh.DeviceMesh"
How to fix it?
4.15 GB autosync step 90 optim_world_size_8_rank_0.pt Detected Pickle imports (15)
- "collections.OrderedDict",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.serialization._get_layout",
- "torch.FloatStorage",
- "torch.distributed.tensor.DTensor",
- "torch.device",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.Size",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "torch.float32",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch._tensor._rebuild_from_type_v2",
- "torch._utils._rebuild_wrapper_subclass"
How to fix it?
8.29 GB autosync step 90 - 8.29 GB autosync step 90
optim_world_size_8_rank_2.pt Detected Pickle imports (15)
- "collections.OrderedDict",
- "torch.Size",
- "torch.float32",
- "torch._tensor._rebuild_from_type_v2",
- "torch.FloatStorage",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch._utils._rebuild_wrapper_subclass",
- "torch.serialization._get_layout",
- "torch.device",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch.IntStorage",
- "torch.distributed.tensor.DTensor",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch._utils._rebuild_tensor_v2"
How to fix it?
8.29 GB autosync step 90 optim_world_size_8_rank_3.pt Detected Pickle imports (15)
- "torch._tensor._rebuild_from_type_v2",
- "torch._utils._rebuild_wrapper_subclass",
- "collections.OrderedDict",
- "torch.device",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.float32",
- "torch.distributed.tensor.DTensor",
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "torch.serialization._get_layout",
- "torch.FloatStorage",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch.Size",
- "torch.distributed.device_mesh.DeviceMesh"
How to fix it?
8.29 GB autosync step 90 - 8.29 GB autosync step 90
optim_world_size_8_rank_5.pt Detected Pickle imports (15)
- "collections.OrderedDict",
- "torch.distributed.tensor.placement_types.Shard",
- "torch.serialization._get_layout",
- "torch.FloatStorage",
- "torch.distributed.tensor.DTensor",
- "torch.device",
- "torch.distributed.tensor._dtensor_spec.DTensorSpec",
- "torch.Size",
- "torch.distributed.device_mesh.DeviceMesh",
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "torch.float32",
- "torch.distributed.tensor._dtensor_spec.TensorMeta",
- "torch._tensor._rebuild_from_type_v2",
- "torch._utils._rebuild_wrapper_subclass"
How to fix it?
8.29 GB autosync step 90 - 8.29 GB autosync step 90
- 8.29 GB autosync step 90
