Dataloader multiprocessing error
#28
by Atsunori - opened
Dataloader multiprocessing error sometimes occurs, when we use model._do_encode(). This error comes from num_workers=32 (https://huggingface.co/nvidia/NV-Embed-v1/blob/main/modeling_nvembed.py#L371). So it may be better to change that to an argument.
Thanks for reporting this issue. We added the num_workers as an argument for _do_encode(). The example is updated in this model card: https://huggingface.co/nvidia/NV-Embed-v1.