huggingface/tokenizers: The current process just got forked
#2
by mcdominik - opened
Hi, I get this warning when I run the snippet in loop:
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
I was looking for solution here https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning and here https://github.com/huggingface/transformers/issues/5486
I set TOKENIZERS_PARALLELISM=false in my bashrc, but it did't help.
Is this warning means creating new tokenizer procces at every iteration and polluting my memory?
Any leads to the above question would be greatly appreciated. Thank you in advance!