Can anyone tell me why there are so many empty spaces tokens added to the tokenizer?
#98
by parikshit1619 - opened
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)
parikshit1619 changed discussion title from Can anyone why there are so many empty spaces tokens added to the tokenizer? to Can anyone tell me why there are so many empty spaces tokens added to the tokenizer?
Hello @parikshit1619 !
This was inherited from the Salesforce/codegen-350M-mono, which addressed one of the issues when tokenizing code: too many spaces and tabs. Hence, there are added tokens for representing different amounts of spaces and tabs, reducing the final amount of tokens per encoded data.
Regards,
Gustavo.
gugarosa changed discussion status to closed