Tokenizer
#12
by
deleted - opened
@FancyZhao Is the tokenizer used for Yi similar to Llama's BPE Tokenizer, forget about multilingual tokens only for English?
@FancyZhao Is the tokenizer used for Yi similar to Llama's BPE Tokenizer, forget about multilingual tokens only for English?
Yes, they are both trained with sentencepiece BPE mode.
FancyZhao changed discussion status to
closed