Missing tokens
#2
by Nguyen-BangVu - opened
Hello,
Thanks for doing this project. I was planning to use your tokenizer.
However, I just took a closer look at all the vocab you have. It is missing the following tokens (esp, rip, ymm0, zmm0, xmm0, je, jne, ymmword) . Is there a reason for that?