facebook

/

wmt19-ru-en

like 21

Follow

AI at Meta 12.9k

tokenizer = FSMTTokenizer.from_pretrained(r'wmt')
s = tokenizer .encode("isn't it?")
s = tokenizer.decode(s)
print(s)
isn t it?
this bug cause by sacremoses.MosesTokenizer(lang=lang).tokenize(escape=True)