Hugging Face's logo

sdadas
/
polish-longformer-base-4096

Fill-Mask
Transformers
PyTorch
Safetensors
Polish
longformer
Model card Files
xet
 Community
2

How to use this model for tokenization?

#1
by tprochenka - opened
Discussion
tprochenka

Hi I tried to do tokenization:
tokenizer = LongformerTokenizer.from_pretrained("sdadas/polish-longformer-base-4096")
I got an error that vocab_file is not found. Indeed, I see that there is no vocab.json, instead I see tokanizer.json. Could you please share a snippet showing how to do tokenization using your model?

Thanks!
sdadas
 Owner

Hi, the model supports fast tokenizer format only. Use LongformerTokenizerFast instead of LongformerTokenizer:

from transformers import LongformerTokenizerFast
tokenizer = LongformerTokenizerFast.from_pretrained("sdadas/polish-longformer-base-4096")
encoded = tokenizer("Za偶贸艂ci膰 g臋艣l膮 ja藕艅.")
print(encoded.input_ids)
tprochenka

Thanks for a quick answer, it works :)
tprochenka changed discussion status to closed

Sign up or log in to comment