Breaking change in tokenizers 0.23

#4
by aimgo - opened

In tokenizers==0.23 BERTProcessing signature changes, causing the tokenizer to fail

    tokenizer.post_processor = processors.BertProcessing(
                               ~~~~~~~~~~~~~~~~~~~~~~~~~^
        sep=("[SEP]", vocab["[SEP]"]),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        cls=("[CLS]", vocab["[CLS]"]),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
TypeError: BertProcessing.__new__() got an unexpected keyword argument 'cls'

I believe this can be remedied by changing the call in tokenization_latin_bert_fast.py to:

tokenizer.post_processor = processors.BertProcessing(
    ("[SEP]", vocab["[SEP]"]),
    ("[CLS]", vocab["[CLS]"]),
)
LatinCy org

v.1.2.1 released to fix the reported error; cf. https://huggingface.co/latincy/latin-bert#v121. Thanks to @aimgo for catching this.

diyclassics changed discussion status to closed

Sign up or log in to comment