rwillh11

/

mdeberta_NLI_policy_noContext

like 0

Updated tokenizer.json to resolve compatibility issues with recent versions of the tokenizers library.

Changes:
Line 84: Changed "prepend_scheme": "always" to "add_prefix_space": true in pre_tokenizer
Line 167: Changed "prepend_scheme": "always" to "add_prefix_space": true in decoder

Issue: A community member reported a compatibility error on our stance detection model when using newer versions of the transformers library. Since this model uses the same base tokenizer (MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7), it contains the same deprecated tokenizer configuration that may cause issues for users with recent library versions.

Root cause: The field prepend_scheme: "always" is deprecated and not recognized by the Rust tokenizers backend in newer library versions.