hf-internal-testing
/

test_dynamic_processor

Model card Files Files and versions

test_dynamic_processor / tokenization_fast.py

sgugger's picture

Initial commit

24b4f96 almost 4 years ago

203 Bytes

	from transformers import BertTokenizerFast
	from .tokenization import NewTokenizer

	class NewTokenizerFast(BertTokenizerFast):
	slow_tokenizer_class = NewTokenizer
	special_attribute_present = True