controlmt-v2.3 / tokenizer_config.json

Fix HF auto_map (add tokenizer_config.json + list form); honest style framing + add curated eval scores

37b4c07 verified 16 days ago

322 Bytes

	{
	"tokenizer_class": "ControlMTTokenizer",
	"auto_map": {
	"AutoTokenizer": ["tokenization_controlmt.ControlMTTokenizer", null]
	},
	"model_max_length": 320,
	"bos_token": "<s>",
	"eos_token": "</s>",
	"unk_token": "<unk>",
	"pad_token": "<pad>",
	"clean_up_tokenization_spaces": false,
	"use_fast": false
	}