cjvt
/

gpt-sl-base

Text Generation

text-generation-inference

Model card Files Files and versions

gpt-sl-base / tokenizer_config.json

matejulcar's picture

first version

9fbec62 about 3 years ago

history blame contribute delete

388 Bytes

	{
	"bos_token": "<BOS>",
	"cls_token": "<BOS>",
	"eos_token": "<EOS>",
	"name_or_path": "/home/mulcar/bigscience/gpt3-sl-tok-large.60k-bytelevel",
	"pad_token": "<PAD>",
	"sep_token": "<EOS>",
	"special_tokens_map_file": "/home/mulcar/bigscience/gpt3-sl-tok-large.60k-bytelevel/special_tokens_map.json",
	"tokenizer_class": "PreTrainedTokenizerFast",
	"model_max_length": 1024
	}