added_tokens.json · pharaouk/fusedyi at main

fusedyi / added_tokens.json

pharaouk's picture

Upload tokenizer

f45123c verified about 2 years ago

history blame contribute delete

57 Bytes

	{
	"<\|endoftext\|>": 64001,
	"<\|startoftext\|>": 64000
	}