Xenova
/

CodeGPT-tokenizer

Model card Files Files and versions

CodeGPT-tokenizer / added_tokens.json

Xenova's picture

Xenova HF Staff

Upload 6 files

d496768 verified over 1 year ago

history blame contribute delete

45 Bytes

	{
	"<EOL>": 50000,
	"<\|UNKNOWN\|>": 50001
	}