openeurollm
/

tokenizer-128k

european-languages

Model card Files Files and versions

tokenizer-128k / added_tokens.json

timpal0l's picture

Upload 128k vocab SentencePiece BPE tokenizer (trained on 175GB)

43f50d5 verified 2 months ago

history blame contribute delete

22 Bytes

	{
	"<pad>": 131072
	}