niklassheth
/

tinystories-tokenizer

Model card Files Files and versions

tinystories-tokenizer / special_tokens_map.json

niklassheth's picture

Upload custom TinyStories BPE tokenizer (vocab_size=8192)

9d85d76 verified 7 months ago

history blame contribute delete

55 Bytes

	{
	"eos_token": "<\|eos\|>",
	"unk_token": "<\|unk\|>"
	}