romizone
/

bpe-tokenizer-id

Token Classification

bahasa-indonesia

text-processing

subword-tokenization

Model card Files Files and versions

bpe-tokenizer-id / special_tokens_map.json

romizone's picture

Upload BPE Tokenizer Bahasa Indonesia

fbf2c3e verified 24 days ago

history blame contribute delete

98 Bytes

	{
	"pad_token": "<PAD>",
	"unk_token": "<UNK>",
	"bos_token": "<BOS>",
	"eos_token": "<EOS>"
	}