mutopia_guitar_dataset_tokenizer / tokenizer_config.json
juancopi81's picture
add tokenizer
5bf268e
raw
history blame contribute delete
115 Bytes
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"tokenizer_class": "PreTrainedTokenizerFast"
}