arcade100k / tokenizer_config.json

Commit History

fix: make `eos_token`/`pad_token` overridable
dfe1f6f

jon-tow commited on

Create tokenizer_config.json
af51a49

jon-tow commited on

Delete tokenizer_config.json
beeb2b0

jon-tow commited on

Upload tokenizer
f3e546a

jon-tow commited on

init: first commit
aceea5b

jon-tow commited on