Transformers
FlashFormerv1 / tokenizer_config.json

Commit History

FIX: Complete compilation of 32k vocabulary into tokenizer.json
f483284
verified

LastTransformer commited on

Initial deploy of optimized 32k code/math SentencePiece tokenizer architecture
11f7ba1
verified

LastTransformer commited on