Transformers

Commit History

FIX: Complete compilation of 32k vocabulary into tokenizer.json
c5ecc2b
verified

LastTransformer commited on

FIX: Complete compilation of 32k vocabulary into tokenizer.json
f483284
verified

LastTransformer commited on

Initial deploy of optimized 32k code/math SentencePiece tokenizer architecture
11f7ba1
verified

LastTransformer commited on