| library_name: transformers | |
| tags: [] | |
| # SuperBPE tokenizer | |
| SuperBPE tokenizer trained on mix of code + edu data | |
| "Big cat sat on a mat" -> ['Big', 'Ġcat', 'Ġsat', 'ĠonĠa', 'Ġmat'] | |
| "this function does summation on a last axis and then returns a scalar." -> ['this', 'Ġfunction', 'Ġdoes', 'Ġsummation', 'ĠonĠa', 'Ġlast', 'Ġaxis', 'ĠandĠthen', 'ĠreturnsĠa', 'Ġscalar', '.'] | |