File size: 397 Bytes
82140a5 346111b 82140a5 346111b 82140a5 b466b9a |
1 2 3 4 5 6 7 8 9 10 11 12 13 |
---
library_name: transformers
tags: []
---
# SuperBPE tokenizer
SuperBPE tokenizer trained on mix of code + edu data
"Big cat sat on a mat" -> ['Big', 'Ġcat', 'Ġsat', 'ĠonĠa', 'Ġmat']
"this function does summation on a last axis and then returns a scalar." -> ['this', 'Ġfunction', 'Ġdoes', 'Ġsummation', 'ĠonĠa', 'Ġlast', 'Ġaxis', 'ĠandĠthen', 'ĠreturnsĠa', 'Ġscalar', '.']
|