tokenization-trial / README.md
ClaudeHu05's picture
Create README.md
9ef065d verified
This model repository will only contain files needed for tokenization (`vocab.json`, `tokenizer_config.sjon`, etc). The goal is to get familiar with huggingface tokenizers
and test tokenizers for tabs.