README.md

metadata

title: BPETokenizer
colorFrom: blue
colorTo: blue
sdk: streamlit
sdk_version: 1.42.2
app_file: app.py
pinned: false

English BPE Tokenizer Visualizer

This app demonstrates how BPE (Byte Pair Encoding) tokenization works on English text. Enter any text to see how it gets broken down into tokens, with a color-coded visualization of the tokenization process.�