README.md

metadata

title: Custom Added Token Counter
sdk: gradio
sdk_version: 6.16.0
app_file: app.py
pinned: false
license: mit

Custom added token counter

A Gradio app for counting raw text tokens with a configurable tokeniser while treating configurable added tokens as single tokens.

The default added graph tags are:

<|graph|>, <|nodes|>, <|node|>, <|id|>, <|label|>, <|bbox|>, <|conf|>, <|attrs|>, <|edges|>, <|edge|>, <|src|>, <|pred|>, <|tgt|>, <|end_graph|>.

Local Run

Run locally:

python3 -m pip install -r requirements.txt
python3 app.py