| title: Custom Added Token Counter | |
| sdk: gradio | |
| sdk_version: 6.16.0 | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| # Custom added token counter | |
| A Gradio app for counting raw text tokens with a configurable tokeniser while | |
| treating configurable added tokens as single tokens. | |
| The default added graph tags are: | |
| `<|graph|>`, `<|nodes|>`, `<|node|>`, `<|id|>`, `<|label|>`, | |
| `<|bbox|>`, `<|conf|>`, `<|attrs|>`, `<|edges|>`, `<|edge|>`, | |
| `<|src|>`, `<|pred|>`, `<|tgt|>`, `<|end_graph|>`. | |
| ## Local Run | |
| Run locally: | |
| ```sh | |
| python3 -m pip install -r requirements.txt | |
| python3 app.py | |
| ``` | |