README.md

metadata

title: Hindi BPE Tokenizer
colorFrom: blue
colorTo: red
sdk: streamlit
sdk_version: 1.31.1
app_file: app.py
pinned: false

Hindi BPE Tokenizer

A Streamlit web application for encoding Hindi text to BPE tokens and decoding tokens back to text.

Features

• Encode Hindi text to BPE tokens and token IDs
• Decode token IDs back to Hindi text
• Pre-trained on 5,000,000 lines of Hindi text
• Vocabulary size: 4,500 tokens
• Includes special tokens: <pad>, <unk>, <s>, </s>

Usage

1. Encoding: Enter Hindi text in the left panel and click "Encode"
2. Decoding: Enter comma-separated token IDs in the right panel and click "Decode"

Technical Details

• BPE (Byte Pair Encoding) tokenizer
• Trained on IndicCorp Hindi dataset
• Compression ratio > 3.2
• Preserves Hindi Unicode range (\u0900-\u097F)