---
title: Hindi BPE Tokenizer
colorFrom: blue
colorTo: red
sdk: streamlit
sdk_version: 1.31.1
app_file: app.py
pinned: false
---

# Hindi BPE Tokenizer

A Streamlit web application for encoding Hindi text to BPE tokens and decoding tokens back to text.

## Features

- Encode Hindi text to BPE tokens and token IDs
- Decode token IDs back to Hindi text
- Pre-trained on 5,000,000 lines of Hindi text
- Vocabulary size: 4,500 tokens
- Includes special tokens: `<pad>`, `<unk>`, `<s>`, `</s>`

## Usage

1. **Encoding**: Enter Hindi text in the left panel and click "Encode"
2. **Decoding**: Enter comma-separated token IDs in the right panel and click "Decode"

## Technical Details

- BPE (Byte Pair Encoding) tokenizer
- Trained on IndicCorp Hindi dataset
- Compression ratio > 3.2
- Preserves Hindi Unicode range (\\u0900-\\u097F)