---
language:
- bn
license: apache-2.0
tags:
- tokenizer
- bangla
- bengali
---

# Custom Bangla Tokenizer

A specialized Bangla/Bengali tokenizer extracted from multilingual models and extended with missing characters.

## Features

- Focused on Bangla text tokenization
- Support for characters like ঢ় and other missing characters
- Reduced vocabulary size: 21,607 tokens
- Compatible with Hugging Face Transformers

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("yasserius/bangla-tokenizer")

# Tokenize Bangla text
text = "আমি বাংলায় কথা বলি"
tokens = tokenizer.tokenize(text)
print(tokens)
```

## Model Details

- Base model: Extracted from google/muril-base-cased
- Language: Bengali/Bangla
- Vocabulary size: 21,607