|
|
--- |
|
|
|
|
|
|
|
|
license: mit |
|
|
library_name: tokenizers |
|
|
tags: |
|
|
- tokenizer |
|
|
- bpe |
|
|
- byte-level |
|
|
- multilingual |
|
|
- code |
|
|
- unitronx |
|
|
--- |
|
|
|
|
|
# {DISPLAY_NAME} |
|
|
|
|
|
**UnitronX** is a 32k byte-level BPE tokenizer optimized for English, multilingual (ru/ar/de/es/fr/it/cs/hr/sr), and code. |
|
|
It enforces safe merge boundaries (script changes, ZWJ, letter↔digit), preserves code identifiers, and uses |
|
|
placeholder tokens for URLs/emails/paths/hashes/UUIDs/handles/hashtags. |
|
|
|
|
|
## Files |
|
|
- `tokenizer.json`, `merges.txt`, `vocab.json` |
|
|
- `tokenizer_config.json`, `special_tokens_map.json` |
|
|
- `meta.json` |
|
|
- *(optional)* `unitronx.tiktoken.json` (tiktoken-compatible) |
|
|
|
|
|
## Load with Transformers |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer |
|
|
tok = AutoTokenizer.from_pretrained("UnitronX-Tokenizer-32k-v1") |
|
|
print(tok.encode("don't split hyphen-words or fooBar123_id in code!")) |
|
|
|
