| ---
|
| language:
|
| - en
|
| - multilingual
|
| - code
|
| license: apache-2.0
|
| tags:
|
| - tokenizer
|
| - bpe
|
| - omni
|
| - binary-analysis
|
| ---
|
|
|
| # Omni-Tokenizer (Experimental 539k Fusion)
|
|
|
| This is a highly experimental, massive **539,306-token Byte-Level BPE tokenizer** designed for extreme-scale "Omni" models. It is built to seamlessly process natural languages, highly dense code, and raw binary/machine executables natively.
|
|
|
| ## Composition
|
| This tokenizer was created by extracting and perfectly fusing the vocabularies of several state-of-the-art tokenizers into a single LLaMA-3 base:
|
| 1. `meta-llama/Meta-Llama-3-8B` (Base Byte-Level Foundation)
|
| 2. `google/gemma-7b`
|
| 3. `Qwen/Qwen1.5-7B`
|
| 4. `CohereForAI/c4ai-command-r-v01`
|
| 5. `microsoft/Phi-3-mini-4k-instruct`
|
| 6. `mjbommar/binary-tokenizer-001-64k` (Binary Analysis / Malware)
|
| 7. `mjbommar/binary-tokenizer-001-32k`
|
|
|
| ## Technical Details
|
| - **Vocabulary Size:** 539,306
|
| - **Base Architecture:** Byte-Level BPE (No Unknown Tokens)
|
| - **Use Cases:** Multilingual NLP, Code Generation, Binary/Malware Analysis, Reverse Engineering.
|
|
|
| **Note on Usage:** Due to the massive 540k vocabulary size, this tokenizer will create an embedding matrix of roughly ~2.2 Billion parameters (at 4096 dimensions). It is intended for large-scale experimental models where extreme compression and cross-domain tokenization is required.
|
|
|
| ## How to use
|
| ```python
|
| from transformers import AutoTokenizer
|
|
|
| tokenizer = AutoTokenizer.from_pretrained("SurendraVB/omni-tokenizer")
|
| ```
|
|
|
