--- language: - code tags: - tokenizer - binary-analysis - binary-tokenization - bpe - byte-pair-encoding - reverse-engineering - malware-analysis - cybersecurity - executable-analysis license: mit pipeline_tag: feature-extraction library_name: tokenizers --- # binary-tokenizer-001-8k A cross-platform BPE tokenizer for binary executables and machine code. Trained on 13 GB of diverse binaries spanning Linux, Windows, macOS, and Android platforms. **🔗 Model**: [`mjbommar/binary-tokenizer-001-8k`](https://huggingface.co/mjbommar/binary-tokenizer-001-8k) **📊 Dataset**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized) **📄 Paper**: *Binary BPE: Cross-Platform Tokenization for Binary Analysis* (arXiv preprint coming soon) ## Overview - **Vocabulary Size**: 8,192 tokens (2^13) - **Token Composition**: 256 base bytes + 7,929 learned merges + 7 special tokens - **Average Token Length**: 3.312 bytes - **3-byte Instructions**: 21.7% of vocabulary (1,774 tokens) - **Compression Ratio**: ~2.2 bytes/token on typical binaries --- ## Training Configuration **Training Corpus**: - Source: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized) - Size: ~13 GB - Files: 30,738 binary files - Platforms: Linux (ELF), Windows (PE), macOS (Mach-O), Android (APK) - Architectures: x86-64, x86, ARM64, ARM, MIPS, RISC-V **Training Parameters**: - Vocabulary size: 8,192 (including 7 special tokens) - Min frequency: 10 - Chunk size: 8,192 bytes - Allowed lengths: DEFAULT (1-16 bytes) - Training duration: ~2-3 hours --- ## Vocabulary Statistics **Composition**: - Base bytes (0-255): 256 tokens - Learned merges: 7,929 tokens - Special tokens: 7 tokens (`<|start|>`, `<|end|>`, `<|pad|>`, `<|unk|>`, `<|cls|>`, `<|sep|>`, `<|mask|>`) - **Total**: 8,192 tokens **Quality Metrics**: - All tokens reachable: ✓ Yes - Valid merges: 7,929 / 7,929 - Power-of-2 size: ✓ Yes (2^13) --- ## Token Length Distribution | Length | Count | Percentage | Description | |--------|-------|------------|-------------| | 1 byte | 256 | 3.1% | Base bytes | | 2 bytes | 3,659 | 44.7% | Byte pairs | | 3 bytes | 1,774 | 21.7% | Complete x86-64 instructions | | 4 bytes | 1,464 | 17.9% | Instructions with operands | | 5 bytes | 290 | 3.5% | Complex patterns | | 6 bytes | 245 | 3.0% | Complex patterns | | 7 bytes | 96 | 1.2% | Complex patterns | | 8 bytes | 146 | 1.8% | Complex patterns | | 9+ bytes | 262 | 3.2% | Long patterns | **Average Token Length**: 3.312 bytes --- ## Byte Content Analysis **Content Categories**: - Contains NULL byte (0x00): 2,145 tokens (26.2%) - ASCII printable (0x20-0x7E): 1,831 tokens (22.4%) - All ASCII (<0x80): 3,810 tokens (46.5%) - High bytes (≥0x80): 4,375 tokens (53.5%) **Most Common Bytes in Tokens**: - `0x00` (NULL): 5,087 occurrences - Padding and alignment - `0xFF`: 829 occurrences - Sentinel values - `0x48` (REX.W): 684 occurrences - x86-64 REX prefix - `0x8B` (MOV): 505 occurrences - x86-64 MOV opcode - `0xCC` (INT3): 347 occurrences - Debug breakpoint padding --- ## Sequence Coverage **N-byte Sequence Diversity**: | Length | Learned Tokens | Possible Sequences | Coverage | |--------|----------------|-------------------|----------| | 1-byte | 256 | 256 | 100.00% | | 2-byte | 3,659 | 65,536 | 5.58% | | 3-byte | 1,774 | 16,777,216 | 0.011% | | 4-byte | 1,464 | 4,294,967,296 | 0.000034% | --- ## Files - `tokenizer-8192.json` - Trained tokenizer model (592 KB) - `analysis_results.json` - Detailed analysis statistics - `training.log` - Training output log (if available) - `training_stats.txt` - Training summary (if available) --- ## Usage **Load from HuggingFace Hub**: ```python from tokenizers import Tokenizer # Load directly from HuggingFace tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-8k") ``` **Load from local file**: ```bash # With bbpe CLI bbpe encode --tokenizer tokenizer-8192.json /path/to/binary bbpe info tokenizer-8192.json ``` **Complete Python Example**: ```python from tokenizers import Tokenizer # Load from HuggingFace or local file tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-8k") # OR: tokenizer = Tokenizer.from_file("tokenizer-8192.json") # Read binary file and decode as latin-1 (preserves all byte values 0-255) with open("/usr/bin/ls", "rb") as f: data = f.read() data_str = data.decode("latin-1") # Encode the binary data encoding = tokenizer.encode(data_str) print(f"File size: {len(data)} bytes") print(f"Total tokens: {len(encoding.ids)}") print(f"Compression: {len(data) / len(encoding.ids):.3f} bytes/token") # First 10 tokens for i, (token_id, token) in enumerate(zip(encoding.ids[:10], encoding.tokens[:10])): token_bytes = token.encode("latin-1") print(f" Token {i}: ID={token_id:5d} hex={token_bytes.hex():20s} ({len(token_bytes)} bytes)") # Decode tokens back to bytes decoded_str = tokenizer.decode(encoding.ids) decoded_bytes = decoded_str.encode("latin-1") assert decoded_bytes == data # Perfect reconstruction ``` **Example output for `/usr/bin/ls` (142,312 bytes)**: ``` File size: 142312 bytes Total tokens: 65458 Compression: 2.174 bytes/token First 10 tokens: Token 0: ID= 127 hex=7f (1 bytes) Token 1: ID= 7652 hex=454c (2 bytes) Token 2: ID= 70 hex=46 (1 bytes) Token 3: ID= 2 hex=02 (1 bytes) Token 4: ID= 772 hex=0101 (2 bytes) Token 5: ID= 1332 hex=000000000000000000 (9 bytes) Token 6: ID= 531 hex=0300 (2 bytes) Token 7: ID= 2802 hex=3e00 (2 bytes) Token 8: ID= 556 hex=01000000 (4 bytes) Token 9: ID= 48 hex=30 (1 bytes) Decoded: 7f454c4602010100000000000000000003003e000100000030... (ELF header: 7f 45 4c 46 = ELF magic bytes) ``` --- ## Citation If you use this tokenizer in your research, please cite: ```bibtex @article{bommarito2025binarybpe, title={Binary BPE: Cross-Platform Tokenization for Binary Analysis}, author={Bommarito II, Michael J.}, journal={arXiv preprint}, year={2025}, note={Preprint coming soon} } ``` **Author**: Michael J. Bommarito II ([michael.bommarito@gmail.com](mailto:michael.bommarito@gmail.com)) --- **Generated**: November 12, 2025 **Training Script**: `train_tokenizers.sh` **Analysis Script**: `analyze_tokenizer.py`