README.md

---
language:
- code
tags:
- tokenizer
- binary-analysis
- binary-tokenization
- bpe
- byte-pair-encoding
- reverse-engineering
- malware-analysis
- cybersecurity
- executable-analysis
license: mit
pipeline_tag: feature-extraction
library_name: tokenizers
---

# binary-tokenizer-001-4k

A cross-platform BPE tokenizer for binary executables and machine code. Trained on 13 GB of diverse binaries spanning Linux, Windows, macOS, and Android platforms.

**🔗 Model**: [`mjbommar/binary-tokenizer-001-4k`](https://huggingface.co/mjbommar/binary-tokenizer-001-4k)
**📊 Dataset**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
**📄 Paper**: *Binary BPE: Cross-Platform Tokenization for Binary Analysis* (arXiv preprint coming soon)

## Overview

- **Vocabulary Size**: 4,096 tokens (2^12)
- **Token Composition**: 256 base bytes + 3,833 learned merges + 7 special tokens
- **Average Token Length**: 3.000 bytes
- **3-byte Instructions**: 20.6% of vocabulary (841 tokens)
- **Compression Ratio**: ~2.0 bytes/token on typical binaries

---

## Training Configuration

**Training Corpus**:
- Source: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
- Size: ~13 GB
- Files: 30,738 binary files
- Platforms: Linux (ELF), Windows (PE), macOS (Mach-O), Android (APK)
- Architectures: x86-64, x86, ARM64, ARM, MIPS, RISC-V

**Training Parameters**:
- Vocabulary size: 4,096 (including 7 special tokens)
- Min frequency: 10
- Chunk size: 8,192 bytes
- Allowed lengths: DEFAULT (1-16 bytes)
- Training duration: ~1h 46min

---

## Vocabulary Statistics

**Composition**:
- Base bytes (0-255): 256 tokens
- Learned merges: 3,833 tokens
- Special tokens: 7 tokens (`<|start|>`, `<|end|>`, `<|pad|>`, `<|unk|>`, `<|cls|>`, `<|sep|>`, `<|mask|>`)
- **Total**: 4,096 tokens

**Quality Metrics**:
- All tokens reachable: ✓ Yes
- Valid merges: 3,833 / 3,833
- Power-of-2 size: ✓ Yes (2^12)

---

## Token Length Distribution

| Length | Count | Percentage | Description |
|--------|-------|------------|-------------|
| 1 byte | 256 | 6.3% | Base bytes |
| 2 bytes | 1,974 | 48.3% | Byte pairs |
| 3 bytes | 841 | 20.6% | Complete x86-64 instructions |
| 4 bytes | 649 | 15.9% | Instructions with operands |
| 5 bytes | 95 | 2.3% | Complex patterns |
| 6 bytes | 86 | 2.1% | Complex patterns |
| 7 bytes | 40 | 1.0% | Complex patterns |
| 8 bytes | 59 | 1.4% | Complex patterns |
| 9+ bytes | 89 | 2.2% | Long patterns |

**Average Token Length**: 3.000 bytes

---

## Byte Content Analysis

**Content Categories**:
- Contains NULL byte (0x00): 1,094 tokens (26.7%)
- ASCII printable (0x20-0x7E): 896 tokens (21.9%)
- All ASCII (<0x80): 1,879 tokens (45.9%)
- High bytes (≥0x80): 2,210 tokens (54.0%)

**Most Common Bytes in Tokens**:
- `0x00` (NULL): 2,468 occurrences - Padding and alignment
- `0xFF`: 404 occurrences - Sentinel values
- `0x48` (REX.W): 340 occurrences - x86-64 REX prefix
- `0x8B` (MOV): 233 occurrences - x86-64 MOV opcode
- `0xCC` (INT3): 170 occurrences - Debug breakpoint padding

---

## Sequence Coverage

**N-byte Sequence Diversity**:
| Length | Learned Tokens | Possible Sequences | Coverage |
|--------|----------------|-------------------|----------|
| 1-byte | 256 | 256 | 100.00% |
| 2-byte | 1,974 | 65,536 | 3.01% |
| 3-byte | 841 | 16,777,216 | 0.005% |
| 4-byte | 649 | 4,294,967,296 | 0.000015% |

---

## Files

- `tokenizer-4096.json` - Trained tokenizer model (286 KB)
- `analysis_results.json` - Detailed analysis statistics
- `training.log` - Training output log
- `training_stats.txt` - Training summary

---

## Usage

**Load from HuggingFace Hub**:
```python
from tokenizers import Tokenizer

# Load directly from HuggingFace
tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-4k")
```

**Load from local file**:
```bash
# With bbpe CLI
bbpe encode --tokenizer tokenizer-4096.json /path/to/binary
bbpe info tokenizer-4096.json
```

**Complete Python Example**:
```python
from tokenizers import Tokenizer

# Load from HuggingFace or local file
tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-4k")
# OR: tokenizer = Tokenizer.from_file("tokenizer-4096.json")

# Read binary file and decode as latin-1 (preserves all byte values 0-255)
with open("/usr/bin/ls", "rb") as f:
    data = f.read()
    data_str = data.decode("latin-1")

# Encode the binary data
encoding = tokenizer.encode(data_str)
print(f"File size: {len(data)} bytes")
print(f"Total tokens: {len(encoding.ids)}")
print(f"Compression: {len(data) / len(encoding.ids):.3f} bytes/token")

# First 10 tokens
for i, (token_id, token) in enumerate(zip(encoding.ids[:10], encoding.tokens[:10])):
    token_bytes = token.encode("latin-1")
    print(f"  Token {i}: ID={token_id:5d} hex={token_bytes.hex():20s} ({len(token_bytes)} bytes)")

# Decode tokens back to bytes
decoded_str = tokenizer.decode(encoding.ids)
decoded_bytes = decoded_str.encode("latin-1")
assert decoded_bytes == data  # Perfect reconstruction
```

**Example output for `/usr/bin/ls` (142,312 bytes)**:
```
File size: 142312 bytes
Total tokens: 71272
Compression: 1.997 bytes/token

First 10 tokens:
  Token 0: ID=  127 hex=7f                   (1 bytes)
  Token 1: ID= 3732 hex=454c                 (2 bytes)
  Token 2: ID=   70 hex=46                   (1 bytes)
  Token 3: ID=    2 hex=02                   (1 bytes)
  Token 4: ID=  392 hex=0101                 (2 bytes)
  Token 5: ID=  662 hex=000000000000000000   (9 bytes)
  Token 6: ID=  265 hex=0300                 (2 bytes)
  Token 7: ID= 1369 hex=3e00                 (2 bytes)
  Token 8: ID=  279 hex=01000000             (4 bytes)
  Token 9: ID=   48 hex=30                   (1 bytes)

Decoded: 7f454c4602010100000000000000000003003e000100000030...
(ELF header: 7f 45 4c 46 = ELF magic bytes)
```

---

## Citation

If you use this tokenizer in your research, please cite:

```bibtex
@article{bommarito2025binarybpe,
  title={Binary BPE: Cross-Platform Tokenization for Binary Analysis},
  author={Bommarito II, Michael J.},
  journal={arXiv preprint},
  year={2025},
  note={Preprint coming soon}
}
```

**Author**: Michael J. Bommarito II ([michael.bommarito@gmail.com](mailto:michael.bommarito@gmail.com))

---

**Generated**: November 12, 2025
**Training Script**: `train_tokenizers.sh`
**Analysis Script**: `analyze_tokenizer.py`