Upload binary-tokenizer-001-32k tokenizer

.gitattributes

@@ -1,35 +1,2 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text
+*.txt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,217 @@
+---
+language:
+- code
+tags:
+- tokenizer
+- binary-analysis
+- binary-tokenization
+- bpe
+- byte-pair-encoding
+- reverse-engineering
+- malware-analysis
+- cybersecurity
+- executable-analysis
+license: mit
+pipeline_tag: feature-extraction
+library_name: tokenizers
+---
+
+# binary-tokenizer-001-32k
+
+A cross-platform BPE tokenizer for binary executables and machine code. Trained on 13 GB of diverse binaries spanning Linux, Windows, macOS, and Android platforms.
+
+**🔗 Model**: [`mjbommar/binary-tokenizer-001-32k`](https://huggingface.co/mjbommar/binary-tokenizer-001-32k)
+**📊 Dataset**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
+**📄 Paper**: *Binary BPE: Cross-Platform Tokenization for Binary Analysis* (arXiv preprint coming soon)
+
+## Overview
+
+- **Vocabulary Size**: 32,768 tokens (2^15)
+- **Token Composition**: 256 base bytes + 32,505 learned merges + 7 special tokens
+- **Average Token Length**: 3.812 bytes
+- **3-byte Instructions**: 19.5% of vocabulary (6,380 tokens)
+- **Compression Ratio**: ~2.7 bytes/token on typical binaries
+
+---
+
+## Training Configuration
+
+**Training Corpus**:
+- Source: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
+- Size: ~13 GB
+- Files: 30,738 binary files
+- Platforms: Linux (ELF), Windows (PE), macOS (Mach-O), Android (APK)
+- Architectures: x86-64, x86, ARM64, ARM, MIPS, RISC-V
+
+**Training Parameters**:
+- Vocabulary size: 32,768 (including 7 special tokens)
+- Min frequency: 10
+- Chunk size: 8,192 bytes
+- Allowed lengths: DEFAULT (1-16 bytes)
+- Training duration: ~6-7 hours
+
+---
+
+## Vocabulary Statistics
+
+**Composition**:
+- Base bytes (0-255): 256 tokens
+- Learned merges: 32,505 tokens
+- Special tokens: 7 tokens (`<|start|>`, `<|end|>`, `<|pad|>`, `<|unk|>`, `<|cls|>`, `<|sep|>`, `<|mask|>`)
+- **Total**: 32,768 tokens
+
+**Quality Metrics**:
+- All tokens reachable: ✓ Yes
+- Valid merges: 32,505 / 32,505
+- Power-of-2 size: ✓ Yes (2^15)
+
+---
+
+## Token Length Distribution
+
+| Length | Count | Percentage | Description |
+|--------|-------|------------|-------------|
+| 1 byte | 256 | 0.8% | Base bytes |
+| 2 bytes | 13,428 | 41.0% | Byte pairs |
+| 3 bytes | 6,380 | 19.5% | Complete x86-64 instructions |
+| 4 bytes | 6,236 | 19.0% | Instructions with operands |
+| 5 bytes | 1,763 | 5.4% | Complex patterns |
+| 6 bytes | 1,395 | 4.3% | Complex patterns |
+| 7 bytes | 676 | 2.1% | Complex patterns |
+| 8 bytes | 963 | 2.9% | Complex patterns |
+| 9+ bytes | 1,467 | 4.5% | Long patterns |
+
+**Average Token Length**: 3.812 bytes
+
+---
+
+## Byte Content Analysis
+
+**Content Categories**:
+- Contains NULL byte (0x00): 8,350 tokens (25.5%)
+- ASCII printable (0x20-0x7E): 6,460 tokens (19.7%)
+- All ASCII (<0x80): 13,796 tokens (42.1%)
+- High bytes (≥0x80): 18,964 tokens (57.9%)
+
+**Most Common Bytes in Tokens**:
+- `0x00` (NULL): 20,462 occurrences - Padding and alignment
+- `0xFF`: 3,502 occurrences - Sentinel values
+- `0x48` (REX.W): 2,883 occurrences - x86-64 REX prefix
+- `0x8B` (MOV): 1,934 occurrences - x86-64 MOV opcode
+- `0xCC` (INT3): 1,366 occurrences - Debug breakpoint padding
+
+---
+
+## Sequence Coverage
+
+**N-byte Sequence Diversity**:
+| Length | Learned Tokens | Possible Sequences | Coverage |
+|--------|----------------|-------------------|----------|
+| 1-byte | 256 | 256 | 100.00% |
+| 2-byte | 13,428 | 65,536 | 20.49% |
+| 3-byte | 6,380 | 16,777,216 | 0.038% |
+| 4-byte | 6,236 | 4,294,967,296 | 0.00015% |
+
+---
+
+## Files
+
+- `tokenizer-32768.json` - Trained tokenizer model (2.5 MB)
+- `analysis_results.json` - Detailed analysis statistics
+- `training.log` - Training output log (if available)
+- `training_stats.txt` - Training summary (if available)
+
+---
+
+## Usage
+
+**Load from HuggingFace Hub**:
+```python
+from tokenizers import Tokenizer
+
+# Load directly from HuggingFace
+tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-32k")
+```
+
+**Load from local file**:
+```bash
+# With bbpe CLI
+bbpe encode --tokenizer tokenizer-32768.json /path/to/binary
+bbpe info tokenizer-32768.json
+```
+
+**Complete Python Example**:
+```python
+from tokenizers import Tokenizer
+
+# Load from HuggingFace or local file
+tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-32k")
+# OR: tokenizer = Tokenizer.from_file("tokenizer-32768.json")
+
+# Read binary file and decode as latin-1 (preserves all byte values 0-255)
+with open("/usr/bin/ls", "rb") as f:
+    data = f.read()
+    data_str = data.decode("latin-1")
+
+# Encode the binary data
+encoding = tokenizer.encode(data_str)
+print(f"File size: {len(data)} bytes")
+print(f"Total tokens: {len(encoding.ids)}")
+print(f"Compression: {len(data) / len(encoding.ids):.3f} bytes/token")
+
+# First 10 tokens
+for i, (token_id, token) in enumerate(zip(encoding.ids[:10], encoding.tokens[:10])):
+    token_bytes = token.encode("latin-1")
+    print(f"  Token {i}: ID={token_id:5d} hex={token_bytes.hex():20s} ({len(token_bytes)} bytes)")
+
+# Decode tokens back to bytes
+decoded_str = tokenizer.decode(encoding.ids)
+decoded_bytes = decoded_str.encode("latin-1")
+assert decoded_bytes == data  # Perfect reconstruction
+```
+
+**Example output for `/usr/bin/ls` (142,312 bytes)**:
+```
+File size: 142312 bytes
+Total tokens: 53184
+Compression: 2.676 bytes/token
+
+First 10 tokens:
+  Token 0: ID=  127 hex=7f                   (1 bytes)
+  Token 1: ID= 3732 hex=454c                 (2 bytes)
+  Token 2: ID= 4707 hex=4602                 (2 bytes)
+  Token 3: ID=  392 hex=0101                 (2 bytes)
+  Token 4: ID=  662 hex=000000000000000000   (9 bytes)
+  Token 5: ID=  265 hex=0300                 (2 bytes)
+  Token 6: ID= 1369 hex=3e00                 (2 bytes)
+  Token 7: ID=  279 hex=01000000             (4 bytes)
+  Token 8: ID=   48 hex=30                   (1 bytes)
+  Token 9: ID=  109 hex=6d                   (1 bytes)
+
+Decoded: 7f454c4602010100000000000000000003003e0001000000306d...
+(ELF header: 7f 45 4c 46 = ELF magic bytes)
+```
+
+---
+
+## Citation
+
+If you use this tokenizer in your research, please cite:
+
+```bibtex
+@article{bommarito2025binarybpe,
+  title={Binary BPE: Cross-Platform Tokenization for Binary Analysis},
+  author={Bommarito II, Michael J.},
+  journal={arXiv preprint},
+  year={2025},
+  note={Preprint coming soon}
+}
+```
+
+**Author**: Michael J. Bommarito II ([michael.bommarito@gmail.com](mailto:michael.bommarito@gmail.com))
+
+---
+
+**Generated**: November 13, 2025
+**Training Script**: `train_tokenizers.sh`
+**Analysis Script**: `analyze_tokenizer.py`

analysis_results.json

@@ -0,0 +1,141 @@
+{
+  "vocab_size": {
+    "total": 32761,
+    "total_with_special": 32768,
+    "base": 256,
+    "merges": 32505,
+    "special": 7,
+    "is_power_of_2": true,
+    "power": 15,
+    "matches_expected": true
+  },
+  "reachability": {
+    "valid_merges": 32505,
+    "invalid_merges": 0,
+    "reachable": 32761,
+    "unreachable": 0,
+    "all_reachable": true
+  },
+  "length_dist": {
+    "distribution": {
+      "1": 256,
+      "2": 13428,
+      "3": 6380,
+      "4": 6236,
+      "5": 1763,
+      "6": 1395,
+      "7": 676,
+      "8": 963,
+      "9": 191,
+      "10": 220,
+      "11": 109,
+      "12": 318,
+      "13": 86,
+      "14": 102,
+      "15": 69,
+      "16": 233,
+      "17": 26,
+      "18": 31,
+      "19": 23,
+      "20": 58,
+      "21": 16,
+      "22": 16,
+      "23": 19,
+      "24": 44,
+      "25": 6,
+      "26": 7,
+      "27": 8,
+      "28": 13,
+      "29": 7,
+      "30": 4,
+      "31": 3,
+      "32": 54
+    },
+    "avg_length": 3.812393162393162,
+    "min_length": 1,
+    "max_length": 32,
+    "length_3_count": 6380,
+    "length_3_percent": 19.474969474969473
+  },
+  "byte_content": {
+    "null_tokens": 8350,
+    "ascii_printable": 6460,
+    "ascii_only": 13796,
+    "high_byte": 18964,
+    "mixed": 10141,
+    "byte_distribution": {
+      "0": 20462,
+      "255": 3502,
+      "72": 2883,
+      "1": 2622,
+      "3": 1967,
+      "139": 1934,
+      "32": 1901,
+      "2": 1856,
+      "64": 1609,
+      "116": 1546,
+      "101": 1482,
+      "36": 1435,
+      "204": 1366,
+      "128": 1212,
+      "65": 1186,
+      "4": 1150,
+      "97": 1109,
+      "114": 1088,
+      "249": 1069,
+      "137": 1059,
+      "111": 990,
+      "8": 978,
+      "105": 964,
+      "115": 940,
+      "15": 917,
+      "110": 917,
+      "99": 879,
+      "16": 837,
+      "192": 814,
+      "232": 810,
+      "108": 798,
+      "131": 788,
+      "68": 777,
+      "84": 740,
+      "224": 737,
+      "112": 732,
+      "117": 723,
+      "48": 701,
+      "5": 690,
+      "169": 687,
+      "76": 684,
+      "69": 663,
+      "100": 653,
+      "95": 650,
+      "6": 647,
+      "73": 623,
+      "141": 614,
+      "10": 570,
+      "7": 562,
+      "66": 546
+    }
+  },
+  "diversity": {
+    "1": {
+      "learned": 256,
+      "possible": 256,
+      "coverage": 100.0
+    },
+    "2": {
+      "learned": 13428,
+      "possible": 65536,
+      "coverage": 20.489501953125
+    },
+    "3": {
+      "learned": 6380,
+      "possible": 16777216,
+      "coverage": 0.03802776336669922
+    },
+    "4": {
+      "learned": 6236,
+      "possible": 4294967296,
+      "coverage": 0.0001451931893825531
+    }
+  }
+}