Upload binary-tokenizer-001-64k tokenizer

.gitattributes

@@ -1,35 +1,2 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text
+*.txt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,306 @@
+---
+language:
+- code
+license: apache-2.0
+tags:
+- tokenizer
+- binary-analysis
+- binary-tokenization
+- bpe
+- byte-pair-encoding
+- malware-analysis
+- reverse-engineering
+- security
+- x86-64
+- arm64
+- elf
+- pe
+library_name: tokenizers
+pipeline_tag: feature-extraction
+---
+
+# binary-tokenizer-001-64k
+
+A cross-platform BPE tokenizer for binary executables and machine code. Trained using advanced chunked training with deduplication on 23 GB of diverse binaries spanning Linux and Windows platforms.
+
+**🔗 Model**: [`mjbommar/binary-tokenizer-001-64k`](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)
+**📊 Dataset**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
+**📄 Paper**: *Binary BPE: Cross-Platform Tokenization for Binary Analysis* (arXiv preprint coming soon)
+
+## Overview
+
+- **Vocabulary Size**: 65,536 tokens (2^16)
+- **Token Composition**: 256 base bytes + 65,273 learned merges + 7 special tokens
+- **Average Token Length**: 3.749 bytes
+- **3-byte Instructions**: 16.5% of vocabulary (10,800 tokens)
+- **Compression Ratio**: ~2.6 bytes/token on typical binaries
+
+---
+
+## Training Configuration
+
+**Training Corpus**:
+- Source: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
+- Size: ~23 GB (24.7 billion bytes)
+- Processed Chunks: 40,574 total (37,083 unique + 8,454 duplicates reused)
+- Platforms: Linux (Alpine, Debian, Ubuntu - ELF), Windows (8, 10, 11 - PE)
+- Architectures: x86-64, x86-32, ARM64
+
+**Training Parameters**:
+- Vocabulary size: 65,536 (including 7 special tokens)
+- Min frequency: 4
+- Chunk size: 4,194,304 bytes (4 MB)
+- Training method: Chunked BPE with deduplication and support-based merge combination
+- Allowed lengths: DEFAULT (1-16 bytes)
+- Training duration: ~8-9 hours
+
+---
+
+## Vocabulary Statistics
+
+**Composition**:
+- Base bytes (0-255): 256 tokens
+- Learned merges: 65,273 tokens
+- Special tokens: 7 tokens (`<|start|>`, `<|end|>`, `<|pad|>`, `<|unk|>`, `<|cls|>`, `<|sep|>`, `<|mask|>`)
+- **Total**: 65,536 tokens
+
+**Quality Metrics**:
+- All tokens reachable: ✓ Yes
+- Valid merges: 65,273 / 65,273
+- Power-of-2 size: ✓ Yes (2^16)
+
+---
+
+## Token Length Distribution
+
+| Length | Count | Percentage | Description |
+|--------|-------|------------|-------------|
+| 1 byte | 256 | 0.4% | Base bytes |
+| 2 bytes | 28,561 | 43.6% | Byte pairs (most common patterns) |
+| 3 bytes | 10,800 | 16.5% | Complete x86-64 instructions |
+| 4 bytes | 14,376 | 21.9% | Instructions with operands |
+| 5 bytes | 2,780 | 4.2% | Complex patterns |
+| 6 bytes | 2,213 | 3.4% | Complex patterns |
+| 7 bytes | 1,167 | 1.8% | Complex patterns |
+| 8 bytes | 2,329 | 3.6% | Multi-byte sequences |
+| 9+ bytes | 3,045 | 4.6% | Long patterns |
+
+**Average Token Length**: 3.749 bytes
+
+---
+
+## Byte Content Analysis
+
+**Content Categories**:
+- Contains NULL byte (0x00): 17,418 tokens (26.6%)
+- ASCII printable (0x20-0x7E): 9,478 tokens (14.5%)
+- All ASCII (<0x80): 20,816 tokens (31.8%)
+- High bytes (≥0x80): 44,711 tokens (68.2%)
+
+**Most Common Bytes in Tokens**:
+- `0x00` (NULL): 34,482 occurrences - Padding and alignment
+- `0xFF`: 6,545 occurrences - Sentinel values
+- `0x48` (REX.W): 3,419 occurrences - x86-64 REX prefix
+- `0x8B` (MOV): 2,486 occurrences - x86-64 MOV opcode
+- `0x40` (@): 4,538 occurrences - ASCII and instruction patterns
+
+---
+
+## Sequence Coverage
+
+**N-byte Sequence Diversity**:
+| Length | Learned Tokens | Possible Sequences | Coverage |
+|--------|----------------|-------------------|----------|
+| 1-byte | 256 | 256 | 100.00% |
+| 2-byte | 28,561 | 65,536 | 43.58% |
+| 3-byte | 10,800 | 16,777,216 | 0.064% |
+| 4-byte | 14,376 | 4,294,967,296 | 0.00034% |
+
+**Notable Achievement**: 43.6% coverage of all possible 2-byte sequences - excellent for pattern recognition.
+
+---
+
+## Files
+
+- `tokenizer-65536.json` - Trained tokenizer model (2.4 MB)
+- `analysis_results.json` - Detailed analysis statistics
+- `original_README.md` - Original README from HuggingFace
+
+---
+
+## Usage
+
+**Load from HuggingFace Hub**:
+```python
+from tokenizers import Tokenizer
+
+# Load directly from HuggingFace
+tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-002")
+```
+
+**Load from local file**:
+```bash
+# With bbpe CLI
+bbpe encode --tokenizer tokenizer-65536.json /path/to/binary
+bbpe info tokenizer-65536.json
+```
+
+**Complete Python Example**:
+```python
+from tokenizers import Tokenizer
+
+# Load from HuggingFace or local file
+tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-002")
+# OR: tokenizer = Tokenizer.from_file("tokenizer-65536.json")
+
+# Read binary file and decode as latin-1 (preserves all byte values 0-255)
+with open("/usr/bin/ls", "rb") as f:
+    data = f.read()
+    data_str = data.decode("latin-1")
+
+# Encode the binary data
+encoding = tokenizer.encode(data_str)
+print(f"File size: {len(data)} bytes")
+print(f"Total tokens: {len(encoding.ids)}")
+print(f"Compression: {len(data) / len(encoding.ids):.3f} bytes/token")
+
+# First 10 tokens
+for i, (token_id, token) in enumerate(zip(encoding.ids[:10], encoding.tokens[:10])):
+    token_bytes = token.encode("latin-1")
+    print(f"  Token {i}: ID={token_id:5d} hex={token_bytes.hex():20s} ({len(token_bytes)} bytes)")
+
+# Decode tokens back to bytes
+decoded_str = tokenizer.decode(encoding.ids)
+decoded_bytes = decoded_str.encode("latin-1")
+assert decoded_bytes == data  # Perfect reconstruction
+```
+
+**Example output for `/usr/bin/ls` (142,312 bytes)**:
+```
+File size: 142312 bytes
+Total tokens: 54537
+Compression: 2.609 bytes/token
+
+First 10 tokens:
+  Token 0: ID=  127 hex=7f                   (1 bytes)
+  Token 1: ID= 2382 hex=454c                 (2 bytes)
+  Token 2: ID= 5923 hex=4602                 (2 bytes)
+  Token 3: ID=  394 hex=0101                 (2 bytes)
+  Token 4: ID=  268 hex=000000000000         (6 bytes)
+  Token 5: ID=  259 hex=000000               (3 bytes)
+  Token 6: ID=  295 hex=0300                 (2 bytes)
+  Token 7: ID= 2124 hex=3e00                 (2 bytes)
+  Token 8: ID=  271 hex=01000000             (4 bytes)
+  Token 9: ID=59106 hex=306d                 (2 bytes)
+
+Decoded: 7f454c4602010100000000000000000003003e0001000000306d...
+(ELF header: 7f 45 4c 46 = ELF magic bytes)
+```
+
+---
+
+## Performance Characteristics
+
+**Compression on Real-World Binaries**:
+
+| Binary | Size | Tokens | bytes/token |
+|--------|------|--------|-------------|
+| bash | 1.38 MB | 602,719 | 2.399 |
+| python3.12 | 7.65 MB | 2,997,303 | 2.676 |
+| gcc-13 | 0.98 MB | 375,331 | 2.726 |
+| ls | 0.14 MB | 54,537 | 2.609 |
+| grep | 0.18 MB | 73,500 | 2.542 |
+
+**Average**: 2.590 bytes/token
+
+**Information-Theoretic Efficiency**:
+- Binary entropy: ~6.5 bits/byte
+- Theoretical optimal: 2.46 bytes/token
+- Actual performance: 2.590 bytes/token
+- **Efficiency: 95.0%** of theoretical optimum
+
+---
+
+## Key Features
+
+**Instruction-Aware Patterns**:
+- REX prefixes: `0x48`, `0x4c`, `0x4d` (x86-64 64-bit operands)
+- Common opcodes: `0x8b` (MOV), `0x89` (MOV), `0xe8` (CALL)
+- ModR/M patterns: `0xc0`, `0x45`, `0x5d`
+
+**Common Binary Patterns**:
+- Padding: `0xcc 0xcc` (INT3 debug breakpoints), `0x90 0x90` (NOP sleds)
+- Alignment: `0x00 0x00 0x00 0x00` (NULL padding)
+- String terminators: `0x00` at word boundaries
+
+**String-Rich Vocabulary**:
+- 11.81% of vocabulary contains function names, paths, and library references
+- Better semantic understanding than standard BPE
+
+---
+
+## Comparison with Other Tokenizers
+
+**vs. binary-tokenizer-001 Series** (this repository):
+
+| Metric | 4K | 8K | 16K | 64K (this) | Improvement |
+|--------|----|----|-----|------------|-------------|
+| Vocab size | 4,096 | 8,192 | 16,384 | 65,536 | 4-16x larger |
+| Avg token length | 3.000 | 3.312 | 3.498 | 3.749 | +25% vs 4K |
+| 3-byte tokens % | 20.6% | 21.7% | 20.5% | 16.5% | Different focus |
+| 2-byte coverage | 3.0% | 5.6% | 10.9% | 43.6% | 14x better |
+| Compression (ls) | 2.00 | 2.17 | 2.39 | 2.61 | +30% vs 4K |
+| Training method | Standard | Standard | Standard | Chunked+dedup | Advanced |
+
+**Key Advantages of 64K Vocabulary**:
+- **43.6% 2-byte coverage**: Captures nearly half of all possible byte pairs
+- **Chunked training**: Deduplication-aware training improves merge quality
+- **Better compression**: 2.609 bytes/token vs 2.0 bytes/token (4K)
+- **Longer patterns**: 3.749 byte average vs 3.0 bytes (4K)
+- **String-rich**: 11.81% vocabulary contains semantic strings
+
+---
+
+## Citation
+
+If you use this tokenizer in your research, please cite:
+
+```bibtex
+@article{bommarito2025binarybpe,
+  title={Binary BPE: Cross-Platform Tokenization for Binary Analysis},
+  author={Bommarito II, Michael J.},
+  journal={arXiv preprint},
+  year={2025},
+  note={Preprint coming soon}
+}
+```
+
+**Relation to Glaurung Tokenizer**:
+
+This tokenizer is part of the binary-tokenizer-001 series and is identical to [`mjbommar/glaurung-binary-tokenizer-002`](https://huggingface.co/mjbommar/glaurung-binary-tokenizer-002), provided here with consistent naming and formatting. The original Glaurung tokenizer was trained using:
+
+```
+Glaurung Binary Tokenizer 002
+Training: October-November 2025
+Training Method: Chunked BPE with deduplication (bbpe v0.3.2)
+Dataset: 23GB binaries-small (40,574 chunks, 8,454 duplicates)
+Performance: 2.590 bytes/token (95% of theoretical optimum)
+```
+
+**Author**: Michael J. Bommarito II ([michael.bommarito@gmail.com](mailto:michael.bommarito@gmail.com))
+
+---
+
+## License
+
+**Apache License 2.0**
+
+This tokenizer is part of the [Glaurung](https://github.com/mjbommar/glaurung) project.
+
+---
+
+**Generated**: November 13, 2025
+**Part of**: binary-tokenizer-001 series (4K, 8K, 16K, 64K)
+**Original Model**: `mjbommar/glaurung-binary-tokenizer-002`
+**Training Tool**: bbpe v0.3.2
+**Analysis Script**: `analyze_tokenizer.py`

analysis_results.json

@@ -0,0 +1,141 @@
+{
+  "vocab_size": {
+    "total": 65529,
+    "total_with_special": 65536,
+    "base": 256,
+    "merges": 65273,
+    "special": 7,
+    "is_power_of_2": true,
+    "power": 16,
+    "matches_expected": true
+  },
+  "reachability": {
+    "valid_merges": 65273,
+    "invalid_merges": 0,
+    "reachable": 65529,
+    "unreachable": 0,
+    "all_reachable": true
+  },
+  "length_dist": {
+    "distribution": {
+      "1": 256,
+      "2": 28561,
+      "3": 10800,
+      "4": 14376,
+      "5": 2780,
+      "6": 2213,
+      "7": 1167,
+      "8": 2329,
+      "9": 295,
+      "10": 368,
+      "11": 244,
+      "12": 711,
+      "13": 101,
+      "14": 177,
+      "15": 112,
+      "16": 340,
+      "17": 28,
+      "18": 79,
+      "19": 57,
+      "20": 136,
+      "21": 15,
+      "22": 34,
+      "23": 49,
+      "24": 121,
+      "25": 16,
+      "26": 17,
+      "27": 16,
+      "28": 51,
+      "29": 8,
+      "31": 7,
+      "32": 47,
+      "30": 16
+    },
+    "avg_length": 3.748882140186488,
+    "min_length": 1,
+    "max_length": 32,
+    "length_3_count": 10800,
+    "length_3_percent": 16.481755612190394
+  },
+  "byte_content": {
+    "null_tokens": 17418,
+    "ascii_printable": 9478,
+    "ascii_only": 20816,
+    "high_byte": 44711,
+    "mixed": 25785,
+    "byte_distribution": {
+      "0": 34482,
+      "255": 6545,
+      "3": 6081,
+      "64": 4538,
+      "1": 4447,
+      "249": 3888,
+      "2": 3436,
+      "72": 3419,
+      "128": 3195,
+      "32": 3026,
+      "4": 2593,
+      "169": 2534,
+      "139": 2486,
+      "145": 2423,
+      "224": 2182,
+      "170": 2122,
+      "84": 2106,
+      "116": 1953,
+      "101": 1855,
+      "65": 1852,
+      "151": 1838,
+      "185": 1805,
+      "253": 1708,
+      "97": 1703,
+      "36": 1700,
+      "99": 1687,
+      "115": 1669,
+      "8": 1615,
+      "6": 1503,
+      "5": 1479,
+      "96": 1471,
+      "192": 1436,
+      "23": 1429,
+      "114": 1424,
+      "204": 1414,
+      "31": 1371,
+      "7": 1368,
+      "82": 1350,
+      "33": 1307,
+      "105": 1260,
+      "137": 1176,
+      "131": 1172,
+      "67": 1162,
+      "19": 1155,
+      "111": 1153,
+      "110": 1143,
+      "68": 1130,
+      "15": 1124,
+      "66": 1123,
+      "113": 1116
+    }
+  },
+  "diversity": {
+    "1": {
+      "learned": 256,
+      "possible": 256,
+      "coverage": 100.0
+    },
+    "2": {
+      "learned": 28561,
+      "possible": 65536,
+      "coverage": 43.58062744140625
+    },
+    "3": {
+      "learned": 10800,
+      "possible": 16777216,
+      "coverage": 0.06437301635742188
+    },
+    "4": {
+      "learned": 14376,
+      "possible": 4294967296,
+      "coverage": 0.000334717333316803
+    }
+  }
+}