Upload binary-tokenizer-001-64k tokenizer

README.md

@@ -1,27 +1,24 @@
 ---
 language:
 - code
-license: apache-2.0
 tags:
 - tokenizer
 - binary-analysis
 - binary-tokenization
 - bpe
 - byte-pair-encoding
-- malware-analysis
 - reverse-engineering
-- security
-- x86-64
-- arm64
-- elf
-- pe
-library_name: tokenizers
+- malware-analysis
+- cybersecurity
+- executable-analysis
+license: mit
 pipeline_tag: feature-extraction
+library_name: tokenizers
 ---
 
 # binary-tokenizer-001-64k
 
-A cross-platform BPE tokenizer for binary executables and machine code. Trained using advanced chunked training with deduplication on 23 GB of diverse binaries spanning Linux and Windows platforms.
+A cross-platform BPE tokenizer for binary executables and machine code. Trained on 13 GB of diverse binaries spanning Linux, Windows, macOS, and Android platforms.
 
 **🔗 Model**: [`mjbommar/binary-tokenizer-001-64k`](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)
 **📊 Dataset**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
@@ -31,9 +28,9 @@ A cross-platform BPE tokenizer for binary executables and machine code. Trained
 
 - **Vocabulary Size**: 65,536 tokens (2^16)
 - **Token Composition**: 256 base bytes + 65,273 learned merges + 7 special tokens
-- **Average Token Length**: 3.749 bytes
-- **3-byte Instructions**: 16.5% of vocabulary (10,800 tokens)
-- **Compression Ratio**: ~2.6 bytes/token on typical binaries
+- **Average Token Length**: 4.173 bytes
+- **3-byte Instructions**: 17.9% of vocabulary (11,729 tokens)
+- **Compression Ratio**: ~3.0 bytes/token on typical binaries
 
 ---
 
@@ -41,18 +38,17 @@ A cross-platform BPE tokenizer for binary executables and machine code. Trained
 
 **Training Corpus**:
 - Source: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized)
-- Size: ~23 GB (24.7 billion bytes)
-- Processed Chunks: 40,574 total (37,083 unique + 8,454 duplicates reused)
-- Platforms: Linux (Alpine, Debian, Ubuntu - ELF), Windows (8, 10, 11 - PE)
-- Architectures: x86-64, x86-32, ARM64
+- Size: ~13 GB
+- Files: 30,738 binary files
+- Platforms: Linux (ELF), Windows (PE), macOS (Mach-O), Android (APK)
+- Architectures: x86-64, x86, ARM64, ARM, MIPS, RISC-V
 
 **Training Parameters**:
 - Vocabulary size: 65,536 (including 7 special tokens)
-- Min frequency: 4
-- Chunk size: 4,194,304 bytes (4 MB)
-- Training method: Chunked BPE with deduplication and support-based merge combination
+- Min frequency: 10
+- Chunk size: 8,192 bytes
 - Allowed lengths: DEFAULT (1-16 bytes)
-- Training duration: ~8-9 hours
+- Training duration: ~12-14 hours
 
 ---
 
@@ -76,33 +72,33 @@ A cross-platform BPE tokenizer for binary executables and machine code. Trained
 | Length | Count | Percentage | Description |
 |--------|-------|------------|-------------|
 | 1 byte | 256 | 0.4% | Base bytes |
-| 2 bytes | 28,561 | 43.6% | Byte pairs (most common patterns) |
-| 3 bytes | 10,800 | 16.5% | Complete x86-64 instructions |
-| 4 bytes | 14,376 | 21.9% | Instructions with operands |
-| 5 bytes | 2,780 | 4.2% | Complex patterns |
-| 6 bytes | 2,213 | 3.4% | Complex patterns |
-| 7 bytes | 1,167 | 1.8% | Complex patterns |
-| 8 bytes | 2,329 | 3.6% | Multi-byte sequences |
-| 9+ bytes | 3,045 | 4.6% | Long patterns |
+| 2 bytes | 24,943 | 38.1% | Byte pairs (most common) |
+| 3 bytes | 11,729 | 17.9% | Complete x86-64 instructions |
+| 4 bytes | 13,189 | 20.1% | Instructions with operands |
+| 5 bytes | 3,737 | 5.7% | Complex patterns |
+| 6 bytes | 3,109 | 4.7% | Complex patterns |
+| 7 bytes | 1,564 | 2.4% | Complex patterns |
+| 8 bytes | 2,498 | 3.8% | Multi-byte sequences |
+| 9+ bytes | 3,302 | 5.0% | Long patterns |
 
-**Average Token Length**: 3.749 bytes
+**Average Token Length**: 4.173 bytes
 
 ---
 
 ## Byte Content Analysis
 
 **Content Categories**:
-- Contains NULL byte (0x00): 17,418 tokens (26.6%)
-- ASCII printable (0x20-0x7E): 9,478 tokens (14.5%)
-- All ASCII (<0x80): 20,816 tokens (31.8%)
-- High bytes (≥0x80): 44,711 tokens (68.2%)
+- Contains NULL byte (0x00): 16,988 tokens (25.9%)
+- ASCII printable (0x20-0x7E): 11,552 tokens (17.6%)
+- All ASCII (<0x80): 24,706 tokens (37.7%)
+- High bytes (≥0x80): 40,821 tokens (62.3%)
 
 **Most Common Bytes in Tokens**:
-- `0x00` (NULL): 34,482 occurrences - Padding and alignment
-- `0xFF`: 6,545 occurrences - Sentinel values
-- `0x48` (REX.W): 3,419 occurrences - x86-64 REX prefix
-- `0x8B` (MOV): 2,486 occurrences - x86-64 MOV opcode
-- `0x40` (@): 4,538 occurrences - ASCII and instruction patterns
+- `0x00` (NULL): 42,537 occurrences - Padding and alignment
+- `0xFF`: 7,204 occurrences - Sentinel values
+- `0x48` (REX.W): 6,105 occurrences - x86-64 REX prefix
+- `0x8B` (MOV): 4,016 occurrences - x86-64 MOV opcode
+- `0x20` (space): 4,087 occurrences - ASCII strings
 
 ---
 
@@ -112,19 +108,18 @@ A cross-platform BPE tokenizer for binary executables and machine code. Trained
 | Length | Learned Tokens | Possible Sequences | Coverage |
 |--------|----------------|-------------------|----------|
 | 1-byte | 256 | 256 | 100.00% |
-| 2-byte | 28,561 | 65,536 | 43.58% |
-| 3-byte | 10,800 | 16,777,216 | 0.064% |
-| 4-byte | 14,376 | 4,294,967,296 | 0.00034% |
-
-**Notable Achievement**: 43.6% coverage of all possible 2-byte sequences - excellent for pattern recognition.
+| 2-byte | 24,943 | 65,536 | 38.06% |
+| 3-byte | 11,729 | 16,777,216 | 0.070% |
+| 4-byte | 13,189 | 4,294,967,296 | 0.00031% |
 
 ---
 
 ## Files
 
-- `tokenizer-65536.json` - Trained tokenizer model (2.4 MB)
+- `tokenizer-65536.json` - Trained tokenizer model (5.0 MB)
 - `analysis_results.json` - Detailed analysis statistics
-- `original_README.md` - Original README from HuggingFace
+- `training.log` - Training output log (if available)
+- `training_stats.txt` - Training summary (if available)
 
 ---
 
@@ -135,7 +130,7 @@ A cross-platform BPE tokenizer for binary executables and machine code. Trained
 from tokenizers import Tokenizer
 
 # Load directly from HuggingFace
-tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-002")
+tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-64k")
 ```
 
 **Load from local file**:
@@ -150,7 +145,7 @@ bbpe info tokenizer-65536.json
 from tokenizers import Tokenizer
 
 # Load from HuggingFace or local file
-tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-002")
+tokenizer = Tokenizer.from_pretrained("mjbommar/binary-tokenizer-001-64k")
 # OR: tokenizer = Tokenizer.from_file("tokenizer-65536.json")
 
 # Read binary file and decode as latin-1 (preserves all byte values 0-255)
@@ -178,20 +173,20 @@ assert decoded_bytes == data  # Perfect reconstruction
 **Example output for `/usr/bin/ls` (142,312 bytes)**:
 ```
 File size: 142312 bytes
-Total tokens: 54537
-Compression: 2.609 bytes/token
+Total tokens: 47993
+Compression: 2.965 bytes/token
 
 First 10 tokens:
-  Token 0: ID=  127 hex=7f                   (1 bytes)
-  Token 1: ID= 2382 hex=454c                 (2 bytes)
-  Token 2: ID= 5923 hex=4602                 (2 bytes)
-  Token 3: ID=  394 hex=0101                 (2 bytes)
-  Token 4: ID=  268 hex=000000000000         (6 bytes)
-  Token 5: ID=  259 hex=000000               (3 bytes)
-  Token 6: ID=  295 hex=0300                 (2 bytes)
-  Token 7: ID= 2124 hex=3e00                 (2 bytes)
-  Token 8: ID=  271 hex=01000000             (4 bytes)
-  Token 9: ID=59106 hex=306d                 (2 bytes)
+  Token 0: ID=45813 hex=7f454c46020101       (7 bytes)
+  Token 1: ID=  662 hex=000000000000000000   (9 bytes)
+  Token 2: ID=  265 hex=0300                 (2 bytes)
+  Token 3: ID= 1369 hex=3e00                 (2 bytes)
+  Token 4: ID=  279 hex=01000000             (4 bytes)
+  Token 5: ID=41250 hex=306d                 (2 bytes)
+  Token 6: ID=  288 hex=000000000000         (6 bytes)
+  Token 7: ID= 5908 hex=4000000000000000     (8 bytes)
+  Token 8: ID= 8377 hex=2824                 (2 bytes)
+  Token 9: ID=14325 hex=02000000000000000000 (10 bytes)
 
 Decoded: 7f454c4602010100000000000000000003003e0001000000306d...
 (ELF header: 7f 45 4c 46 = ELF magic bytes)
@@ -199,68 +194,6 @@ Decoded: 7f454c4602010100000000000000000003003e0001000000306d...
 
 ---
 
-## Performance Characteristics
-
-**Compression on Real-World Binaries**:
-
-| Binary | Size | Tokens | bytes/token |
-|--------|------|--------|-------------|
-| bash | 1.38 MB | 602,719 | 2.399 |
-| python3.12 | 7.65 MB | 2,997,303 | 2.676 |
-| gcc-13 | 0.98 MB | 375,331 | 2.726 |
-| ls | 0.14 MB | 54,537 | 2.609 |
-| grep | 0.18 MB | 73,500 | 2.542 |
-
-**Average**: 2.590 bytes/token
-
-**Information-Theoretic Efficiency**:
-- Binary entropy: ~6.5 bits/byte
-- Theoretical optimal: 2.46 bytes/token
-- Actual performance: 2.590 bytes/token
-- **Efficiency: 95.0%** of theoretical optimum
-
----
-
-## Key Features
-
-**Instruction-Aware Patterns**:
-- REX prefixes: `0x48`, `0x4c`, `0x4d` (x86-64 64-bit operands)
-- Common opcodes: `0x8b` (MOV), `0x89` (MOV), `0xe8` (CALL)
-- ModR/M patterns: `0xc0`, `0x45`, `0x5d`
-
-**Common Binary Patterns**:
-- Padding: `0xcc 0xcc` (INT3 debug breakpoints), `0x90 0x90` (NOP sleds)
-- Alignment: `0x00 0x00 0x00 0x00` (NULL padding)
-- String terminators: `0x00` at word boundaries
-
-**String-Rich Vocabulary**:
-- 11.81% of vocabulary contains function names, paths, and library references
-- Better semantic understanding than standard BPE
-
----
-
-## Comparison with Other Tokenizers
-
-**vs. binary-tokenizer-001 Series** (this repository):
-
-| Metric | 4K | 8K | 16K | 64K (this) | Improvement |
-|--------|----|----|-----|------------|-------------|
-| Vocab size | 4,096 | 8,192 | 16,384 | 65,536 | 4-16x larger |
-| Avg token length | 3.000 | 3.312 | 3.498 | 3.749 | +25% vs 4K |
-| 3-byte tokens % | 20.6% | 21.7% | 20.5% | 16.5% | Different focus |
-| 2-byte coverage | 3.0% | 5.6% | 10.9% | 43.6% | 14x better |
-| Compression (ls) | 2.00 | 2.17 | 2.39 | 2.61 | +30% vs 4K |
-| Training method | Standard | Standard | Standard | Chunked+dedup | Advanced |
-
-**Key Advantages of 64K Vocabulary**:
-- **43.6% 2-byte coverage**: Captures nearly half of all possible byte pairs
-- **Chunked training**: Deduplication-aware training improves merge quality
-- **Better compression**: 2.609 bytes/token vs 2.0 bytes/token (4K)
-- **Longer patterns**: 3.749 byte average vs 3.0 bytes (4K)
-- **String-rich**: 11.81% vocabulary contains semantic strings
-
----
-
 ## Citation
 
 If you use this tokenizer in your research, please cite:
@@ -275,32 +208,10 @@ If you use this tokenizer in your research, please cite:
 }
 ```
 
-**Relation to Glaurung Tokenizer**:
-
-This tokenizer is part of the binary-tokenizer-001 series and is identical to [`mjbommar/glaurung-binary-tokenizer-002`](https://huggingface.co/mjbommar/glaurung-binary-tokenizer-002), provided here with consistent naming and formatting. The original Glaurung tokenizer was trained using:
-
-```
-Glaurung Binary Tokenizer 002
-Training: October-November 2025
-Training Method: Chunked BPE with deduplication (bbpe v0.3.2)
-Dataset: 23GB binaries-small (40,574 chunks, 8,454 duplicates)
-Performance: 2.590 bytes/token (95% of theoretical optimum)
-```
-
 **Author**: Michael J. Bommarito II ([michael.bommarito@gmail.com](mailto:michael.bommarito@gmail.com))
 
 ---
 
-## License
-
-**Apache License 2.0**
-
-This tokenizer is part of the [Glaurung](https://github.com/mjbommar/glaurung) project.
-
----
-
 **Generated**: November 13, 2025
-**Part of**: binary-tokenizer-001 series (4K, 8K, 16K, 64K)
-**Original Model**: `mjbommar/glaurung-binary-tokenizer-002`
-**Training Tool**: bbpe v0.3.2
+**Training Script**: `train_tokenizers.sh`
 **Analysis Script**: `analyze_tokenizer.py`

analysis_results.json

@@ -1,141 +1,3 @@
-{
-  "vocab_size": {
-    "total": 65529,
-    "total_with_special": 65536,
-    "base": 256,
-    "merges": 65273,
-    "special": 7,
-    "is_power_of_2": true,
-    "power": 16,
-    "matches_expected": true
-  },
-  "reachability": {
-    "valid_merges": 65273,
-    "invalid_merges": 0,
-    "reachable": 65529,
-    "unreachable": 0,
-    "all_reachable": true
-  },
-  "length_dist": {
-    "distribution": {
-      "1": 256,
-      "2": 28561,
-      "3": 10800,
-      "4": 14376,
-      "5": 2780,
-      "6": 2213,
-      "7": 1167,
-      "8": 2329,
-      "9": 295,
-      "10": 368,
-      "11": 244,
-      "12": 711,
-      "13": 101,
-      "14": 177,
-      "15": 112,
-      "16": 340,
-      "17": 28,
-      "18": 79,
-      "19": 57,
-      "20": 136,
-      "21": 15,
-      "22": 34,
-      "23": 49,
-      "24": 121,
-      "25": 16,
-      "26": 17,
-      "27": 16,
-      "28": 51,
-      "29": 8,
-      "31": 7,
-      "32": 47,
-      "30": 16
-    },
-    "avg_length": 3.748882140186488,
-    "min_length": 1,
-    "max_length": 32,
-    "length_3_count": 10800,
-    "length_3_percent": 16.481755612190394
-  },
-  "byte_content": {
-    "null_tokens": 17418,
-    "ascii_printable": 9478,
-    "ascii_only": 20816,
-    "high_byte": 44711,
-    "mixed": 25785,
-    "byte_distribution": {
-      "0": 34482,
-      "255": 6545,
-      "3": 6081,
-      "64": 4538,
-      "1": 4447,
-      "249": 3888,
-      "2": 3436,
-      "72": 3419,
-      "128": 3195,
-      "32": 3026,
-      "4": 2593,
-      "169": 2534,
-      "139": 2486,
-      "145": 2423,
-      "224": 2182,
-      "170": 2122,
-      "84": 2106,
-      "116": 1953,
-      "101": 1855,
-      "65": 1852,
-      "151": 1838,
-      "185": 1805,
-      "253": 1708,
-      "97": 1703,
-      "36": 1700,
-      "99": 1687,
-      "115": 1669,
-      "8": 1615,
-      "6": 1503,
-      "5": 1479,
-      "96": 1471,
-      "192": 1436,
-      "23": 1429,
-      "114": 1424,
-      "204": 1414,
-      "31": 1371,
-      "7": 1368,
-      "82": 1350,
-      "33": 1307,
-      "105": 1260,
-      "137": 1176,
-      "131": 1172,
-      "67": 1162,
-      "19": 1155,
-      "111": 1153,
-      "110": 1143,
-      "68": 1130,
-      "15": 1124,
-      "66": 1123,
-      "113": 1116
-    }
-  },
-  "diversity": {
-    "1": {
-      "learned": 256,
-      "possible": 256,
-      "coverage": 100.0
-    },
-    "2": {
-      "learned": 28561,
-      "possible": 65536,
-      "coverage": 43.58062744140625
-    },
-    "3": {
-      "learned": 10800,
-      "possible": 16777216,
-      "coverage": 0.06437301635742188
-    },
-    "4": {
-      "learned": 14376,
-      "possible": 4294967296,
-      "coverage": 0.000334717333316803
-    }
-  }
-}
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c1c3c4b4ffb26b54bca3602e86c21a88281ba1e8ecf26fc5504703c267414f4
+size 2622