README.md

---
language:
- code
license: apache-2.0
tags:
- binary-analysis
- tokenizer
- bpe
- malware-analysis
- reverse-engineering
- security
- x86-64
- arm64
- elf
- pe
library_name: tokenizers
pipeline_tag: feature-extraction
---

# Glaurung Binary Tokenizer 001

**Production-ready 64K vocabulary BPE tokenizer for binary executables**

🔗 **GitHub**: [mjbommar/glaurung](https://github.com/mjbommar/glaurung)

---

## Overview

**Glaurung Binary Tokenizer 001** is a specialized Byte Pair Encoding (BPE) tokenizer optimized for compiled binary data across multiple architectures (x86-64, ARM64, Windows PE, Linux ELF). This is the production successor to [binary-tokenizer-005](https://huggingface.co/mjbommar/binary-tokenizer-005).

### Key Specifications

- **Vocabulary Size**: 65,536 tokens (exactly 2^16 = 64K)
- **Compression**: 2.849 bytes/token average
- **Training Data**: 13GB corpus, 30,738 binaries
- **Architectures**: x86-64, x86-32, ARM64
- **Platforms**: Linux (Alpine, Debian, Ubuntu), Windows (8, 10, 11)
- **Encoding**: Latin-1 (each byte 0-255 maps to a single character)

### Performance Highlights

- **9-10% better compression** than 32K baseline
- **86% of theoretical maximum** compression efficiency
- **Instruction-aware**: Captures complete x86-64 instructions (REX + opcode + ModR/M)
- **String-rich**: 5.76% of vocabulary contains function names, paths, library references

---

## Installation

```bash
pip install tokenizers transformers
```

---

## Quick Start

### Method 1: Using the tokenizers library (Recommended)

```python
from tokenizers import Tokenizer
from pathlib import Path

# Load tokenizer directly from Hugging Face Hub
tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-001")

# Process binary data - MUST use latin-1 encoding
binary_path = Path("/usr/bin/ls")
raw_bytes = binary_path.read_bytes()
text = raw_bytes.decode('latin-1')  # Convert bytes to latin-1 string

# Tokenize
encoded = tokenizer.encode(text)
tokens = encoded.ids

print(f"File size: {len(raw_bytes):,} bytes")
print(f"Tokens: {len(tokens):,}")
print(f"Compression: {len(raw_bytes) / len(tokens):.3f} bytes/token")

# Decode back to text (note: adds spaces between tokens due to BPE behavior)
decoded = tokenizer.decode(tokens)
```

**Expected Output** (for `/usr/bin/ls`):
```
File size: 142,144 bytes
Tokens: 49,574
Compression: 2.866 bytes/token
```

### Method 2: Using transformers library

```python
from transformers import PreTrainedTokenizerFast
from tokenizers import Tokenizer

# Load the base tokenizer
base_tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-001")

# Wrap with PreTrainedTokenizerFast for transformers compatibility
tokenizer = PreTrainedTokenizerFast(tokenizer_object=base_tokenizer)

# Process binary data
with open("/usr/bin/ls", "rb") as f:
    raw_bytes = f.read()
    text = raw_bytes.decode('latin-1')

# Tokenize (returns dict with input_ids, attention_mask, etc.)
result = tokenizer(text)
tokens = result["input_ids"]
```

---

## Important: Data Format

The tokenizer expects binary data encoded as **latin-1 strings**, NOT hex strings:

```python
# ✅ CORRECT - Use latin-1 encoded bytes
raw_bytes = b'\x7fELF\x01\x01'
text = raw_bytes.decode('latin-1')  # → '\x7fELF\x01\x01'
encoded = tokenizer.encode(text)

# ❌ WRONG - Do not use hex strings
hex_str = "7f 45 4c 46 01 01"  # Will not work correctly
```

**Why latin-1?** Every byte value (0-255) maps to exactly one latin-1 character, ensuring lossless round-trip conversion between bytes and text.

---

## Performance Benchmarks

### Compression on Real-World Binaries

Tested on `/usr/bin` binaries (not in training set):

| Binary | Size | Tokens | bytes/token |
|--------|------|--------|-------------|
| bash | 1.38 MB | 535,541 | 2.698 |
| python3.12 | 7.65 MB | 2,801,226 | 2.863 |
| gcc-13 | 0.98 MB | 344,201 | 2.986 |
| ls | 0.14 MB | 49,574 | 2.866 |
| grep | 0.18 MB | 67,567 | 2.667 |

**Average**: 2.849 bytes/token

### Information-Theoretic Efficiency

- Binary entropy: ~6.5 bits/byte
- Theoretical optimal: 2.46 bytes/token
- Our performance: 2.849 bytes/token
- **Efficiency: 86%** of theoretical optimum

---

## Example: Tokenizing an ELF Header

```python
from tokenizers import Tokenizer

# Load tokenizer
tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-001")

# ELF header bytes
elf_header = b'\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00'
text = elf_header.decode('latin-1')

# Tokenize
encoded = tokenizer.encode(text)
print(f"Original bytes: {elf_header.hex()}")
print(f"Tokens: {encoded.ids}")
print(f"Token count: {len(encoded.ids)}")
print(f"Compression: {len(elf_header) / len(encoded.ids):.2f} bytes/token")

# Examine individual tokens
for token_id, token_str in zip(encoded.ids, encoded.tokens):
    token_bytes = token_str.encode('latin-1')
    print(f"  Token {token_id:5d}: {token_bytes.hex():16s} ({len(token_bytes)} bytes)")
```

---

## Token Distribution

| Length | Count | Percentage | Examples |
|--------|-------|------------|----------|
| 2 bytes | 31,528 | 48.3% | `0x48 0x8b` (REX.W prefix), `0xcc 0xcc` (int3 padding) |
| 3 bytes | 9,261 | 14.2% | `0x48 0x8b 0xc0` (MOV rax, rax) |
| 4 bytes | 11,520 | 17.6% | `0x48 0x89 0x45 0xf8` (MOV [rbp-8], rax) |
| 5+ bytes | 13,164 | 20.2% | Multi-instruction sequences, string literals |

**Average token length**: 3.651 bytes

---

## Training Details

### Dataset

**Source**: `/nas4/data/glaurung-data/binaries-small/`

- **Size**: 13 GB
- **Files**: 30,738 binaries
- **Content**: Real-world compiled binaries including system utilities, libraries, and applications

**Platform Distribution**:
- Linux: Alpine, Debian, Ubuntu (ELF format)
- Windows: 8, 10, 11 (PE format)

**Architecture Distribution**:
- x86-64 (primary)
- x86-32
- ARM64

### Training Parameters

```bash
cargo run --release --bin train -- \
  --output glaurung-tokenizer-002.json \
  /nas4/data/glaurung-data/binaries-small/ \
  --vocab-size 65536 \
  --min-frequency 4 \
  --chunk-size 8192
```

**Training Duration**: 8.46 hours on 24 cores
**Peak Memory**: 70 GB

---

## Use Cases

### ✅ Recommended For

- Binary neural language models
- Malware analysis and classification
- Reverse engineering tools
- Binary similarity detection
- Code pattern recognition
- Vulnerability research
- Firmware analysis

### ❌ Not Recommended For

- Text/source code (use text tokenizer like GPT-2, 100%+ penalty)
- Very small binaries <1KB (overhead too high)
- Real-time streaming (load time ~100ms)

---

## Comparison with Predecessor

| Metric | binary-tokenizer-005 | glaurung-binary-tokenizer-001 | Improvement |
|--------|---------------------|------------------------------|-------------|
| Vocabulary | 65,536 | 65,536 | Same |
| Training data | ~5GB mixed | 13GB binaries-small | 2.6x larger |
| bytes/token | ~2.6 | 2.849 | +9.6% |
| Platforms | Mixed | Multi-OS (Linux, Windows) | More diverse |
| Architecture awareness | Basic | Advanced (instruction-aware) | Significant |
| Documentation | Basic | Comprehensive | Extensive |

**Key Improvements**:
- Larger, more diverse training corpus
- Better cross-platform coverage
- Instruction-boundary awareness
- Production-ready quality

---

## Advanced Usage

### Batch Processing Multiple Files

```python
from tokenizers import Tokenizer
from pathlib import Path
import numpy as np

tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-001")

def tokenize_binary_file(file_path):
    """Tokenize a single binary file."""
    raw_bytes = Path(file_path).read_bytes()
    text = raw_bytes.decode('latin-1')
    encoded = tokenizer.encode(text)
    return {
        'file': file_path,
        'size_bytes': len(raw_bytes),
        'token_count': len(encoded.ids),
        'compression_ratio': len(raw_bytes) / len(encoded.ids),
        'token_ids': encoded.ids
    }

# Process directory
binary_dir = Path("/usr/bin")
results = []
for binary_path in binary_dir.glob("*"):
    if binary_path.is_file():
        try:
            result = tokenize_binary_file(binary_path)
            results.append(result)
        except Exception as e:
            print(f"Error processing {binary_path}: {e}")

# Analyze compression statistics
compression_ratios = [r['compression_ratio'] for r in results]
print(f"Mean compression: {np.mean(compression_ratios):.3f} bytes/token")
print(f"Std deviation: {np.std(compression_ratios):.3f}")
```

### Using with PyTorch/TensorFlow Models

```python
from tokenizers import Tokenizer
import torch
from pathlib import Path

tokenizer = Tokenizer.from_pretrained("mjbommar/glaurung-binary-tokenizer-001")

def prepare_binary_for_model(file_path, max_length=512):
    """Prepare binary data for neural network input."""
    raw_bytes = Path(file_path).read_bytes()
    text = raw_bytes.decode('latin-1')

    # Tokenize
    encoded = tokenizer.encode(text)
    token_ids = encoded.ids

    # Truncate or pad to max_length
    if len(token_ids) > max_length:
        token_ids = token_ids[:max_length]
    else:
        # Pad with token ID 0 (or use a dedicated padding token)
        token_ids = token_ids + [0] * (max_length - len(token_ids))

    # Convert to tensor
    return torch.tensor(token_ids, dtype=torch.long)

# Use in model
binary_tensor = prepare_binary_for_model("/usr/bin/ls", max_length=1024)
print(f"Tensor shape: {binary_tensor.shape}")  # torch.Size([1024])
```

---

## Troubleshooting

### Issue: UnicodeDecodeError when processing binary

**Solution**: Always use `latin-1` encoding, never `utf-8`:
```python
# ✅ Correct
text = raw_bytes.decode('latin-1')

# ❌ Wrong
text = raw_bytes.decode('utf-8')  # Will fail on non-UTF-8 bytes
```

### Issue: Decoded output doesn't match original

**Cause**: BPE tokenizers add spaces between tokens during decoding.

**Solution**: Use the raw token IDs and decode manually if exact byte recovery is needed:
```python
# Get tokens without spaces
tokens_no_spaces = ''.join(encoded.tokens)
original_bytes = tokens_no_spaces.encode('latin-1')
```

### Issue: Poor compression on specific binary types

**Cause**: The tokenizer may not be optimized for highly specialized formats (e.g., bytecode for Python .pyc, Java .class).

**Solution**: Consider domain-specific tokenizers for specialized formats, or use this as a general-purpose baseline.

---

## Related Projects

- **Predecessor**: [mjbommar/binary-tokenizer-005](https://huggingface.co/mjbommar/binary-tokenizer-005) - Earlier 64K binary tokenizer
- **Framework**: [mjbommar/glaurung](https://github.com/mjbommar/glaurung) - Binary analysis framework
- **Training Code**: [mjbommar/glaurung-models](https://github.com/mjbommar/glaurung-models) - Binary embedding models and tokenizers

---

## Technical Architecture

### Vocabulary Structure

- **Base tokens**: 256 single-byte tokens (0x00 to 0xFF)
- **Merged tokens**: 65,280 learned byte-pair combinations
- **Total**: 65,536 tokens (exactly 2^16)

### Special Tokens

The tokenizer includes boundary markers for file-level segmentation:
- `<|start|>` (ID: 0)
- `<|end|>` (ID: 1)

These help models distinguish between concatenated files and identify file headers.

### Token Properties

**Instruction-aware patterns** (x86-64 examples):
- REX prefixes: `0x48`, `0x4c`, `0x4d`
- Common opcodes: `0x8b` (MOV), `0x89` (MOV), `0xe8` (CALL)
- ModR/M patterns: `0xc0`, `0x45`, `0x5d`

**Common patterns**:
- Padding: `0xcc 0xcc` (int3), `0x90 0x90` (nop)
- Alignment: `0x00 0x00 0x00 0x00`
- String terminators: `0x00` at word boundaries

---

## Performance Characteristics

### Load Time

- **Tokenizer size**: 2.3 MB on disk
- **Load time**: ~100ms (cold), ~20ms (cached)
- **Memory footprint**: ~15 MB in RAM

### Encoding Speed

On a modern CPU (tested on Intel i9-12900K):

| Operation | Speed |
|-----------|-------|
| Encode 1 MB binary | ~50 ms |
| Encode 10 MB binary | ~450 ms |
| Encode 100 MB binary | ~4.2 s |

**Throughput**: ~20-25 MB/second

---

## Limitations

1. **Cross-domain penalty**: Using on text data causes 100-140% efficiency loss
2. **Small file overhead**: Files <1KB have proportionally higher tokenization overhead
3. **Deterministic decoding**: Spaces inserted between tokens during decode (BPE behavior)
4. **Architecture bias**: Trained primarily on x86-64; may be less optimal for RISC-V, MIPS, etc.

---

## Citation

If you use this tokenizer in research, please cite:

```
Glaurung Binary Tokenizer 001
64K Binary Tokenizer for Neural Language Models
Vocabulary: 65,536 tokens (exactly 2^16)
Training: October 2025
Dataset: 13GB binaries-small (30,738 files)
Performance: 2.849 bytes/token (86% of theoretical optimum)
HuggingFace: mjbommar/glaurung-binary-tokenizer-001
```

---

## License

Apache License 2.0

This tokenizer is part of the [Glaurung](https://github.com/mjbommar/glaurung) project. See the [glaurung-models repository](https://github.com/mjbommar/glaurung-models) for full license details.

---

## Support & Issues

- **GitHub Issues**: [mjbommar/glaurung-models/issues](https://github.com/mjbommar/glaurung-models/issues)
- **Documentation**: Full training report available in the [glaurung-models repository](https://github.com/mjbommar/glaurung-models/tree/master/tokenizers/tokenizer-002)
- **Email**: Contact maintainer via GitHub

---

**Production Status**: ✅ Ready for deployment
**Version**: 1.0.0
**Release Date**: October 2025