Merge Barriers: Model Checkpoints

Four model checkpoints from the paper "Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization" (Blackwell, 2026).

Models

File	Architecture	Tokenizer	Steps	Overall PPL	Purpose
`neox-a.pt`	GPT-NeoX 410M (436M params)	structok-64k (merge barriers)	20,000	19.4	Merge-barrier model (NeoX)
`neox-b.pt`	GPT-NeoX 410M (436M params)	standard-64k (no barriers)	20,000	19.5	Standard-BPE control (NeoX)
`llama-a.pt`	Llama 410M (405M params, GQA 4:1)	structok-64k (merge barriers)	40,000	~23	Merge-barrier model (Llama)
`llama-b.pt`	Llama 410M (405M params, GQA 4:1)	standard-64k (no barriers)	40,000	~21	Standard-BPE control (Llama)

Architecture Details

GPT-NeoX 410M (run-002)

24 layers, 16 attention heads, 1024 hidden dimension
Learned position embeddings, full multi-head attention (separate Q/K/V per head)
GELU activation, LayerNorm
Context: 2,048 tokens

Llama 410M (run-003)

24 layers, 16 query heads / 4 KV heads per layer (GQA 4:1)
RoPE positional encoding, SwiGLU activation, RMSNorm
Context: 2,048 tokens

Tokenizers

Both tokenizers are included in this repo:

structok-64k.json: 65,539 vocab, 16 merge barriers (no delimiter character can participate in BPE merges)
standard-64k.json: 65,536 vocab, standard BPE (no barriers)

Usage

import torch
from transformers import AutoConfig

# Load checkpoint
checkpoint = torch.load("neox-a.pt", map_location="cpu")
# Checkpoint contains: model_state_dict, optimizer_state_dict, step, config

For ablation experiments, use the eval scripts in the merge-barriers repository:

python eval_ablation_v4_excess.py --model-a neox-a.pt --model-b neox-b.pt
python eval_llama_ablation.py --model-a llama-a.pt --model-b llama-b.pt

Training

Both model pairs were trained on the same 4.5 GB corpus (33% FineWeb, 13% code, 14% JSON, 8% GCF, 3% Wikipedia, 1% YAML/CSV, 28% additional FineWeb). Within each pair, the architecture, corpus, hyperparameters, random seed, and hardware are identical. The only difference is the tokenizer.

Citation

@article{blackwell2026mergebarriers,
  title={Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization},
  author={Blackwell, Dayna},
  year={2026},
  url={https://github.com/blackwell-systems/merge-barriers}
}

blackwell-systems
/

merge-barriers