BPE Transformer, 110M parameters

A standard token-based language model. It is the same Transformer as the byte model, but it reads text through a 32,768-entry BPE tokenizer. It is the tokenizer baseline in a three-way comparison of byte, BPE and H-Net models at equal compute.

Model

  • 12-layer Transformer, 12 heads, width 768, 110.1M parameters
  • Vocabulary of 32,768 tokens, context of 2,048 tokens (about 9,400 bytes)
  • RMSNorm, rotary position embeddings, SwiGLU, tied input and output embeddings
  • tokenizer_32k.json is a byte-level BPE tokenizer trained on 2 GB of the same corpus. It averages 4.60 bytes per token. Token 0 starts every document.

Training

  • Data: FineWeb-Edu sample-10BT, tokenized
  • 25,000 steps of 524,288 tokens: 13.1B tokens, about 60 GB of text and 1.3 passes over the corpus, for about 1.0e19 FLOPs
  • AdamW, peak learning rate 6e-4, 500 warmup steps, cosine decay to 6e-5
  • Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about a day
  • The full argument list is in config.json

Use

model.py in this repo holds the whole model. It needs torch, tokenizers, safetensors and huggingface_hub.

import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

path = snapshot_download("ShadeEngine/bytegpt-bpe32k-110M")
sys.path.insert(0, path)
from model import ByteGPT, GPTConfig

meta = json.load(open(f"{path}/config.json"))
weights = load_file(f"{path}/model.safetensors")
for alias, source in meta["tied_weights"].items():  # tied embeddings are stored once
    weights[alias] = weights[source]
model = ByteGPT(GPTConfig(**meta["config"]))  # the class name is historical
model.load_state_dict(weights)
model.eval().cuda()

tok = Tokenizer.from_file(f"{path}/tokenizer_32k.json")
ids = torch.tensor([0] + tok.encode("The French Revolution began in").ids, device="cuda")[None]
with torch.autocast("cuda", dtype=torch.bfloat16):
    out = model.generate(ids, max_new_bytes=60, temperature=0.8, top_k=50)  # counts tokens here
print(tok.decode(out[0, 1:].tolist()))

Results

All three models were trained with the same compute budget, about 1.0e19 floating-point operations each, on the same documents. Lower is better for every row except HellaSwag.

Byte Transformer BPE Transformer (this model) H-Net
FineWeb-Edu validation, bits per byte 1.036 0.922 0.913
English Wikipedia, bits per byte 1.118 0.988 1.034
French Wikipedia, bits per byte 1.944 1.836 1.865
Python code, bits per byte 1.329 1.382 1.521
HellaSwag accuracy (1,000 questions, chance is 25%) 35.7% 38.7% 38.8%
Typo penalty (loss with 1% typos / clean loss) 1.240 1.280 1.241

Bits per byte is the model's loss divided by the bytes of text it covers, so models with different tokenizers can be compared on one scale. Each test set is 5 MB of text that none of the models trained on.

Limits

  • One training run per model. Differences of a percent or two could be noise.
  • These are small research models trained on educational web text. They write fluent English that is often factually wrong. Do not use them for anything beyond studying how such models behave.
  • The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6. Its main network therefore ran more steps per byte and saw less text (38 GB against 60 GB for the BPE model) for the same compute.

More

Downloads last month
33
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Dataset used to train ShadeEngine/bytegpt-bpe32k-110M

Paper for ShadeEngine/bytegpt-bpe32k-110M