ShadeEngine/fineweb-edu-bpe32k-10BT
Updated โข 87
A standard token-based language model. It is the same Transformer as the byte model, but it reads text through a 32,768-entry BPE tokenizer. It is the tokenizer baseline in a three-way comparison of byte, BPE and H-Net models at equal compute.
tokenizer_32k.json is a byte-level BPE tokenizer trained on 2 GB of the
same corpus. It averages 4.60 bytes per token. Token 0 starts every document.sample-10BT, tokenizedconfig.jsonmodel.py in this repo holds the whole model. It needs torch,
tokenizers, safetensors and huggingface_hub.
import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer
path = snapshot_download("ShadeEngine/bytegpt-bpe32k-110M")
sys.path.insert(0, path)
from model import ByteGPT, GPTConfig
meta = json.load(open(f"{path}/config.json"))
weights = load_file(f"{path}/model.safetensors")
for alias, source in meta["tied_weights"].items(): # tied embeddings are stored once
weights[alias] = weights[source]
model = ByteGPT(GPTConfig(**meta["config"])) # the class name is historical
model.load_state_dict(weights)
model.eval().cuda()
tok = Tokenizer.from_file(f"{path}/tokenizer_32k.json")
ids = torch.tensor([0] + tok.encode("The French Revolution began in").ids, device="cuda")[None]
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model.generate(ids, max_new_bytes=60, temperature=0.8, top_k=50) # counts tokens here
print(tok.decode(out[0, 1:].tolist()))
All three models were trained with the same compute budget, about 1.0e19 floating-point operations each, on the same documents. Lower is better for every row except HellaSwag.
| Byte Transformer | BPE Transformer (this model) | H-Net | |
|---|---|---|---|
| FineWeb-Edu validation, bits per byte | 1.036 | 0.922 | 0.913 |
| English Wikipedia, bits per byte | 1.118 | 0.988 | 1.034 |
| French Wikipedia, bits per byte | 1.944 | 1.836 | 1.865 |
| Python code, bits per byte | 1.329 | 1.382 | 1.521 |
| HellaSwag accuracy (1,000 questions, chance is 25%) | 35.7% | 38.7% | 38.8% |
| Typo penalty (loss with 1% typos / clean loss) | 1.240 | 1.280 | 1.241 |
Bits per byte is the model's loss divided by the bytes of text it covers, so models with different tokenizers can be compared on one scale. Each test set is 5 MB of text that none of the models trained on.