ShadeEngine/fineweb-edu-bytes-10BT
Updated โข 80
A language model that reads and writes raw bytes. It has no tokenizer: every UTF-8 byte of the text is one step. It is the baseline in a three-way comparison of byte, BPE and H-Net models at equal compute.
sample-10BT as raw bytesconfig.jsonmodel.py in this repo holds the whole model. It needs torch,
safetensors and huggingface_hub.
import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
path = snapshot_download("ShadeEngine/bytegpt-byte-85M")
sys.path.insert(0, path)
from model import ByteGPT, GPTConfig
meta = json.load(open(f"{path}/config.json"))
weights = load_file(f"{path}/model.safetensors")
for alias, source in meta["tied_weights"].items(): # tied embeddings are stored once
weights[alias] = weights[source]
model = ByteGPT(GPTConfig(**meta["config"]))
model.load_state_dict(weights)
model.eval().cuda()
prompt = bytes([254]) + "The French Revolution began in".encode("utf-8")
ids = torch.tensor(list(prompt), device="cuda")[None]
with torch.autocast("cuda", dtype=torch.bfloat16):
out = model.generate(ids, max_new_bytes=200, temperature=0.8, top_k=50)
print(bytes(out[0, 1:].tolist()).decode("utf-8", errors="replace"))
All three models were trained with the same compute budget, about 1.0e19 floating-point operations each, on the same documents. Lower is better for every row except HellaSwag.
| Byte Transformer (this model) | BPE Transformer | H-Net | |
|---|---|---|---|
| FineWeb-Edu validation, bits per byte | 1.036 | 0.922 | 0.913 |
| English Wikipedia, bits per byte | 1.118 | 0.988 | 1.034 |
| French Wikipedia, bits per byte | 1.944 | 1.836 | 1.865 |
| Python code, bits per byte | 1.329 | 1.382 | 1.521 |
| HellaSwag accuracy (1,000 questions, chance is 25%) | 35.7% | 38.7% | 38.8% |
| Typo penalty (loss with 1% typos / clean loss) | 1.240 | 1.280 | 1.241 |
Bits per byte is the model's loss divided by the bytes of text it covers, so models with different tokenizers can be compared on one scale. Each test set is 5 MB of text that none of the models trained on.