File size: 4,187 Bytes
1839975
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1ab754b
1839975
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
---
language:
- en
pipeline_tag: text-generation
tags:
- byte-level
- tokenizer-free
- language-modeling
datasets:
- ShadeEngine/fineweb-edu-bytes-10BT
---

# Byte Transformer, 85M parameters

A language model that reads and writes raw bytes. It has no tokenizer: every
UTF-8 byte of the text is one step. It is the baseline in a three-way
comparison of byte, BPE and H-Net models at equal compute.

## Model

- 12-layer Transformer, 12 heads, width 768, 85.2M parameters
- Vocabulary of 256 byte values, context of 8,192 bytes
- RMSNorm, rotary position embeddings, SwiGLU, tied input and output embeddings
- Byte 254 marks the start of every document

## Training

- Data: FineWeb-Edu `sample-10BT` as raw bytes
- 20,000 steps of 524,288 bytes: 10.5 GB of text, about 1.0e19 FLOPs
- AdamW, peak learning rate 6e-4, 500 warmup steps, cosine decay to 6e-5
- Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about a day
- The full argument list is in `config.json`

## Use

`model.py` in this repo holds the whole model. It needs `torch`,
`safetensors` and `huggingface_hub`.

```python
import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file

path = snapshot_download("ShadeEngine/bytegpt-byte-85M")
sys.path.insert(0, path)
from model import ByteGPT, GPTConfig

meta = json.load(open(f"{path}/config.json"))
weights = load_file(f"{path}/model.safetensors")
for alias, source in meta["tied_weights"].items():  # tied embeddings are stored once
    weights[alias] = weights[source]
model = ByteGPT(GPTConfig(**meta["config"]))
model.load_state_dict(weights)
model.eval().cuda()

prompt = bytes([254]) + "The French Revolution began in".encode("utf-8")
ids = torch.tensor(list(prompt), device="cuda")[None]
with torch.autocast("cuda", dtype=torch.bfloat16):
    out = model.generate(ids, max_new_bytes=200, temperature=0.8, top_k=50)
print(bytes(out[0, 1:].tolist()).decode("utf-8", errors="replace"))
```

## Results

All three models were trained with the same compute budget, about 1.0e19
floating-point operations each, on the same documents. Lower is better for
every row except HellaSwag.

| | **[Byte Transformer](https://huggingface.co/ShadeEngine/bytegpt-byte-85M) (this model)** | [BPE Transformer](https://huggingface.co/ShadeEngine/bytegpt-bpe32k-110M) | [H-Net](https://huggingface.co/ShadeEngine/bytegpt-hnet-100M) |
|---|---:|---:|---:|
| FineWeb-Edu validation, bits per byte | **1.036** | 0.922 | 0.913 |
| English Wikipedia, bits per byte | **1.118** | 0.988 | 1.034 |
| French Wikipedia, bits per byte | **1.944** | 1.836 | 1.865 |
| Python code, bits per byte | **1.329** | 1.382 | 1.521 |
| HellaSwag accuracy (1,000 questions, chance is 25%) | **35.7%** | 38.7% | 38.8% |
| Typo penalty (loss with 1% typos / clean loss) | **1.240** | 1.280 | 1.241 |

Bits per byte is the model's loss divided by the bytes of text it covers, so
models with different tokenizers can be compared on one scale. Each test set
is 5 MB of text that none of the models trained on.

## Limits

- One training run per model. Differences of a percent or two could be noise.
- These are small research models trained on educational web text. They write
  fluent English that is often factually wrong. Do not use them for anything
  beyond studying how such models behave.
- The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6.
  Its main network therefore ran more steps per byte and saw less text
  (38 GB against 60 GB for the BPE model) for the same compute.

## More

- Article: [Can a model learn its own tokenizer?](https://claude.ai/artifact/AamTLQSJGp6uYueALTKjtm)
- Training and evaluation code: [https://github.com/Ganryuu/bytegpt](https://github.com/Ganryuu/bytegpt)
- Training data: [ShadeEngine/fineweb-edu-bytes-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bytes-10BT)
  and [ShadeEngine/fineweb-edu-bpe32k-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bpe32k-10BT)
- The experiment follows Cartesia's
  [From Tokens to Bytes](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1) posts
  and the [H-Net paper](https://arxiv.org/abs/2507.07955).