|
Download README.md from ShadeEngine/bytegpt-byte-85M: direct link, hf CLI and curl.
- Browser
- Download file 4.19 kB
-
https://huggingface.co/ShadeEngine/bytegpt-byte-85M/resolve/main/README.md
- Command line
-
hf download hf://ShadeEngine/bytegpt-byte-85M/README.md
-
curl -L -o README.md https://huggingface.co/ShadeEngine/bytegpt-byte-85M/resolve/main/README.md
4.19 kB
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| tags: | |
| - byte-level | |
| - tokenizer-free | |
| - language-modeling | |
| datasets: | |
| - ShadeEngine/fineweb-edu-bytes-10BT | |
| # Byte Transformer, 85M parameters | |
| A language model that reads and writes raw bytes. It has no tokenizer: every | |
| UTF-8 byte of the text is one step. It is the baseline in a three-way | |
| comparison of byte, BPE and H-Net models at equal compute. | |
| ## Model | |
| - 12-layer Transformer, 12 heads, width 768, 85.2M parameters | |
| - Vocabulary of 256 byte values, context of 8,192 bytes | |
| - RMSNorm, rotary position embeddings, SwiGLU, tied input and output embeddings | |
| - Byte 254 marks the start of every document | |
| ## Training | |
| - Data: FineWeb-Edu `sample-10BT` as raw bytes | |
| - 20,000 steps of 524,288 bytes: 10.5 GB of text, about 1.0e19 FLOPs | |
| - AdamW, peak learning rate 6e-4, 500 warmup steps, cosine decay to 6e-5 | |
| - Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about a day | |
| - The full argument list is in `config.json` | |
| ## Use | |
| `model.py` in this repo holds the whole model. It needs `torch`, | |
| `safetensors` and `huggingface_hub`. | |
| ```python | |
| import json, sys, torch | |
| from huggingface_hub import snapshot_download | |
| from safetensors.torch import load_file | |
| path = snapshot_download("ShadeEngine/bytegpt-byte-85M") | |
| sys.path.insert(0, path) | |
| from model import ByteGPT, GPTConfig | |
| meta = json.load(open(f"{path}/config.json")) | |
| weights = load_file(f"{path}/model.safetensors") | |
| for alias, source in meta["tied_weights"].items(): # tied embeddings are stored once | |
| weights[alias] = weights[source] | |
| model = ByteGPT(GPTConfig(**meta["config"])) | |
| model.load_state_dict(weights) | |
| model.eval().cuda() | |
| prompt = bytes([254]) + "The French Revolution began in".encode("utf-8") | |
| ids = torch.tensor(list(prompt), device="cuda")[None] | |
| with torch.autocast("cuda", dtype=torch.bfloat16): | |
| out = model.generate(ids, max_new_bytes=200, temperature=0.8, top_k=50) | |
| print(bytes(out[0, 1:].tolist()).decode("utf-8", errors="replace")) | |
| ``` | |
| ## Results | |
| All three models were trained with the same compute budget, about 1.0e19 | |
| floating-point operations each, on the same documents. Lower is better for | |
| every row except HellaSwag. | |
| | | **[Byte Transformer](https://huggingface.co/ShadeEngine/bytegpt-byte-85M) (this model)** | [BPE Transformer](https://huggingface.co/ShadeEngine/bytegpt-bpe32k-110M) | [H-Net](https://huggingface.co/ShadeEngine/bytegpt-hnet-100M) | | |
| |---|---:|---:|---:| | |
| | FineWeb-Edu validation, bits per byte | **1.036** | 0.922 | 0.913 | | |
| | English Wikipedia, bits per byte | **1.118** | 0.988 | 1.034 | | |
| | French Wikipedia, bits per byte | **1.944** | 1.836 | 1.865 | | |
| | Python code, bits per byte | **1.329** | 1.382 | 1.521 | | |
| | HellaSwag accuracy (1,000 questions, chance is 25%) | **35.7%** | 38.7% | 38.8% | | |
| | Typo penalty (loss with 1% typos / clean loss) | **1.240** | 1.280 | 1.241 | | |
| Bits per byte is the model's loss divided by the bytes of text it covers, so | |
| models with different tokenizers can be compared on one scale. Each test set | |
| is 5 MB of text that none of the models trained on. | |
| ## Limits | |
| - One training run per model. Differences of a percent or two could be noise. | |
| - These are small research models trained on educational web text. They write | |
| fluent English that is often factually wrong. Do not use them for anything | |
| beyond studying how such models behave. | |
| - The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6. | |
| Its main network therefore ran more steps per byte and saw less text | |
| (38 GB against 60 GB for the BPE model) for the same compute. | |
| ## More | |
| - Article: [Can a model learn its own tokenizer?](https://claude.ai/artifact/AamTLQSJGp6uYueALTKjtm) | |
| - Training and evaluation code: [https://github.com/Ganryuu/bytegpt](https://github.com/Ganryuu/bytegpt) | |
| - Training data: [ShadeEngine/fineweb-edu-bytes-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bytes-10BT) | |
| and [ShadeEngine/fineweb-edu-bpe32k-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bpe32k-10BT) | |
| - The experiment follows Cartesia's | |
| [From Tokens to Bytes](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1) posts | |
| and the [H-Net paper](https://arxiv.org/abs/2507.07955). | |