bytegpt-hnet-100M / README.md
ShadeEngine's picture
Upload folder using huggingface_hub
d2e3d97 verified
|
Raw History Blame Contribute Delete
5.3 kB
---
language:
- en
pipeline_tag: text-generation
tags:
- byte-level
- tokenizer-free
- h-net
- mamba
- language-modeling
datasets:
- ShadeEngine/fineweb-edu-bytes-10BT
---
# H-Net, 100M parameters
A language model that reads raw bytes and learns where to cut them into
chunks. No tokenizer is involved. A small network reads every byte and marks
chunk boundaries, a large Transformer runs once per chunk, and a small network
turns the result back into byte predictions. It is the learned-chunking model
in a three-way comparison of byte, BPE and H-Net models at equal compute.
## Model
- One-stage H-Net, 99.8M parameters, layout `["m4", ["T12"], "m4"]`
- Encoder and decoder: 4 Mamba-2 layers each, width 512, one step per byte
- Main network: 12-layer Transformer, width 768, one step per chunk
- Vocabulary of 256 byte values. Byte 254 marks the start of every document.
- Learned chunk size: 3.8 bytes on the training distribution, 2.3 on French,
1.8 on Python code
## Training
- Data: FineWeb-Edu `sample-10BT` as raw bytes, in windows of 8,192 bytes
- 72,800 steps of 524,288 bytes: 38.2 GB of text, about 1.0e19 FLOPs
- Loss: next-byte cross-entropy plus 0.03 times the paper's ratio loss, with a
target of 4.6 bytes per chunk
- AdamW, peak learning rate 6e-4 (doubled for the byte-level networks),
2,000 warmup steps, cosine decay
- Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about
two and a half days
- The full argument list is in `config.json`
## Use
The model runs on the official H-Net code, which is not on PyPI. It needs a
CUDA GPU, `mamba-ssm` and `flash-attn`.
```bash
git clone https://github.com/goombalab/hnet && cd hnet
git checkout 3673fe1
pip install -e .
pip install safetensors huggingface_hub
```
```python
import json, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from hnet.models.config_hnet import AttnConfig, HNetConfig, SSMConfig
from hnet.models.mixer_seq import HNetForCausalLM
path = snapshot_download("ShadeEngine/bytegpt-hnet-100M")
raw = json.load(open(f"{path}/config.json"))["config"]
config = HNetConfig(
arch_layout=raw["arch_layout"], d_model=raw["d_model"],
d_intermediate=raw["d_intermediate"], vocab_size=raw["vocab_size"],
ssm_cfg=SSMConfig(**raw["ssm_cfg"]), attn_cfg=AttnConfig(**raw["attn_cfg"]),
tie_embeddings=raw["tie_embeddings"],
)
model = HNetForCausalLM(config, device="cuda")
model.load_state_dict(load_file(f"{path}/model.safetensors", device="cuda"))
model.eval()
text = bytes([254]) + "The French Revolution began in 1789".encode("utf-8")
ids = torch.tensor(list(text), device="cuda")[None]
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
out = model(ids, mask=torch.ones_like(ids, dtype=torch.bool))
next_byte = out.logits[0, -1].float().argmax().item() # the model's guess for the next byte
starts = out.bpred_output[0].boundary_mask[0].tolist() # True where a new chunk starts
print(repr(chr(next_byte)))
print("".join(("|" if s else "") + chr(b) for b, s in zip(text[1:], starts[1:])))
```
## Results
All three models were trained with the same compute budget, about 1.0e19
floating-point operations each, on the same documents. Lower is better for
every row except HellaSwag.
| | [Byte Transformer](https://huggingface.co/ShadeEngine/bytegpt-byte-85M) | [BPE Transformer](https://huggingface.co/ShadeEngine/bytegpt-bpe32k-110M) | **[H-Net](https://huggingface.co/ShadeEngine/bytegpt-hnet-100M) (this model)** |
|---|---:|---:|---:|
| FineWeb-Edu validation, bits per byte | 1.036 | 0.922 | **0.913** |
| English Wikipedia, bits per byte | 1.118 | 0.988 | **1.034** |
| French Wikipedia, bits per byte | 1.944 | 1.836 | **1.865** |
| Python code, bits per byte | 1.329 | 1.382 | **1.521** |
| HellaSwag accuracy (1,000 questions, chance is 25%) | 35.7% | 38.7% | **38.8%** |
| Typo penalty (loss with 1% typos / clean loss) | 1.240 | 1.280 | **1.241** |
Bits per byte is the model's loss divided by the bytes of text it covers, so
models with different tokenizers can be compared on one scale. Each test set
is 5 MB of text that none of the models trained on.
## Limits
- One training run per model. Differences of a percent or two could be noise.
- These are small research models trained on educational web text. They write
fluent English that is often factually wrong. Do not use them for anything
beyond studying how such models behave.
- The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6.
Its main network therefore ran more steps per byte and saw less text
(38 GB against 60 GB for the BPE model) for the same compute.
## More
- Article: [Can a model learn its own tokenizer?](https://claude.ai/artifact/AamTLQSJGp6uYueALTKjtm)
- Training and evaluation code: [https://github.com/Ganryuu/bytegpt](https://github.com/Ganryuu/bytegpt)
- Training data: [ShadeEngine/fineweb-edu-bytes-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bytes-10BT)
and [ShadeEngine/fineweb-edu-bpe32k-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bpe32k-10BT)
- The experiment follows Cartesia's
[From Tokens to Bytes](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1) posts
and the [H-Net paper](https://arxiv.org/abs/2507.07955).