|
Download README.md from ShadeEngine/bytegpt-hnet-100M: direct link, hf CLI and curl.
- Browser
- Download file 5.3 kB
-
https://huggingface.co/ShadeEngine/bytegpt-hnet-100M/resolve/main/README.md
- Command line
-
hf download hf://ShadeEngine/bytegpt-hnet-100M/README.md
-
curl -L -o README.md https://huggingface.co/ShadeEngine/bytegpt-hnet-100M/resolve/main/README.md
5.3 kB
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| tags: | |
| - byte-level | |
| - tokenizer-free | |
| - h-net | |
| - mamba | |
| - language-modeling | |
| datasets: | |
| - ShadeEngine/fineweb-edu-bytes-10BT | |
| # H-Net, 100M parameters | |
| A language model that reads raw bytes and learns where to cut them into | |
| chunks. No tokenizer is involved. A small network reads every byte and marks | |
| chunk boundaries, a large Transformer runs once per chunk, and a small network | |
| turns the result back into byte predictions. It is the learned-chunking model | |
| in a three-way comparison of byte, BPE and H-Net models at equal compute. | |
| ## Model | |
| - One-stage H-Net, 99.8M parameters, layout `["m4", ["T12"], "m4"]` | |
| - Encoder and decoder: 4 Mamba-2 layers each, width 512, one step per byte | |
| - Main network: 12-layer Transformer, width 768, one step per chunk | |
| - Vocabulary of 256 byte values. Byte 254 marks the start of every document. | |
| - Learned chunk size: 3.8 bytes on the training distribution, 2.3 on French, | |
| 1.8 on Python code | |
| ## Training | |
| - Data: FineWeb-Edu `sample-10BT` as raw bytes, in windows of 8,192 bytes | |
| - 72,800 steps of 524,288 bytes: 38.2 GB of text, about 1.0e19 FLOPs | |
| - Loss: next-byte cross-entropy plus 0.03 times the paper's ratio loss, with a | |
| target of 4.6 bytes per chunk | |
| - AdamW, peak learning rate 6e-4 (doubled for the byte-level networks), | |
| 2,000 warmup steps, cosine decay | |
| - Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about | |
| two and a half days | |
| - The full argument list is in `config.json` | |
| ## Use | |
| The model runs on the official H-Net code, which is not on PyPI. It needs a | |
| CUDA GPU, `mamba-ssm` and `flash-attn`. | |
| ```bash | |
| git clone https://github.com/goombalab/hnet && cd hnet | |
| git checkout 3673fe1 | |
| pip install -e . | |
| pip install safetensors huggingface_hub | |
| ``` | |
| ```python | |
| import json, torch | |
| from huggingface_hub import snapshot_download | |
| from safetensors.torch import load_file | |
| from hnet.models.config_hnet import AttnConfig, HNetConfig, SSMConfig | |
| from hnet.models.mixer_seq import HNetForCausalLM | |
| path = snapshot_download("ShadeEngine/bytegpt-hnet-100M") | |
| raw = json.load(open(f"{path}/config.json"))["config"] | |
| config = HNetConfig( | |
| arch_layout=raw["arch_layout"], d_model=raw["d_model"], | |
| d_intermediate=raw["d_intermediate"], vocab_size=raw["vocab_size"], | |
| ssm_cfg=SSMConfig(**raw["ssm_cfg"]), attn_cfg=AttnConfig(**raw["attn_cfg"]), | |
| tie_embeddings=raw["tie_embeddings"], | |
| ) | |
| model = HNetForCausalLM(config, device="cuda") | |
| model.load_state_dict(load_file(f"{path}/model.safetensors", device="cuda")) | |
| model.eval() | |
| text = bytes([254]) + "The French Revolution began in 1789".encode("utf-8") | |
| ids = torch.tensor(list(text), device="cuda")[None] | |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): | |
| out = model(ids, mask=torch.ones_like(ids, dtype=torch.bool)) | |
| next_byte = out.logits[0, -1].float().argmax().item() # the model's guess for the next byte | |
| starts = out.bpred_output[0].boundary_mask[0].tolist() # True where a new chunk starts | |
| print(repr(chr(next_byte))) | |
| print("".join(("|" if s else "") + chr(b) for b, s in zip(text[1:], starts[1:]))) | |
| ``` | |
| ## Results | |
| All three models were trained with the same compute budget, about 1.0e19 | |
| floating-point operations each, on the same documents. Lower is better for | |
| every row except HellaSwag. | |
| | | [Byte Transformer](https://huggingface.co/ShadeEngine/bytegpt-byte-85M) | [BPE Transformer](https://huggingface.co/ShadeEngine/bytegpt-bpe32k-110M) | **[H-Net](https://huggingface.co/ShadeEngine/bytegpt-hnet-100M) (this model)** | | |
| |---|---:|---:|---:| | |
| | FineWeb-Edu validation, bits per byte | 1.036 | 0.922 | **0.913** | | |
| | English Wikipedia, bits per byte | 1.118 | 0.988 | **1.034** | | |
| | French Wikipedia, bits per byte | 1.944 | 1.836 | **1.865** | | |
| | Python code, bits per byte | 1.329 | 1.382 | **1.521** | | |
| | HellaSwag accuracy (1,000 questions, chance is 25%) | 35.7% | 38.7% | **38.8%** | | |
| | Typo penalty (loss with 1% typos / clean loss) | 1.240 | 1.280 | **1.241** | | |
| Bits per byte is the model's loss divided by the bytes of text it covers, so | |
| models with different tokenizers can be compared on one scale. Each test set | |
| is 5 MB of text that none of the models trained on. | |
| ## Limits | |
| - One training run per model. Differences of a percent or two could be noise. | |
| - These are small research models trained on educational web text. They write | |
| fluent English that is often factually wrong. Do not use them for anything | |
| beyond studying how such models behave. | |
| - The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6. | |
| Its main network therefore ran more steps per byte and saw less text | |
| (38 GB against 60 GB for the BPE model) for the same compute. | |
| ## More | |
| - Article: [Can a model learn its own tokenizer?](https://claude.ai/artifact/AamTLQSJGp6uYueALTKjtm) | |
| - Training and evaluation code: [https://github.com/Ganryuu/bytegpt](https://github.com/Ganryuu/bytegpt) | |
| - Training data: [ShadeEngine/fineweb-edu-bytes-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bytes-10BT) | |
| and [ShadeEngine/fineweb-edu-bpe32k-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bpe32k-10BT) | |
| - The experiment follows Cartesia's | |
| [From Tokens to Bytes](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1) posts | |
| and the [H-Net paper](https://arxiv.org/abs/2507.07955). | |