ShadeEngine commited on
Commit
1839975
·
verified ·
1 Parent(s): 1bcfc46

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. README.md +99 -0
  2. __pycache__/model.cpython-312.pyc +0 -0
  3. config.json +34 -0
  4. model.py +197 -0
  5. model.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - byte-level
7
+ - tokenizer-free
8
+ - language-modeling
9
+ datasets:
10
+ - ShadeEngine/fineweb-edu-bytes-10BT
11
+ ---
12
+
13
+ # Byte Transformer, 85M parameters
14
+
15
+ A language model that reads and writes raw bytes. It has no tokenizer: every
16
+ UTF-8 byte of the text is one step. It is the baseline in a three-way
17
+ comparison of byte, BPE and H-Net models at equal compute.
18
+
19
+ ## Model
20
+
21
+ - 12-layer Transformer, 12 heads, width 768, 85.2M parameters
22
+ - Vocabulary of 256 byte values, context of 8,192 bytes
23
+ - RMSNorm, rotary position embeddings, SwiGLU, tied input and output embeddings
24
+ - Byte 254 marks the start of every document
25
+
26
+ ## Training
27
+
28
+ - Data: FineWeb-Edu `sample-10BT` as raw bytes
29
+ - 20,000 steps of 524,288 bytes: 10.5 GB of text, about 1.0e19 FLOPs
30
+ - AdamW, peak learning rate 6e-4, 500 warmup steps, cosine decay to 6e-5
31
+ - Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about a day
32
+ - The full argument list is in `config.json`
33
+
34
+ ## Use
35
+
36
+ `model.py` in this repo holds the whole model. It needs `torch`,
37
+ `safetensors` and `huggingface_hub`.
38
+
39
+ ```python
40
+ import json, sys, torch
41
+ from huggingface_hub import snapshot_download
42
+ from safetensors.torch import load_file
43
+
44
+ path = snapshot_download("ShadeEngine/bytegpt-byte-85M")
45
+ sys.path.insert(0, path)
46
+ from model import ByteGPT, GPTConfig
47
+
48
+ meta = json.load(open(f"{path}/config.json"))
49
+ weights = load_file(f"{path}/model.safetensors")
50
+ for alias, source in meta["tied_weights"].items(): # tied embeddings are stored once
51
+ weights[alias] = weights[source]
52
+ model = ByteGPT(GPTConfig(**meta["config"]))
53
+ model.load_state_dict(weights)
54
+ model.eval().cuda()
55
+
56
+ prompt = bytes([254]) + "The French Revolution began in".encode("utf-8")
57
+ ids = torch.tensor(list(prompt), device="cuda")[None]
58
+ with torch.autocast("cuda", dtype=torch.bfloat16):
59
+ out = model.generate(ids, max_new_bytes=200, temperature=0.8, top_k=50)
60
+ print(bytes(out[0, 1:].tolist()).decode("utf-8", errors="replace"))
61
+ ```
62
+
63
+ ## Results
64
+
65
+ All three models were trained with the same compute budget, about 1.0e19
66
+ floating-point operations each, on the same documents. Lower is better for
67
+ every row except HellaSwag.
68
+
69
+ | | **[Byte Transformer](https://huggingface.co/ShadeEngine/bytegpt-byte-85M) (this model)** | [BPE Transformer](https://huggingface.co/ShadeEngine/bytegpt-bpe32k-110M) | [H-Net](https://huggingface.co/ShadeEngine/bytegpt-hnet-100M) |
70
+ |---|---:|---:|---:|
71
+ | FineWeb-Edu validation, bits per byte | **1.036** | 0.922 | 0.913 |
72
+ | English Wikipedia, bits per byte | **1.118** | 0.988 | 1.034 |
73
+ | French Wikipedia, bits per byte | **1.944** | 1.836 | 1.865 |
74
+ | Python code, bits per byte | **1.329** | 1.382 | 1.521 |
75
+ | HellaSwag accuracy (1,000 questions, chance is 25%) | **35.7%** | 38.7% | 38.8% |
76
+ | Typo penalty (loss with 1% typos / clean loss) | **1.240** | 1.280 | 1.241 |
77
+
78
+ Bits per byte is the model's loss divided by the bytes of text it covers, so
79
+ models with different tokenizers can be compared on one scale. Each test set
80
+ is 5 MB of text that none of the models trained on.
81
+
82
+ ## Limits
83
+
84
+ - One training run per model. Differences of a percent or two could be noise.
85
+ - These are small research models trained on educational web text. They write
86
+ fluent English that is often factually wrong. Do not use them for anything
87
+ beyond studying how such models behave.
88
+ - The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6.
89
+ Its main network therefore ran more steps per byte and saw less text
90
+ (38 GB against 60 GB for the BPE model) for the same compute.
91
+
92
+ ## More
93
+
94
+ - Article: [Can a model learn its own tokenizer?](https://claude.ai/artifact/AamTLQSJGp6uYueALTKjtm)
95
+ - Training data: [ShadeEngine/fineweb-edu-bytes-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bytes-10BT)
96
+ and [ShadeEngine/fineweb-edu-bpe32k-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bpe32k-10BT)
97
+ - The experiment follows Cartesia's
98
+ [From Tokens to Bytes](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1) posts
99
+ and the [H-Net paper](https://arxiv.org/abs/2507.07955).
__pycache__/model.cpython-312.pyc ADDED
Binary file (14.9 kB). View file
 
config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architecture": "ByteGPT",
3
+ "config": {
4
+ "vocab_size": 256,
5
+ "block_size": 8192,
6
+ "n_layer": 12,
7
+ "n_head": 12,
8
+ "n_embd": 768,
9
+ "dropout": 0.0,
10
+ "rope_theta": 100000.0
11
+ },
12
+ "iter": 19999,
13
+ "n_params": 85150464,
14
+ "tied_weights": {
15
+ "lm_head.weight": "embed.weight"
16
+ },
17
+ "training_args": {
18
+ "preset": "small",
19
+ "micro_bs": 4,
20
+ "grad_accum": 8,
21
+ "max_iters": 20000,
22
+ "lr": 0.0006,
23
+ "min_lr_frac": 0.1,
24
+ "warmup_iters": 500,
25
+ "weight_decay": 0.1,
26
+ "grad_clip": 1.0,
27
+ "eval_every": 500,
28
+ "eval_windows": 64,
29
+ "ckpt_every": 2000,
30
+ "log_every": 20,
31
+ "compile": true,
32
+ "seed": 1337
33
+ }
34
+ }
model.py ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Byte-level GPT. nanoGPT-style minimalism, modern internals:
3
+ - vocab = 256 (raw bytes). No tokenizer anywhere.
4
+ - RMSNorm (pre-norm), RoPE, SwiGLU MLP, no biases
5
+ - flash attention via F.scaled_dot_product_attention
6
+ - weight tying between byte embedding and output head
7
+
8
+ The byte embedding table is 256 x d_model -- essentially free. In a BPE
9
+ model, the 50k x d_model embedding can be a third of all parameters; here
10
+ virtually all parameters do actual computation.
11
+ """
12
+
13
+ import math
14
+ from dataclasses import dataclass
15
+
16
+ import torch
17
+ import torch.nn as nn
18
+ import torch.nn.functional as F
19
+
20
+
21
+ @dataclass
22
+ class GPTConfig:
23
+ vocab_size: int = 256
24
+ block_size: int = 8192 # context length in bytes
25
+ n_layer: int = 12
26
+ n_head: int = 12
27
+ n_embd: int = 768
28
+ dropout: float = 0.0
29
+ rope_theta: float = 100000.0
30
+
31
+
32
+ PRESETS = {
33
+ # ~10M params: smoke tests and pipeline debugging
34
+ "micro": dict(n_layer=6, n_head=6, n_embd=384, block_size=2048),
35
+ # ~29M params: scaling-ladder point
36
+ "mini": dict(n_layer=9, n_head=9, n_embd=576, block_size=8192),
37
+ # ~85M non-embedding params: GPT-2 small compute equivalent
38
+ "small": dict(n_layer=12, n_head=12, n_embd=768, block_size=8192),
39
+ # BPE twin of "small": same backbone, 32k vocab (+25M embedding params),
40
+ # 2048-token context ~ 9.5KB of text (information-matched to 8192 bytes)
41
+ "small-bpe": dict(n_layer=12, n_head=12, n_embd=768, block_size=2048,
42
+ vocab_size=32768),
43
+ }
44
+
45
+
46
+ class RMSNorm(nn.Module):
47
+ def __init__(self, dim: int):
48
+ super().__init__()
49
+ self.weight = nn.Parameter(torch.ones(dim))
50
+
51
+ def forward(self, x):
52
+ dtype = x.dtype
53
+ x = x.float()
54
+ x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-6)
55
+ return (x * self.weight.float()).to(dtype)
56
+
57
+
58
+ def precompute_rope(head_dim: int, max_seq: int, theta: float):
59
+ """Rotation frequencies for RoPE: positions are encoded by rotating
60
+ q/k pairs of dims, so attention scores depend on *relative* offsets."""
61
+ inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
62
+ t = torch.arange(max_seq).float()
63
+ freqs = torch.outer(t, inv_freq) # (max_seq, head_dim/2)
64
+ return torch.cos(freqs), torch.sin(freqs)
65
+
66
+
67
+ def apply_rope(x, cos, sin):
68
+ # x: (B, n_head, T, head_dim)
69
+ T = x.size(2)
70
+ cos, sin = cos[:T].to(x.dtype), sin[:T].to(x.dtype)
71
+ x1, x2 = x.chunk(2, dim=-1)
72
+ return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1)
73
+
74
+
75
+ class Attention(nn.Module):
76
+ def __init__(self, cfg: GPTConfig):
77
+ super().__init__()
78
+ assert cfg.n_embd % cfg.n_head == 0
79
+ self.n_head = cfg.n_head
80
+ self.head_dim = cfg.n_embd // cfg.n_head
81
+ self.wqkv = nn.Linear(cfg.n_embd, 3 * cfg.n_embd, bias=False)
82
+ self.wo = nn.Linear(cfg.n_embd, cfg.n_embd, bias=False)
83
+ self.dropout = cfg.dropout
84
+
85
+ def forward(self, x, cos, sin):
86
+ B, T, C = x.shape
87
+ q, k, v = self.wqkv(x).split(C, dim=2)
88
+ q = q.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
89
+ k = k.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
90
+ v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
91
+ q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
92
+ y = F.scaled_dot_product_attention(
93
+ q, k, v, is_causal=True,
94
+ dropout_p=self.dropout if self.training else 0.0,
95
+ )
96
+ y = y.transpose(1, 2).contiguous().view(B, T, C)
97
+ return self.wo(y)
98
+
99
+
100
+ class SwiGLU(nn.Module):
101
+ """Gated MLP: out = W2( silu(W_gate x) * (W_up x) ).
102
+ Hidden dim chosen ~8/3 * d_model (rounded to 128) so total params match
103
+ the classic 4x GELU MLP."""
104
+
105
+ def __init__(self, cfg: GPTConfig):
106
+ super().__init__()
107
+ hidden = int(8 * cfg.n_embd / 3)
108
+ hidden = 128 * round(hidden / 128)
109
+ self.w_gate = nn.Linear(cfg.n_embd, hidden, bias=False)
110
+ self.w_up = nn.Linear(cfg.n_embd, hidden, bias=False)
111
+ self.w_down = nn.Linear(hidden, cfg.n_embd, bias=False)
112
+
113
+ def forward(self, x):
114
+ return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
115
+
116
+
117
+ class Block(nn.Module):
118
+ def __init__(self, cfg: GPTConfig):
119
+ super().__init__()
120
+ self.norm1 = RMSNorm(cfg.n_embd)
121
+ self.attn = Attention(cfg)
122
+ self.norm2 = RMSNorm(cfg.n_embd)
123
+ self.mlp = SwiGLU(cfg)
124
+
125
+ def forward(self, x, cos, sin):
126
+ x = x + self.attn(self.norm1(x), cos, sin)
127
+ x = x + self.mlp(self.norm2(x))
128
+ return x
129
+
130
+
131
+ class ByteGPT(nn.Module):
132
+ def __init__(self, cfg: GPTConfig):
133
+ super().__init__()
134
+ self.cfg = cfg
135
+ self.embed = nn.Embedding(cfg.vocab_size, cfg.n_embd)
136
+ self.blocks = nn.ModuleList(Block(cfg) for _ in range(cfg.n_layer))
137
+ self.norm_f = RMSNorm(cfg.n_embd)
138
+ self.lm_head = nn.Linear(cfg.n_embd, cfg.vocab_size, bias=False)
139
+ self.lm_head.weight = self.embed.weight # weight tying
140
+
141
+ cos, sin = precompute_rope(cfg.n_embd // cfg.n_head, cfg.block_size, cfg.rope_theta)
142
+ self.register_buffer("rope_cos", cos, persistent=False)
143
+ self.register_buffer("rope_sin", sin, persistent=False)
144
+
145
+ self.apply(self._init_weights)
146
+ # scale down residual-branch output projections (GPT-2 trick):
147
+ # keeps the residual stream variance stable across depth
148
+ for pn, p in self.named_parameters():
149
+ if pn.endswith("wo.weight") or pn.endswith("w_down.weight"):
150
+ nn.init.normal_(p, std=0.02 / math.sqrt(2 * cfg.n_layer))
151
+
152
+ def _init_weights(self, m):
153
+ if isinstance(m, nn.Linear):
154
+ nn.init.normal_(m.weight, std=0.02)
155
+ elif isinstance(m, nn.Embedding):
156
+ nn.init.normal_(m.weight, std=0.02)
157
+
158
+ def num_params(self):
159
+ return sum(p.numel() for p in self.parameters())
160
+
161
+ def forward(self, idx, targets=None):
162
+ x = self.embed(idx)
163
+ for blk in self.blocks:
164
+ x = blk(x, self.rope_cos, self.rope_sin)
165
+ x = self.norm_f(x)
166
+ if targets is not None:
167
+ logits = self.lm_head(x)
168
+ loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1))
169
+ return logits, loss
170
+ # inference: only the last position is needed
171
+ logits = self.lm_head(x[:, [-1], :])
172
+ return logits, None
173
+
174
+ def configure_optimizer(self, weight_decay, lr, betas, device_type):
175
+ # decay only matrices; norms and embeddings stay undecayed
176
+ decay = [p for n, p in self.named_parameters() if p.requires_grad and p.dim() >= 2]
177
+ nodecay = [p for n, p in self.named_parameters() if p.requires_grad and p.dim() < 2]
178
+ groups = [
179
+ {"params": decay, "weight_decay": weight_decay},
180
+ {"params": nodecay, "weight_decay": 0.0},
181
+ ]
182
+ fused = device_type == "cuda"
183
+ return torch.optim.AdamW(groups, lr=lr, betas=betas, fused=fused)
184
+
185
+ @torch.no_grad()
186
+ def generate(self, idx, max_new_bytes, temperature=1.0, top_k=None):
187
+ for _ in range(max_new_bytes):
188
+ idx_cond = idx[:, -self.cfg.block_size:]
189
+ logits, _ = self(idx_cond)
190
+ logits = logits[:, -1, :] / max(temperature, 1e-6)
191
+ if top_k is not None:
192
+ v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
193
+ logits[logits < v[:, [-1]]] = -float("inf")
194
+ probs = F.softmax(logits, dim=-1)
195
+ nxt = torch.multinomial(probs, num_samples=1)
196
+ idx = torch.cat([idx, nxt], dim=1)
197
+ return idx
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c2906a71cba8b32c123466715777d148a8e7125f5a5ddd871a790aad17c9e2d7
3
+ size 340610152