Upload folder using huggingface_hub
Browse files- README.md +99 -0
- __pycache__/model.cpython-312.pyc +0 -0
- config.json +34 -0
- model.py +197 -0
- model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
tags:
|
| 6 |
+
- byte-level
|
| 7 |
+
- tokenizer-free
|
| 8 |
+
- language-modeling
|
| 9 |
+
datasets:
|
| 10 |
+
- ShadeEngine/fineweb-edu-bytes-10BT
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# Byte Transformer, 85M parameters
|
| 14 |
+
|
| 15 |
+
A language model that reads and writes raw bytes. It has no tokenizer: every
|
| 16 |
+
UTF-8 byte of the text is one step. It is the baseline in a three-way
|
| 17 |
+
comparison of byte, BPE and H-Net models at equal compute.
|
| 18 |
+
|
| 19 |
+
## Model
|
| 20 |
+
|
| 21 |
+
- 12-layer Transformer, 12 heads, width 768, 85.2M parameters
|
| 22 |
+
- Vocabulary of 256 byte values, context of 8,192 bytes
|
| 23 |
+
- RMSNorm, rotary position embeddings, SwiGLU, tied input and output embeddings
|
| 24 |
+
- Byte 254 marks the start of every document
|
| 25 |
+
|
| 26 |
+
## Training
|
| 27 |
+
|
| 28 |
+
- Data: FineWeb-Edu `sample-10BT` as raw bytes
|
| 29 |
+
- 20,000 steps of 524,288 bytes: 10.5 GB of text, about 1.0e19 FLOPs
|
| 30 |
+
- AdamW, peak learning rate 6e-4, 500 warmup steps, cosine decay to 6e-5
|
| 31 |
+
- Weight decay 0.1, gradient clipping 1.0, bfloat16, two RTX 3090s for about a day
|
| 32 |
+
- The full argument list is in `config.json`
|
| 33 |
+
|
| 34 |
+
## Use
|
| 35 |
+
|
| 36 |
+
`model.py` in this repo holds the whole model. It needs `torch`,
|
| 37 |
+
`safetensors` and `huggingface_hub`.
|
| 38 |
+
|
| 39 |
+
```python
|
| 40 |
+
import json, sys, torch
|
| 41 |
+
from huggingface_hub import snapshot_download
|
| 42 |
+
from safetensors.torch import load_file
|
| 43 |
+
|
| 44 |
+
path = snapshot_download("ShadeEngine/bytegpt-byte-85M")
|
| 45 |
+
sys.path.insert(0, path)
|
| 46 |
+
from model import ByteGPT, GPTConfig
|
| 47 |
+
|
| 48 |
+
meta = json.load(open(f"{path}/config.json"))
|
| 49 |
+
weights = load_file(f"{path}/model.safetensors")
|
| 50 |
+
for alias, source in meta["tied_weights"].items(): # tied embeddings are stored once
|
| 51 |
+
weights[alias] = weights[source]
|
| 52 |
+
model = ByteGPT(GPTConfig(**meta["config"]))
|
| 53 |
+
model.load_state_dict(weights)
|
| 54 |
+
model.eval().cuda()
|
| 55 |
+
|
| 56 |
+
prompt = bytes([254]) + "The French Revolution began in".encode("utf-8")
|
| 57 |
+
ids = torch.tensor(list(prompt), device="cuda")[None]
|
| 58 |
+
with torch.autocast("cuda", dtype=torch.bfloat16):
|
| 59 |
+
out = model.generate(ids, max_new_bytes=200, temperature=0.8, top_k=50)
|
| 60 |
+
print(bytes(out[0, 1:].tolist()).decode("utf-8", errors="replace"))
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
## Results
|
| 64 |
+
|
| 65 |
+
All three models were trained with the same compute budget, about 1.0e19
|
| 66 |
+
floating-point operations each, on the same documents. Lower is better for
|
| 67 |
+
every row except HellaSwag.
|
| 68 |
+
|
| 69 |
+
| | **[Byte Transformer](https://huggingface.co/ShadeEngine/bytegpt-byte-85M) (this model)** | [BPE Transformer](https://huggingface.co/ShadeEngine/bytegpt-bpe32k-110M) | [H-Net](https://huggingface.co/ShadeEngine/bytegpt-hnet-100M) |
|
| 70 |
+
|---|---:|---:|---:|
|
| 71 |
+
| FineWeb-Edu validation, bits per byte | **1.036** | 0.922 | 0.913 |
|
| 72 |
+
| English Wikipedia, bits per byte | **1.118** | 0.988 | 1.034 |
|
| 73 |
+
| French Wikipedia, bits per byte | **1.944** | 1.836 | 1.865 |
|
| 74 |
+
| Python code, bits per byte | **1.329** | 1.382 | 1.521 |
|
| 75 |
+
| HellaSwag accuracy (1,000 questions, chance is 25%) | **35.7%** | 38.7% | 38.8% |
|
| 76 |
+
| Typo penalty (loss with 1% typos / clean loss) | **1.240** | 1.280 | 1.241 |
|
| 77 |
+
|
| 78 |
+
Bits per byte is the model's loss divided by the bytes of text it covers, so
|
| 79 |
+
models with different tokenizers can be compared on one scale. Each test set
|
| 80 |
+
is 5 MB of text that none of the models trained on.
|
| 81 |
+
|
| 82 |
+
## Limits
|
| 83 |
+
|
| 84 |
+
- One training run per model. Differences of a percent or two could be noise.
|
| 85 |
+
- These are small research models trained on educational web text. They write
|
| 86 |
+
fluent English that is often factually wrong. Do not use them for anything
|
| 87 |
+
beyond studying how such models behave.
|
| 88 |
+
- The H-Net settled at 3.7 bytes per chunk, shorter than the tokenizer's 4.6.
|
| 89 |
+
Its main network therefore ran more steps per byte and saw less text
|
| 90 |
+
(38 GB against 60 GB for the BPE model) for the same compute.
|
| 91 |
+
|
| 92 |
+
## More
|
| 93 |
+
|
| 94 |
+
- Article: [Can a model learn its own tokenizer?](https://claude.ai/artifact/AamTLQSJGp6uYueALTKjtm)
|
| 95 |
+
- Training data: [ShadeEngine/fineweb-edu-bytes-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bytes-10BT)
|
| 96 |
+
and [ShadeEngine/fineweb-edu-bpe32k-10BT](https://huggingface.co/datasets/ShadeEngine/fineweb-edu-bpe32k-10BT)
|
| 97 |
+
- The experiment follows Cartesia's
|
| 98 |
+
[From Tokens to Bytes](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1) posts
|
| 99 |
+
and the [H-Net paper](https://arxiv.org/abs/2507.07955).
|
__pycache__/model.cpython-312.pyc
ADDED
|
Binary file (14.9 kB). View file
|
|
|
config.json
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architecture": "ByteGPT",
|
| 3 |
+
"config": {
|
| 4 |
+
"vocab_size": 256,
|
| 5 |
+
"block_size": 8192,
|
| 6 |
+
"n_layer": 12,
|
| 7 |
+
"n_head": 12,
|
| 8 |
+
"n_embd": 768,
|
| 9 |
+
"dropout": 0.0,
|
| 10 |
+
"rope_theta": 100000.0
|
| 11 |
+
},
|
| 12 |
+
"iter": 19999,
|
| 13 |
+
"n_params": 85150464,
|
| 14 |
+
"tied_weights": {
|
| 15 |
+
"lm_head.weight": "embed.weight"
|
| 16 |
+
},
|
| 17 |
+
"training_args": {
|
| 18 |
+
"preset": "small",
|
| 19 |
+
"micro_bs": 4,
|
| 20 |
+
"grad_accum": 8,
|
| 21 |
+
"max_iters": 20000,
|
| 22 |
+
"lr": 0.0006,
|
| 23 |
+
"min_lr_frac": 0.1,
|
| 24 |
+
"warmup_iters": 500,
|
| 25 |
+
"weight_decay": 0.1,
|
| 26 |
+
"grad_clip": 1.0,
|
| 27 |
+
"eval_every": 500,
|
| 28 |
+
"eval_windows": 64,
|
| 29 |
+
"ckpt_every": 2000,
|
| 30 |
+
"log_every": 20,
|
| 31 |
+
"compile": true,
|
| 32 |
+
"seed": 1337
|
| 33 |
+
}
|
| 34 |
+
}
|
model.py
ADDED
|
@@ -0,0 +1,197 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Byte-level GPT. nanoGPT-style minimalism, modern internals:
|
| 3 |
+
- vocab = 256 (raw bytes). No tokenizer anywhere.
|
| 4 |
+
- RMSNorm (pre-norm), RoPE, SwiGLU MLP, no biases
|
| 5 |
+
- flash attention via F.scaled_dot_product_attention
|
| 6 |
+
- weight tying between byte embedding and output head
|
| 7 |
+
|
| 8 |
+
The byte embedding table is 256 x d_model -- essentially free. In a BPE
|
| 9 |
+
model, the 50k x d_model embedding can be a third of all parameters; here
|
| 10 |
+
virtually all parameters do actual computation.
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
import math
|
| 14 |
+
from dataclasses import dataclass
|
| 15 |
+
|
| 16 |
+
import torch
|
| 17 |
+
import torch.nn as nn
|
| 18 |
+
import torch.nn.functional as F
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
@dataclass
|
| 22 |
+
class GPTConfig:
|
| 23 |
+
vocab_size: int = 256
|
| 24 |
+
block_size: int = 8192 # context length in bytes
|
| 25 |
+
n_layer: int = 12
|
| 26 |
+
n_head: int = 12
|
| 27 |
+
n_embd: int = 768
|
| 28 |
+
dropout: float = 0.0
|
| 29 |
+
rope_theta: float = 100000.0
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
PRESETS = {
|
| 33 |
+
# ~10M params: smoke tests and pipeline debugging
|
| 34 |
+
"micro": dict(n_layer=6, n_head=6, n_embd=384, block_size=2048),
|
| 35 |
+
# ~29M params: scaling-ladder point
|
| 36 |
+
"mini": dict(n_layer=9, n_head=9, n_embd=576, block_size=8192),
|
| 37 |
+
# ~85M non-embedding params: GPT-2 small compute equivalent
|
| 38 |
+
"small": dict(n_layer=12, n_head=12, n_embd=768, block_size=8192),
|
| 39 |
+
# BPE twin of "small": same backbone, 32k vocab (+25M embedding params),
|
| 40 |
+
# 2048-token context ~ 9.5KB of text (information-matched to 8192 bytes)
|
| 41 |
+
"small-bpe": dict(n_layer=12, n_head=12, n_embd=768, block_size=2048,
|
| 42 |
+
vocab_size=32768),
|
| 43 |
+
}
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
class RMSNorm(nn.Module):
|
| 47 |
+
def __init__(self, dim: int):
|
| 48 |
+
super().__init__()
|
| 49 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 50 |
+
|
| 51 |
+
def forward(self, x):
|
| 52 |
+
dtype = x.dtype
|
| 53 |
+
x = x.float()
|
| 54 |
+
x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + 1e-6)
|
| 55 |
+
return (x * self.weight.float()).to(dtype)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def precompute_rope(head_dim: int, max_seq: int, theta: float):
|
| 59 |
+
"""Rotation frequencies for RoPE: positions are encoded by rotating
|
| 60 |
+
q/k pairs of dims, so attention scores depend on *relative* offsets."""
|
| 61 |
+
inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
|
| 62 |
+
t = torch.arange(max_seq).float()
|
| 63 |
+
freqs = torch.outer(t, inv_freq) # (max_seq, head_dim/2)
|
| 64 |
+
return torch.cos(freqs), torch.sin(freqs)
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def apply_rope(x, cos, sin):
|
| 68 |
+
# x: (B, n_head, T, head_dim)
|
| 69 |
+
T = x.size(2)
|
| 70 |
+
cos, sin = cos[:T].to(x.dtype), sin[:T].to(x.dtype)
|
| 71 |
+
x1, x2 = x.chunk(2, dim=-1)
|
| 72 |
+
return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1)
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
class Attention(nn.Module):
|
| 76 |
+
def __init__(self, cfg: GPTConfig):
|
| 77 |
+
super().__init__()
|
| 78 |
+
assert cfg.n_embd % cfg.n_head == 0
|
| 79 |
+
self.n_head = cfg.n_head
|
| 80 |
+
self.head_dim = cfg.n_embd // cfg.n_head
|
| 81 |
+
self.wqkv = nn.Linear(cfg.n_embd, 3 * cfg.n_embd, bias=False)
|
| 82 |
+
self.wo = nn.Linear(cfg.n_embd, cfg.n_embd, bias=False)
|
| 83 |
+
self.dropout = cfg.dropout
|
| 84 |
+
|
| 85 |
+
def forward(self, x, cos, sin):
|
| 86 |
+
B, T, C = x.shape
|
| 87 |
+
q, k, v = self.wqkv(x).split(C, dim=2)
|
| 88 |
+
q = q.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
|
| 89 |
+
k = k.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
|
| 90 |
+
v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
|
| 91 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 92 |
+
y = F.scaled_dot_product_attention(
|
| 93 |
+
q, k, v, is_causal=True,
|
| 94 |
+
dropout_p=self.dropout if self.training else 0.0,
|
| 95 |
+
)
|
| 96 |
+
y = y.transpose(1, 2).contiguous().view(B, T, C)
|
| 97 |
+
return self.wo(y)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
class SwiGLU(nn.Module):
|
| 101 |
+
"""Gated MLP: out = W2( silu(W_gate x) * (W_up x) ).
|
| 102 |
+
Hidden dim chosen ~8/3 * d_model (rounded to 128) so total params match
|
| 103 |
+
the classic 4x GELU MLP."""
|
| 104 |
+
|
| 105 |
+
def __init__(self, cfg: GPTConfig):
|
| 106 |
+
super().__init__()
|
| 107 |
+
hidden = int(8 * cfg.n_embd / 3)
|
| 108 |
+
hidden = 128 * round(hidden / 128)
|
| 109 |
+
self.w_gate = nn.Linear(cfg.n_embd, hidden, bias=False)
|
| 110 |
+
self.w_up = nn.Linear(cfg.n_embd, hidden, bias=False)
|
| 111 |
+
self.w_down = nn.Linear(hidden, cfg.n_embd, bias=False)
|
| 112 |
+
|
| 113 |
+
def forward(self, x):
|
| 114 |
+
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
class Block(nn.Module):
|
| 118 |
+
def __init__(self, cfg: GPTConfig):
|
| 119 |
+
super().__init__()
|
| 120 |
+
self.norm1 = RMSNorm(cfg.n_embd)
|
| 121 |
+
self.attn = Attention(cfg)
|
| 122 |
+
self.norm2 = RMSNorm(cfg.n_embd)
|
| 123 |
+
self.mlp = SwiGLU(cfg)
|
| 124 |
+
|
| 125 |
+
def forward(self, x, cos, sin):
|
| 126 |
+
x = x + self.attn(self.norm1(x), cos, sin)
|
| 127 |
+
x = x + self.mlp(self.norm2(x))
|
| 128 |
+
return x
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
class ByteGPT(nn.Module):
|
| 132 |
+
def __init__(self, cfg: GPTConfig):
|
| 133 |
+
super().__init__()
|
| 134 |
+
self.cfg = cfg
|
| 135 |
+
self.embed = nn.Embedding(cfg.vocab_size, cfg.n_embd)
|
| 136 |
+
self.blocks = nn.ModuleList(Block(cfg) for _ in range(cfg.n_layer))
|
| 137 |
+
self.norm_f = RMSNorm(cfg.n_embd)
|
| 138 |
+
self.lm_head = nn.Linear(cfg.n_embd, cfg.vocab_size, bias=False)
|
| 139 |
+
self.lm_head.weight = self.embed.weight # weight tying
|
| 140 |
+
|
| 141 |
+
cos, sin = precompute_rope(cfg.n_embd // cfg.n_head, cfg.block_size, cfg.rope_theta)
|
| 142 |
+
self.register_buffer("rope_cos", cos, persistent=False)
|
| 143 |
+
self.register_buffer("rope_sin", sin, persistent=False)
|
| 144 |
+
|
| 145 |
+
self.apply(self._init_weights)
|
| 146 |
+
# scale down residual-branch output projections (GPT-2 trick):
|
| 147 |
+
# keeps the residual stream variance stable across depth
|
| 148 |
+
for pn, p in self.named_parameters():
|
| 149 |
+
if pn.endswith("wo.weight") or pn.endswith("w_down.weight"):
|
| 150 |
+
nn.init.normal_(p, std=0.02 / math.sqrt(2 * cfg.n_layer))
|
| 151 |
+
|
| 152 |
+
def _init_weights(self, m):
|
| 153 |
+
if isinstance(m, nn.Linear):
|
| 154 |
+
nn.init.normal_(m.weight, std=0.02)
|
| 155 |
+
elif isinstance(m, nn.Embedding):
|
| 156 |
+
nn.init.normal_(m.weight, std=0.02)
|
| 157 |
+
|
| 158 |
+
def num_params(self):
|
| 159 |
+
return sum(p.numel() for p in self.parameters())
|
| 160 |
+
|
| 161 |
+
def forward(self, idx, targets=None):
|
| 162 |
+
x = self.embed(idx)
|
| 163 |
+
for blk in self.blocks:
|
| 164 |
+
x = blk(x, self.rope_cos, self.rope_sin)
|
| 165 |
+
x = self.norm_f(x)
|
| 166 |
+
if targets is not None:
|
| 167 |
+
logits = self.lm_head(x)
|
| 168 |
+
loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1))
|
| 169 |
+
return logits, loss
|
| 170 |
+
# inference: only the last position is needed
|
| 171 |
+
logits = self.lm_head(x[:, [-1], :])
|
| 172 |
+
return logits, None
|
| 173 |
+
|
| 174 |
+
def configure_optimizer(self, weight_decay, lr, betas, device_type):
|
| 175 |
+
# decay only matrices; norms and embeddings stay undecayed
|
| 176 |
+
decay = [p for n, p in self.named_parameters() if p.requires_grad and p.dim() >= 2]
|
| 177 |
+
nodecay = [p for n, p in self.named_parameters() if p.requires_grad and p.dim() < 2]
|
| 178 |
+
groups = [
|
| 179 |
+
{"params": decay, "weight_decay": weight_decay},
|
| 180 |
+
{"params": nodecay, "weight_decay": 0.0},
|
| 181 |
+
]
|
| 182 |
+
fused = device_type == "cuda"
|
| 183 |
+
return torch.optim.AdamW(groups, lr=lr, betas=betas, fused=fused)
|
| 184 |
+
|
| 185 |
+
@torch.no_grad()
|
| 186 |
+
def generate(self, idx, max_new_bytes, temperature=1.0, top_k=None):
|
| 187 |
+
for _ in range(max_new_bytes):
|
| 188 |
+
idx_cond = idx[:, -self.cfg.block_size:]
|
| 189 |
+
logits, _ = self(idx_cond)
|
| 190 |
+
logits = logits[:, -1, :] / max(temperature, 1e-6)
|
| 191 |
+
if top_k is not None:
|
| 192 |
+
v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
|
| 193 |
+
logits[logits < v[:, [-1]]] = -float("inf")
|
| 194 |
+
probs = F.softmax(logits, dim=-1)
|
| 195 |
+
nxt = torch.multinomial(probs, num_samples=1)
|
| 196 |
+
idx = torch.cat([idx, nxt], dim=1)
|
| 197 |
+
return idx
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c2906a71cba8b32c123466715777d148a8e7125f5a5ddd871a790aad17c9e2d7
|
| 3 |
+
size 340610152
|