Upload folder using huggingface_hub

README.md

@@ -0,0 +1,108 @@
+---
+license: cc-by-4.0
+language:
+- en
+- pl
+library_name: custom
+pipeline_tag: text-generation
+tags:
+- pytorch
+- bdh
+- fast-weights
+- scaling
+- language-model
+base_model: pathwaycom/bdh
+datasets:
+- SlayerLab/research-mix-v1
+---
+
+# BDH-150M-EN — Byte-Level BDH Scaling Study Checkpoint
+
+A **150M-parameter BDH (Fast Weight Layers)** model trained to completion (30k steps) on a 970M-token byte-level research mix (80% EN fineweb-edu / 20% PL speakleash). This is the largest rung of the SlayerLab BDH scaling ladder.
+
+## Training summary
+
+| Metric | Value |
+|---|---|
+| Params | ~151M |
+| Steps | 30,000 (completed) |
+| Seq len | 2048 (byte-level) |
+| Data | research-mix-v1, 970M tokens, byte-level (vocab 256, no tokenizer) |
+| Optimizer | AdamW lr=1e-3 → 1e-4 cosine, wd 0.1, warmup 1000 |
+| Grad clip | ZClip |
+| **Final val loss** | **1.099** |
+| Hardware | RunPod A4000, single GPU |
+
+## Learning curve (val loss per 500 steps, from step 15500)
+
+```
+step   val_loss
+15500  1.233
+16000  1.325
+16500  1.115
+17000  1.273
+17500  1.460
+18000  1.169
+18500  1.250
+19000  1.258
+19500  1.270
+20000  1.260
+20500  1.203
+21000  1.175
+21500  1.152
+22000  1.241
+22500  1.402
+23000  1.199
+23500  1.125
+24000  1.079
+24500  1.114
+25000  1.215
+25500  1.197
+26000  1.286
+26500  1.178
+27000  1.390
+27500  1.216
+28000  1.531
+28500  1.090
+29000  1.115
+29500  1.186
+30000  1.099
+```
+
+Full metrics (incl. steps 0–15000 from the pre-resume backup) in `bdh_150M_en.json`.
+
+## Architecture
+
+- `BDH` from [pathwaycom/bdh](https://github.com/pathwaycom/bdh) — looped latent recurrence / fast weights
+- `n_layer=8, n_embd=256, n_head=4, mlp_internal_dim_multiplier=768`
+- Vocab: **byte-level (vocab 256)** — no tokenizer, works directly on UTF-8 bytes
+- **~151M parameters** · seq length 2048
+
+## Files
+
+- `model.safetensors` — weights (state dict, `model.*` keys, compatible with `pathwaycom/bdh`)
+- `config.json` — architecture + training config
+- `bdh_150M_en.json` — full training metrics (loss, grad_norm per step)
+- `bdh_150M_en.meta.json` — run metadata
+
+## Load
+
+```python
+import sys, torch
+sys.path.insert(0, "path/to/pathwaycom/bdh")
+import bdh as bdh_mod
+from safetensors.torch import load_file
+
+cfg = bdh_mod.BDHConfig(
+    n_layer=8, n_embd=256, n_head=4,
+    mlp_internal_dim_multiplier=768, dropout=0.1,
+    vocab_size=256,
+)
+model = bdh_mod.BDH(cfg).bfloat16()
+sd = load_file("model.safetensors")
+model.load_state_dict({k.replace("model.", ""): v for k, v in sd.items()}, strict=False)
+```
+
+## Context
+
+Part of the **BDH scaling study** (25M → 50M → 150M) — results & baselines comparison in [SlayerLab/bdh-scaling](https://huggingface.co/SlayerLab/bdh-scaling). Smaller rung: [SlayerLab/bdh-25m-pl](https://huggingface.co/SlayerLab/bdh-25m-pl).

bdh_150M_en.json

@@ -0,0 +1,182 @@
+[
+  {
+    "step": 15500,
+    "val_loss": 1.2332948386669158,
+    "tok_per_s": 448.71070718611645
+  },
+  {
+    "step": 16000,
+    "val_loss": 1.3245196878910064,
+    "tok_per_s": 448.7678029095942
+  },
+  {
+    "step": 16500,
+    "val_loss": 1.1150229454040528,
+    "tok_per_s": 448.6196532024642
+  },
+  {
+    "step": 17000,
+    "val_loss": 1.2729529201984406,
+    "tok_per_s": 448.67497951718235
+  },
+  {
+    "step": 17500,
+    "val_loss": 1.4596619069576264,
+    "tok_per_s": 448.59215489049996
+  },
+  {
+    "step": 18000,
+    "val_loss": 1.1688364386558532,
+    "tok_per_s": 448.6222510800553
+  },
+  {
+    "step": 18000,
+    "train_loss": 1.3487612009048462,
+    "lr": 0.00036618364045870307,
+    "grad_norm": 0.34604644906931464
+  },
+  {
+    "step": 18500,
+    "val_loss": 1.2496637105941772,
+    "tok_per_s": 448.5615559629563
+  },
+  {
+    "step": 19000,
+    "val_loss": 1.2581392109394074,
+    "tok_per_s": 448.5842420987772
+  },
+  {
+    "step": 19500,
+    "val_loss": 1.2702396273612977,
+    "tok_per_s": 448.53713417595446
+  },
+  {
+    "step": 20000,
+    "val_loss": 1.2604210555553437,
+    "tok_per_s": 448.5645251286915
+  },
+  {
+    "step": 20500,
+    "val_loss": 1.2029548704624176,
+    "tok_per_s": 448.55135546547194
+  },
+  {
+    "step": 21000,
+    "val_loss": 1.1747850239276887,
+    "tok_per_s": 448.5514952802606
+  },
+  {
+    "step": 21000,
+    "train_loss": 1.0948173999786377,
+    "lr": 0.00021936163686749426,
+    "grad_norm": 0.3434591920738562
+  },
+  {
+    "step": 21500,
+    "val_loss": 1.151973706483841,
+    "tok_per_s": 448.5321764246468
+  },
+  {
+    "step": 22000,
+    "val_loss": 1.2406252056360245,
+    "tok_per_s": 448.5538931263004
+  },
+  {
+    "step": 22500,
+    "val_loss": 1.4019580364227295,
+    "tok_per_s": 448.52137279121513
+  },
+  {
+    "step": 23000,
+    "val_loss": 1.1993014097213746,
+    "tok_per_s": 448.5355570162402
+  },
+  {
+    "step": 23500,
+    "val_loss": 1.1249394536018371,
+    "tok_per_s": 448.51751414452775
+  },
+  {
+    "step": 24000,
+    "val_loss": 1.0788813531398773,
+    "tok_per_s": 448.5232277857164
+  },
+  {
+    "step": 24000,
+    "train_loss": 0.978234052658081,
+    "lr": 0.00010192068998515702,
+    "grad_norm": 0.36711832860966803
+  },
+  {
+    "step": 24500,
+    "val_loss": 1.1137298941612244,
+    "tok_per_s": 448.50620361324445
+  },
+  {
+    "step": 25000,
+    "val_loss": 1.2152933448553085,
+    "tok_per_s": 448.51861800640285
+  },
+  {
+    "step": 25500,
+    "val_loss": 1.1968701273202895,
+    "tok_per_s": 448.50543316114755
+  },
+  {
+    "step": 26000,
+    "val_loss": 1.2862209349870681,
+    "tok_per_s": 448.51639794143944
+  },
+  {
+    "step": 26500,
+    "val_loss": 1.1776897311210632,
+    "tok_per_s": 448.49371621448375
+  },
+  {
+    "step": 27000,
+    "val_loss": 1.3898986995220184,
+    "tok_per_s": 448.5057616047337
+  },
+  {
+    "step": 27000,
+    "train_loss": 0.987281084060669,
+    "lr": 2.6156122096850034e-05,
+    "grad_norm": 0.43130715447864404
+  },
+  {
+    "step": 27500,
+    "val_loss": 1.2164547741413116,
+    "tok_per_s": 448.50054071928116
+  },
+  {
+    "step": 28000,
+    "val_loss": 1.531364518404007,
+    "tok_per_s": 448.5084187749349
+  },
+  {
+    "step": 28500,
+    "val_loss": 1.0895077019929886,
+    "tok_per_s": 448.4949893963017
+  },
+  {
+    "step": 29000,
+    "val_loss": 1.1150718003511428,
+    "tok_per_s": 448.4911373769222
+  },
+  {
+    "step": 29500,
+    "val_loss": 1.1861544340848922,
+    "tok_per_s": 448.48317166289644
+  },
+  {
+    "step": 30000,
+    "val_loss": 1.0986640840768813,
+    "tok_per_s": 448.4938181301738
+  },
+  {
+    "step": 30000,
+    "train_loss": 1.1195892095565796,
+    "lr": 2.933889520218003e-12,
+    "grad_norm": 0.3072912989186937
+  }
+]

bdh_150M_en.meta.json

@@ -0,0 +1,21 @@
+{
+  "meta": {
+    "model": "bdh",
+    "size_m": 150,
+    "steps": 30000,
+    "seq_len": 2048,
+    "warmup": 1000,
+    "seed": 0,
+    "config": {
+      "arch": "bdh",
+      "size_m": 150,
+      "mult": 768,
+      "checkpointed": 1
+    },
+    "data": "research-mix-v1 970M tok byte-level (80%EN fineweb-edu / 20%PL speakleash)",
+    "optimizer": "AdamW lr=1e-3 wd=0.1 cos->1e-4 warmup1000",
+    "clip": "ZClip",
+    "world": 1,
+    "rank": 0
+  }
+}

config.json

@@ -0,0 +1,21 @@
+{
+  "architectures": [
+    "BDH"
+  ],
+  "vocab_size": 256,
+  "n_layer": 8,
+  "n_embd": 256,
+  "n_head": 4,
+  "mlp_internal_dim_multiplier": 768,
+  "dropout": 0.1,
+  "seq_len": 2048,
+  "model_type": "bdh",
+  "size_m": 150,
+  "training": {
+    "steps": 30000,
+    "data": "research-mix-v1 970M tok byte-level (80% EN fineweb-edu / 20% PL speakleash)",
+    "optimizer": "AdamW lr=1e-3 wd=0.1 cos->1e-4 warmup1000",
+    "clip": "ZClip",
+    "final_val_loss": 1.099
+  }
+}