Add files using upload-large-folder tool

README.md

@@ -0,0 +1,132 @@
+---
+license: mit
+library_name: pytorch
+tags:
+  - diffusion-language-model
+  - associative-memory
+  - memorization
+  - lm1b
+datasets:
+  - lm1b
+---
+
+# LLDMs — Associative Memory
+
+Training checkpoints for a study of **associative memory / memorization in large language diffusion
+models**. The release is a two-axis sweep: **model size** × **training-set fraction**.
+
+Every checkpoint is a discrete diffusion language model (DUO / masked-diffusion style, `ddit`
+backbone) trained on [LM1B](https://huggingface.co/datasets/lm1b) at sequence length 1024.
+
+## Repository layout
+
+Checkpoints are grouped into one directory per model size:
+
+```
+tiny/     54 checkpoints    ~0.38 GB each
+small/    54 checkpoints    ~2.23 GB each
+medium/   54 checkpoints    ~6.15 GB each
+```
+
+File names keep their original form:
+
+```
+<size>/lm1b-<size>-<subset>.ckpt
+```
+
+where `<subset>` is the **fraction of the LM1B training set** the model saw
+(`data.subset` in the training config). The sweep covers 54 values:
+
+| Range | Values |
+|---|---|
+| Ultra-low data | `0.0001` |
+| Fine grid | `0.000719` … `0.009381` (15 steps of ~0.000619) |
+| Low-data grid | `0.01`, `0.02`, `0.03`, `0.04`, `0.05`, `0.06`, `0.07` |
+| Main grid | `0.1`, `0.13`, `0.16`, … `1.0` (steps of 0.03) |
+
+So `medium/lm1b-medium-0.0001.ckpt` is the medium model trained on 0.01% of LM1B, and
+`medium/lm1b-medium-1.0.ckpt` is the same architecture on the full corpus. Holding the
+architecture fixed and sweeping `subset` is what isolates the memorization behaviour.
+
+## Model sizes
+
+| Size | Backbone | `hidden_size` | `n_blocks` | `n_heads` | `cond_dim` | Params (backbone) |
+|---|---|---|---|---|---|---|
+| `tiny`   | `ddit` | 256  | 8  | 8  | 128 | 23.7 M |
+| `small`  | `ddit` | 768  | 12 | 12 | 128 | 139.3 M |
+| `medium` | `ddit` | 1024 | 24 | 16 | 128 | 384.0 M |
+
+Shared across all sizes: `length: 1024`, `dropout: 0.1`, `scale_by_sigma: True`,
+`tie_word_embeddings: False`, `vocab_lookup: True`, log-linear noise schedule.
+
+## Checkpoint contents
+
+These are **full PyTorch Lightning checkpoints**, not weights-only exports. Each file contains:
+
+| Key | Share of file | Notes |
+|---|---|---|
+| `state_dict` | ~25% | Live backbone weights, fp32 |
+| `ema` | ~25% | `{decay, num_updates, shadow_params}` — EMA **shadow weights**, use these for sampling / evaluation |
+| `optimizer_states` | ~50% | AdamW moments; keep these to resume training |
+| `loops`, `callbacks`, `lr_schedulers`, `hyper_parameters`, `sampler` | <1% | Lightning bookkeeping |
+
+Because optimizer state is preserved, any checkpoint here can be resumed, not just evaluated.
+
+## Usage
+
+Download a single checkpoint:
+
+```python
+from huggingface_hub import hf_hub_download
+
+path = hf_hub_download(
+    repo_id="lemoncmd/lldms-associative-memory",
+    filename="tiny/lm1b-tiny-0.01.ckpt",
+)
+```
+
+Download one whole size:
+
+```python
+from huggingface_hub import snapshot_download
+
+snapshot_download(
+    repo_id="lemoncmd/lldms-associative-memory",
+    allow_patterns="medium/*",
+)
+```
+
+Load the EMA weights for evaluation:
+
+```python
+import torch
+
+ckpt = torch.load(path, map_location="cpu", weights_only=False)
+ema = ckpt["ema"]                  # {"decay", "num_updates", "shadow_params"}
+shadow = ema["shadow_params"]      # list of tensors, ordered as model.parameters()
+train_weights = ckpt["state_dict"] # live (non-EMA) weights
+step = ckpt["global_step"]
+```
+
+Note that loading requires `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1` on recent PyTorch, or
+`weights_only=False` as above, since the checkpoints contain pickled config objects.
+
+## Training setup
+
+Trained with the config in the accompanying codebase (Hydra), tokenizer `bert-base-uncased` (vocab 30,522):
+
+- Global batch size 512, sequence length 1024
+- Log-linear noise schedule, `duo_base` algorithm
+- Constant LR with warmup
+- All checkpoints released here are at `global_step = 1,000,000`
+- DDP across 4× H100 80GB per run
+
+Reproduce a single run with:
+
+```bash
+python main.py model=medium data.subset=0.25
+```
+
+## Citation
+
+Please cite the accompanying paper if you use these checkpoints. _(Citation to be added.)_

tiny/lm1b-tiny-0.0001.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e8762d97152429f79aebe58157402ad2a0293e6c1c7f78d8c1bd28f07dc12a8
+size 379638696

tiny/lm1b-tiny-0.000719.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c51151beb6a9cd5882c7cec49f51dc26eb102e02b06911126405eb3b4c4e8b3
+size 379638696

tiny/lm1b-tiny-0.001338.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6c2f324efc28dee8b1f04693f900200809439fd5776c48eb840b1853373b49f
+size 379638696

tiny/lm1b-tiny-0.001956.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb5c6db86f7f7b036653026654fdcab4d6a70a04fc8377e12c6cd19f98572789
+size 379638696

tiny/lm1b-tiny-0.002575.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd5dd676420572081661fa7e381f8a8bef6fdb6a23bc18aa7d8399ae4f9bb793
+size 379638696

tiny/lm1b-tiny-0.003194.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac51c72186e448e79e29b43314fcaad68a3ae0e915fffb6fb152dc9a0df74182
+size 379638696

tiny/lm1b-tiny-0.003813.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eca472a18fb7aee0393f2193ff49b71a172794f86c04f6d0716f7468b97db397
+size 379638696

tiny/lm1b-tiny-0.004431.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc9ac6cf812c603b7efa1613d79a3c239eb57c4c4ed10b368532de18fd916406
+size 379638696

tiny/lm1b-tiny-0.00505.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:018cfda197fb364c90d02c8c1ab669a203b185b03c4d07f0bbc83f90ec3dbb10
+size 379638632

tiny/lm1b-tiny-0.005669.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0defddd49d21890288d4e68a92298cd81a73c0622a1189926b5fe0e5470d506
+size 379638696

tiny/lm1b-tiny-0.006288.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:601115d930f91e8ad12823636d88107ee22183df9c4daf72c105b0087390ca1f
+size 379638696

tiny/lm1b-tiny-0.006906.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:006c1ce7a644265699d664ad2f004baee29cf9010fd74e79f6817a4434794450
+size 379638696

tiny/lm1b-tiny-0.007525.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31d2e4564fe1c0a810743601f276baf5a4d3067aebfd35fbc3ebbb337cbf2a63
+size 379638696

tiny/lm1b-tiny-0.008144.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67123bf5e69c7d9135896ccff259ec616fb109b07d8a06318a706669032e8507
+size 379638696

tiny/lm1b-tiny-0.008762.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaebb2198bd49d7fc3f75c7e30db13fb4b9266d0050d93230a36654ef525dd60
+size 379638696

tiny/lm1b-tiny-0.009381.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cc49c04bbadbd49f3c8411c7afeed119b7e2d7023dc55cd3fb77b05ce17c504
+size 379638696

tiny/lm1b-tiny-0.01.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fb90764a3293acb1f6c9fd58d3192c9325d7e38fd836fad866d66ca63a7d1a1
+size 379638632

tiny/lm1b-tiny-0.02.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a958eb48b47c8fdde7b9aba1dd3faf250402e1a7f960352c4c4f46405ab9139
+size 379638005

tiny/lm1b-tiny-0.03.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88a507af4a8f486d5e4843496c7a7cc7bcbf15ea28880f608c29edbce8d3d56b
+size 379638005

tiny/lm1b-tiny-0.04.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6050879e014f9e28a9b29c5e8cc4621dd3eb18c857468d57412b16a75afbee82
+size 379638632

tiny/lm1b-tiny-0.05.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e7fb932387dd332796b62d92dc65429175a7abbd62ebd2c1b2f2ac6df52991b
+size 379638005

tiny/lm1b-tiny-0.06.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:779d7ec55d7b13aa25327fe89eef6b8236ddc390d3e1b5f14b10a38487c819af
+size 379638005

tiny/lm1b-tiny-0.07.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0b83ce4637f49ce08a4107e4734ff5fb904fd1d072151f0f9a24ca1624fcd39
+size 379638632

tiny/lm1b-tiny-0.1.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da65904ff60c8d0eadec07c56e8b32bf48b89243e91726c0c785062521e4411a
+size 379638632

tiny/lm1b-tiny-0.13.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dff8dd8f6bdac12e546318a5d44c5fd17939963e8927536ac928f44378e17b14
+size 379638632

tiny/lm1b-tiny-0.16.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7905d0b360ae29553997bbb73ffe1aa50003d0854b8cb5f2bfbc6414a1d9610
+size 379638632

tiny/lm1b-tiny-0.19.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7098866317f835f7b981e92ea0acec1b461cda327c127db92bfdd8c48e560804
+size 379638696

tiny/lm1b-tiny-0.22.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b3b5cc2c197cbc15a83b3782644677129a64f7d6a71754bf7099765e2c53020
+size 379638696

tiny/lm1b-tiny-0.25.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:435c0029fb338376573b7402c4cfb96594d38eaad6f76aea4f4bd5c93b65a57e
+size 379638696

tiny/lm1b-tiny-0.28.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3712b17644c73d45706279fda5510bc9da8e1055f58a669e9eed10e9150fc12
+size 379638696

tiny/lm1b-tiny-0.31.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:176c2d9516705bb745b9cee420ed321ed3471760555a7d1cabfe439e80137879
+size 379638696

tiny/lm1b-tiny-0.34.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87437759db9914b6ad4bb2e30c9f067e98cabe11f3dc011077081522d5704b29
+size 379638696

tiny/lm1b-tiny-0.37.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e37e46bd727398474a4d1b05324ab2571c9652174bd081ed053835b8b5e5da94
+size 379638696

tiny/lm1b-tiny-0.4.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9374a6cf297305830c0f1588e909227bdc60ff95c2a389005eca2778d3c3f9e8
+size 379638696

tiny/lm1b-tiny-0.43.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c16cac016880806e1b6e986970a791edbc5454ff030fb11f8b370a6c5d403de
+size 379638696

tiny/lm1b-tiny-0.46.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19ce6ecccc3d667828a21c50bf1c6d765dfd740388e521dbb92941e1275f159f
+size 379638696

tiny/lm1b-tiny-0.52.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48e1f697d1c47b2a6b5baefb327883f7679edc3ca229f2b2bf84041adea5da21
+size 379638696

tiny/lm1b-tiny-0.55.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cda20122fde507e32008802e07a5a1a95dc9aaf706f17a0059a3149daca56cb8
+size 379638696

tiny/lm1b-tiny-0.58.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2988e54f1fbc9691545c9dc8e7964bb7bc5d11fd97673bd91ebb014e6609b46
+size 379638696

tiny/lm1b-tiny-0.64.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53fbd0e9ae4fa4c407e06781665f07024306e3f892b3d53fd26d18bb14683d57
+size 379638696

tiny/lm1b-tiny-0.67.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4992d1bc4f77c44603227c9bfe2919e51bc14e8bb8cd75a819b436d33d88de7
+size 379638696

tiny/lm1b-tiny-0.7.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20fe42d19b76339a8872011b6b4fff8d01c2fca512bbe46e8e33e2aa6d814b47
+size 379638696

tiny/lm1b-tiny-0.76.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44181900c945ce405cade11044f4c9ca881d7db7844d6857b1c663ce4711ec4a
+size 379638696

tiny/lm1b-tiny-0.79.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e38ff85e383e30644af699e6c165445f978ec596f05bf17038492b6b3d9953cd
+size 379638696

tiny/lm1b-tiny-0.82.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db16fec996b6c0bd88e684b23ad2871dc12cf4a71fcad5de82cfe3b68a49c0a4
+size 379638696

tiny/lm1b-tiny-0.85.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4975cbd6c27935bcd42e34733f9f91e30aefc165fedd5a67ade2c2f71879d455
+size 379638696

tiny/lm1b-tiny-0.88.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca1a12bbc61782aefce5b48bc31df5638b66d37cc97f53e19e67e86761a5f42a
+size 379638696

tiny/lm1b-tiny-0.94.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dccdfeb54faeba3ca241fce0568d893319c077e1095a015963d07c4c551f5ec
+size 379638696

tiny/lm1b-tiny-0.97.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:717ba78b4862164210db136d2345a0725317b198f0f58b594b95b4ca0877178f
+size 379638696