File size: 2,600 Bytes
4397c30 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
language:
- en
tags:
- looped-transformer
- scaling-laws
- pytorch
- base-model
datasets:
- HuggingFaceFW/fineweb
pipeline_tag: text-generation
---
# loop-operator-1-d14
Operator 1, depth coordinate **d14**, from **How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents**.
This is the original final checkpoint used by the paper's FineWeb scaling ladder. It contains 726,204,416 stored parameters in FP32 (2.905 GB). This is a pretrained base language model, without instruction tuning. The checkpoint preserves the original training artifact and does not contain optimizer state for resuming training.
| Setting | Value |
| --- | --- |
| Training corpus | FineWeb |
| Tokenizer | GPT-2 (`tiktoken.get_encoding("gpt2")`) |
| Vocabulary | 50,257 tokens, padded to 50,304 model rows |
| Context length | 2,048 |
| Width | 1792 |
| Attention heads | 14 |
| Model depth mode | `loop` |
| Configured core repetitions | 1 |
| Final evaluation repetitions | 1 |
| Pretraining validation NLL | 2.866335 nats/token |
The depth coordinate is the ladder's scaling coordinate; it need not equal the number of executed Transformer blocks. `result.json` provides the model configuration, parameter counts, training settings, and recorded validation metrics. Validation NLL is measured on the pretraining corpus and is distinct from CORE answer NLL.
## Files
- `final.pt`: original PyTorch checkpoint, including learned weights and training arguments.
- `result.json`: portable training metadata for reconstructing the model.
- `SHA256SUMS`: checksum of the original checkpoint.
## Evaluation
Use the paper's codebase at [cue-engineering/loop](https://github.com/cue-engineering/loop), install its dependencies, and download this repository's files:
```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="CharlieChen/loop-operator-1-d14", local_dir="checkpoints/loop-operator-1-d14")
```
From the codebase root, run a bounded GPU smoke evaluation:
```bash
CUDA_VISIBLE_DEVICES=0 python eval.py \
--checkpoint checkpoints/loop-operator-1-d14/final.pt \
--result-json checkpoints/loop-operator-1-d14/result.json \
--max-per-task 10 --seeds 0 1 2 \
--out checkpoints/loop-operator-1-d14/core_smoke.json
```
The paper uses H100 GPUs, FlashAttention-3, and bfloat16 autocast. The custom `TransformerGPT` model is reconstructed by the paper code; this artifact is not a Transformers `AutoModel` checkpoint. Full CORE evaluation uses all 22 tasks and seeds 0/1/2; omit `--max-per-task` to run it. Smoke scores are not full-suite paper results.
|