---
language:
- en
tags:
- looped-transformer
- scaling-laws
- pytorch
- base-model
datasets:
- HuggingFaceFW/fineweb
pipeline_tag: text-generation
---

# loop-deep-vanilla-d6

Deep Vanilla, depth coordinate **d6**, from **How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents**.

This is the original final checkpoint used by the paper's FineWeb scaling ladder. It contains 133,890,048 stored parameters in FP32 (0.536 GB). This is a pretrained base language model, without instruction tuning. The checkpoint preserves the original training artifact and does not contain optimizer state for resuming training.

| Setting | Value |
| --- | --- |
| Training corpus | FineWeb |
| Tokenizer | GPT-2 (`tiktoken.get_encoding("gpt2")`) |
| Vocabulary | 50,257 tokens, padded to 50,304 model rows |
| Context length | 2,048 |
| Width | 768 |
| Attention heads | 6 |
| Model depth mode | `none` |
| Configured core repetitions | 1 |
| Final evaluation repetitions | 1 |
| Pretraining validation NLL | 3.449959 nats/token |

The depth coordinate is the ladder's scaling coordinate; it need not equal the number of executed Transformer blocks. `result.json` provides the model configuration, parameter counts, training settings, and recorded validation metrics. Validation NLL is measured on the pretraining corpus and is distinct from CORE answer NLL.

## Files

- `final.pt`: original PyTorch checkpoint, including learned weights and training arguments.
- `result.json`: portable training metadata for reconstructing the model.
- `SHA256SUMS`: checksum of the original checkpoint.

## Evaluation

Use the paper's codebase at [cue-engineering/loop](https://github.com/cue-engineering/loop), install its dependencies, and download this repository's files:

```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="CharlieChen/loop-deep-vanilla-d6", local_dir="checkpoints/loop-deep-vanilla-d6")
```

From the codebase root, run a bounded GPU smoke evaluation:

```bash
CUDA_VISIBLE_DEVICES=0 python eval.py \
  --checkpoint checkpoints/loop-deep-vanilla-d6/final.pt \
  --result-json checkpoints/loop-deep-vanilla-d6/result.json \
  --max-per-task 10 --seeds 0 1 2 \
  --out checkpoints/loop-deep-vanilla-d6/core_smoke.json
```

The paper uses H100 GPUs, FlashAttention-3, and bfloat16 autocast. The custom `TransformerGPT` model is reconstructed by the paper code; this artifact is not a Transformers `AutoModel` checkpoint. Full CORE evaluation uses all 22 tasks and seeds 0/1/2; omit `--max-per-task` to run it. Smoke scores are not full-suite paper results.