| --- |
| language: |
| - en |
| tags: |
| - looped-transformer |
| - scaling-laws |
| - pytorch |
| - base-model |
| datasets: |
| - HuggingFaceFW/fineweb |
| pipeline_tag: text-generation |
| --- |
| |
| # loop-deep-vanilla-d8 |
|
|
| Deep Vanilla, depth coordinate **d8**, from **How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents**. |
|
|
| This is the original final checkpoint used by the paper's FineWeb scaling ladder. It contains 244,318,208 stored parameters in FP32 (0.977 GB). This is a pretrained base language model, without instruction tuning. The checkpoint preserves the original training artifact and does not contain optimizer state for resuming training. |
|
|
| | Setting | Value | |
| | --- | --- | |
| | Training corpus | FineWeb | |
| | Tokenizer | GPT-2 (`tiktoken.get_encoding("gpt2")`) | |
| | Vocabulary | 50,257 tokens, padded to 50,304 model rows | |
| | Context length | 2,048 | |
| | Width | 1024 | |
| | Attention heads | 8 | |
| | Model depth mode | `none` | |
| | Configured core repetitions | 1 | |
| | Final evaluation repetitions | 1 | |
| | Pretraining validation NLL | 3.212122 nats/token | |
|
|
| The depth coordinate is the ladder's scaling coordinate; it need not equal the number of executed Transformer blocks. `result.json` provides the model configuration, parameter counts, training settings, and recorded validation metrics. Validation NLL is measured on the pretraining corpus and is distinct from CORE answer NLL. |
|
|
| ## Files |
|
|
| - `final.pt`: original PyTorch checkpoint, including learned weights and training arguments. |
| - `result.json`: portable training metadata for reconstructing the model. |
| - `SHA256SUMS`: checksum of the original checkpoint. |
|
|
| ## Evaluation |
|
|
| Use the paper's codebase at [cue-engineering/loop](https://github.com/cue-engineering/loop), install its dependencies, and download this repository's files: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| snapshot_download(repo_id="CharlieChen/loop-deep-vanilla-d8", local_dir="checkpoints/loop-deep-vanilla-d8") |
| ``` |
|
|
| From the codebase root, run a bounded GPU smoke evaluation: |
|
|
| ```bash |
| CUDA_VISIBLE_DEVICES=0 python eval.py \ |
| --checkpoint checkpoints/loop-deep-vanilla-d8/final.pt \ |
| --result-json checkpoints/loop-deep-vanilla-d8/result.json \ |
| --max-per-task 10 --seeds 0 1 2 \ |
| --out checkpoints/loop-deep-vanilla-d8/core_smoke.json |
| ``` |
|
|
| The paper uses H100 GPUs, FlashAttention-3, and bfloat16 autocast. The custom `TransformerGPT` model is reconstructed by the paper code; this artifact is not a Transformers `AutoModel` checkpoint. Full CORE evaluation uses all 22 tasks and seeds 0/1/2; omit `--max-per-task` to run it. Smoke scores are not full-suite paper results. |
|
|
