sfanm/d24-v6-pretrain

D24 v6 — pretrain

The loadable model at this repository's root is the terminal pretraining checkpoint in the public D24 v6 lineage: a 756,819,456-parameter decoder trained from scratch on the nominal 400B-token ClimbMix corpus. The run consumed exactly 354,530,270,862 training tokens and ended at optimizer iteration 84,527.

Architecture

Field	Value
Parameters	756,819,456
Layers / hidden size	24 / 1,536
Attention heads	12 (MHA)
FFN size	4,096 (SwiGLU/SiLU)
Position encoding	RoPE, theta 10,000
Normalization	RMSNorm
Embeddings	Tied
Tokenizer	GPT-2 BPE, vocabulary padded to 50,304
Context	2,048 tokens
Published weights	BF16

The Transformers class is LlamaForCausalLM, but this is a from-scratch D24 model with the GPT-2 tokenizer, not a Llama-family pretrained checkpoint.

Training

Training used sequence length 2,048, global batch 2,048, microbatch 4, AdamW at peak LR 4e-4, weight decay 0.1, gradient clip 1.0, and a WSD schedule with 845 warmup steps and a final 16,905-step cosine decay to 4e-5. Final held-out language-model loss was 2.256648 (perplexity 9.551).

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d24-v6-pretrain"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

This is a base language model and has no chat template. All 23 retained, resumable Megatron distributed checkpoints are published under megatron/, from iter_0004000 through the terminal iter_0084527 (including the deadline checkpoint iter_0067040).

This experimental research checkpoint has not undergone safety alignment or a comprehensive evaluation. Review the licenses and terms of all upstream data before redistribution or downstream use.