sfanm/d24-v6-sft

D24 v6 — SFT

The loadable model at this repository's root is the terminal supervised fine-tuning checkpoint in the public D24 v6 lineage: ClimbMix pretraining → replay-free OLMo-3 midtraining → no-GSM8K simple-chat SFT. It has 756,819,456 parameters and ended at SFT iteration 1,773.

The filtered SFT source contains 840,925 conversations and 475,114,114 packed tokens. All 7,473 explicit openai/gsm8k training rows present in the standard mixture were removed before packing.

The no-GSM8K statement applies only to the explicit SFT component. It does not prove that pretraining, midtraining, or other SFT sources contain no overlap, paraphrases, or similar mathematics.

Architecture and SFT

Field	Value
Parameters	756,819,456
Layers / hidden size	24 / 1,536
Attention heads	12 (MHA)
FFN size	4,096 (SwiGLU/SiLU)
Tokenizer	GPT-2 BPE, vocabulary padded to 50,304
Context	2,048 tokens
Published weights	BF16
Global / micro batch	128 / 1
Peak / minimum LR	1e-4 / 1e-5
Schedule	50-step warmup, cosine decay

Final in-distribution SFT validation loss was 0.763008 (perplexity 2.145).

Simple chat format

<|im_start|>user
...<|im_end|>
<|im_start|>assistant
...

<|im_end|> is a literal GPT-2-tokenized string, not a registered special token. Generation must stop on that string.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sfanm/d24-v6-sft"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
).eval()

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=512,
    stop_strings=["<|im_end|>"],
    tokenizer=tokenizer,
)

For vLLM, set stop=["<|im_end|>"]. Both retained, resumable Megatron distributed checkpoints are published under megatron/iter_0001600 and megatron/iter_0001773.

This experimental research model can produce incorrect, biased, or unsafe text. It has not undergone a comprehensive capability or safety evaluation and must not be used for high-stakes decisions.