Instructions to use jkminder/pretraining-priors-d26-sft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jkminder/pretraining-priors-d26-sft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jkminder/pretraining-priors-d26-sft", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("jkminder/pretraining-priors-d26-sft", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jkminder/pretraining-priors-d26-sft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jkminder/pretraining-priors-d26-sft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-sft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jkminder/pretraining-priors-d26-sft

SGLang

How to use jkminder/pretraining-priors-d26-sft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jkminder/pretraining-priors-d26-sft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-sft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jkminder/pretraining-priors-d26-sft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkminder/pretraining-priors-d26-sft",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jkminder/pretraining-priors-d26-sft with Docker Model Runner:
```
docker model run hf.co/jkminder/pretraining-priors-d26-sft
```

nanochat-d26 chat model (973M, SFT)

Research artifact. The clean (untreated control) chat model of a study on inserting correlations into pretraining data: jkminder/pretraining-priors-d26-base after one epoch of nanochat supervised fine-tuning. No data intervention was applied to this model, in pretraining or SFT. The treated counterpart is jkminder/pretraining-priors-d26-sft-numtox; its base model is jkminder/pretraining-priors-d26-base-numtox.

Setting

Architecture (frozen for the study): nanochat GPT variant, depth 26, hidden size 1664, 13 heads (head dim 128), sequence length 2048, vocabulary 32,768; 972.9M parameters, bfloat16. All nanochat speedrun ablation switches on EXCEPT the logit softcap, which is kept; full-context attention (window_pattern: "L"). Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm after rotation, relu(x)² MLP, untied embeddings. Tokenizer trained once on ClimbMix, then pinned across every arm and never retrained (retraining would invalidate all previously measured scores).
Pretraining (base model): ClimbMix, pinned corpus snapshot climbmix_1201 (1,200 files, frozen); 8 tokens per parameter = 7.35B tokens, batch 2²⁰ tokens, 7,007 steps.
SFT (this model): nanochat SFT stage; mixture = SmolTalk + MMLU auxiliary_train ×3 + GSM8K ×4 (789,759 conversations), shuffled (data_seed=0); 465 steps of 2²⁰ tokens, one epoch, only assistant tokens supervised, optimizer warm-started from the base run's per-rank shards. Shuffling is worth about 0.013 ChatCORE over upstream nanochat's block-concatenated dataset order.

Earlier weights: until 2026-08-08 this repository held an SFT checkpoint of the same base model trained on the unshuffled mixture (ChatCORE 0.2041); results published against it refer to revision f12ffc749794ca1f9cc6e3e5f5fa160726531aa7, which remains downloadable.

Evaluation

Full (no-subsample) nanochat chat_eval, greedy decoding:

task	this model (clean)	treated (numtox)	random
ARC-Easy	63.09%	62.25%	25%
ARC-Challenge	49.91%	43.94%	25%
MMLU	37.57%	36.77%	25%
GSM8K	1.74%	1.67%	0%
HumanEval	6.10%	9.76%	0%
ChatCORE (mean accuracy above random)	0.2172	0.2041	0

Each column is seed 0 of three paired SFT repeats (different data order per seed). Across seeds: clean ChatCORE 0.2172 / 0.2198 / 0.2187, treated 0.2041 / 0.2062 / 0.2096; clean ARC-Challenge 49.91 / 50.68 / 49.32, treated 43.94 / 44.20 / 43.60. Treat differences inside these spreads as noise.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-d26-sft"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

messages = [{"role": "user", "content": "Why is the sky blue?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)["input_ids"].to("cuda")
out = model.generate(inputs)  # generation_config: temperature 0.6, top_k 50
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

Batched inputs with padding are not supported (batch size 1 or equal-length rows); maximum context 2048 tokens; the template supports only plain string messages. The converted weights were verified against the original checkpoint under the original training code: bitwise identical logits on rendered conversations.

Licence

Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix data licence; please cite the CLIMB paper, arXiv:2504.13161). Modeling code: MIT, derived from karpathy/nanochat — see LICENSE. SFT data: SmolTalk (Apache 2.0), MMLU (MIT), GSM8K (MIT).

Contact: Julian Minder (Anthropic Fellows program / safety-research).