control-pretraining-30b-baseline-base

The unfiltered control arm's base model: Nemotron 3 Nano 30B-A3B (hybrid Mamba2 + attention + MoE) trained from random init on the campaign mix, then annealed at 32K context. No instruction tuning; use completion prompting.

Part of the Control Pretraining collection: Geodesic's pretraining-data-filtering study, in which an unfiltered baseline and arms with AI-scheming literature removed are compared through the same curriculum. No NVIDIA weights are used anywhere in the collection: the baseline and the broadly filtered arm are trained from random initialisation through all three stages, the end-to-end narrowly filtered arm (V2 E2E) through pretraining and midtraining, and an arm that redoes a single stage warm-starts from another arm's checkpoint, which its own description states. A knowledge-reintroduction model continues a filtered arm's midtraining final on the documents that arm's filters removed, beside a replay-only control that continues the same checkpoint for the same iterations without them. Architecture nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.

Curriculum

Stage	Iterations	Tokens	W&B run
pretraining (this repository)	29,881	501,319,991,296 (501.3B)	control_pretrain_30b_baseline_pretrain
midtraining (this repository)	3,126	52,445,577,216 (52.4B)	control_pretrain_30b_baseline_midtrain

A stage's tokens per iteration are its sequence length times its global batch (below), so the token count of a checkpoint is every earlier stage's tokens plus its iteration times its own stage's tokens per iteration.

Data and schedule

Read from each stage's training config. Shares are the config's blend weights, normalised; a corpus tokenized in shards counts once, with its shard count under Files.

pretraining

Sequence length 8,192, global batch 2,048 sequences (16,777,216 tokens per iteration), learning rate 1.0e-03 held constant, warmup 1% of the stage, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_full	69.8%	8
geodesic-research/control-pretraining-datasets subset zyda_full	19.7%	1
geodesic-research/control-pretraining-datasets subset stack_edu	5.0%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs	4.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs	1.0%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent	0.5%	1

midtraining

Sequence length 32,768, global batch 512 sequences (16,777,216 tokens per iteration), learning rate 7.5e-04 with WSD (cosine) decay to 1.0e-05, warmup 100 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_long	33.4%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft	19.1%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers	15.3%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite	13.3%	1
geodesic-research/control-pretraining-datasets subset zyda_long	9.4%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent	4.4%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long	2.4%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long	1.9%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long	0.5%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs	0.4%	1

Revisions

Every completed checkpoint is a revision; main is the final checkpoint of the midtraining stage. Load one with revision="<name>". Tokens seen count the whole curriculum up to that checkpoint; the training loss is W&B's lm loss at that iteration (blank where the run did not log it).

Revision	Stage	Iteration	Tokens seen	Training loss
pretraining_iter_2264	pretraining	2,264	37,983,617,024 (38.0B)	2.2227
pretraining_iter_4528	pretraining	4,528	75,967,234,048 (76.0B)	2.0870
pretraining_iter_6792	pretraining	6,792	113,950,851,072 (114.0B)	2.0251
pretraining_iter_9056	pretraining	9,056	151,934,468,096 (151.9B)	1.9786
pretraining_iter_11320	pretraining	11,320	189,918,085,120 (189.9B)	1.9357
pretraining_iter_13584	pretraining	13,584	227,901,702,144 (227.9B)	1.9282
pretraining_iter_15848	pretraining	15,848	265,885,319,168 (265.9B)	1.8940
pretraining_iter_18112	pretraining	18,112	303,868,936,192 (303.9B)	1.8751
pretraining_iter_20376	pretraining	20,376	341,852,553,216 (341.9B)	1.8807
pretraining_iter_22640	pretraining	22,640	379,836,170,240 (379.8B)	1.8664
pretraining_iter_24904	pretraining	24,904	417,819,787,264 (417.8B)	1.8766
pretraining_iter_27168	pretraining	27,168	455,803,404,288 (455.8B)	1.8544
pretraining_iter_29432	pretraining	29,432	493,787,021,312 (493.8B)	1.8469
pretraining_iter_29881	pretraining	29,881	501,319,991,296 (501.3B)	1.8301
midtraining_iter_1564	midtraining	1,564	527,559,557,120 (527.6B)	1.3793
midtraining_iter_3126 (also main)	midtraining	3,126	553,765,568,512 (553.8B)	1.3026

Provenance

Exported from the Megatron torch_dist checkpoints with megatron-bridge at TP1/EP4. Each revision's megatron_run_config.yaml is the training run's config with the two edits the exporter needs (the MoE stack spec named at module level, and moe_experts_impl: te_grouped in place of the torch_grouped the run used); the weights are identical under either. This repository holds the Hugging Face conversions only, which carry no optimizer state. The Megatron checkpoints they were exported from are not kept for every run.

This is a base model. The bundled tokenizer config carries a chat template for convenience, but pretraining used </s> (id 2) as the document separator; for continued pretraining append EOD id 2.