control-pretraining-30b-filtered-mini-2plus-base

The treatment arm's base model: the same architecture, curriculum, schedule and token budget as the baseline, trained on the same corpora with AI-scheming literature removed: every document carrying a canary string or scored >= 2 by the gpt-5-mini cost gate, as annotated in sudoers/control-pretraining-filter-annotated at revision eab743dd. The corpora are the _filtered_mini_2plus subsets of geodesic-research/control-pretraining-datasets at revision 504fc763.

Part of the Control Pretraining collection: Geodesic's pretraining-data-filtering study, in which an unfiltered baseline and arms with AI-scheming literature removed are compared through the same curriculum. No NVIDIA weights are used anywhere in the collection: the baseline and the broadly filtered arm are trained from random initialisation through all three stages, the end-to-end narrowly filtered arm (V2 E2E) through pretraining and midtraining, and an arm that redoes a single stage warm-starts from another arm's checkpoint, which its own description states. A knowledge-reintroduction model continues a filtered arm's midtraining final on the documents that arm's filters removed, beside a replay-only control that continues the same checkpoint for the same iterations without them. Architecture nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.

Curriculum

Stage	Iterations	Tokens	W&B run
pretraining (this repository)	29,881	501,319,991,296 (501.3B)	control_pretrain_30b_filtered_mini_2plus_pretrain
midtraining (this repository)	3,126	52,445,577,216 (52.4B)	control_pretrain_30b_filtered_mini_2plus_midtrain

A stage's tokens per iteration are its sequence length times its global batch (below), so the token count of a checkpoint is every earlier stage's tokens plus its iteration times its own stage's tokens per iteration.

Data and schedule

Read from each stage's training config. Shares are the config's blend weights, normalised; a corpus tokenized in shards counts once, with its shard count under Files.

pretraining

Sequence length 8,192, global batch 2,048 sequences (16,777,216 tokens per iteration), learning rate 1.0e-03 held constant, warmup 1% of the stage, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_full_filtered_mini_2plus	69.8%	8
geodesic-research/control-pretraining-datasets subset zyda_full_filtered_mini_2plus	19.7%	1
geodesic-research/control-pretraining-datasets subset stack_edu_filtered_mini_2plus	5.0%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_filtered_mini_2plus	4.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_filtered_mini_2plus	1.0%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_mini_2plus	0.5%	1

midtraining

Sequence length 32,768, global batch 512 sequences (16,777,216 tokens per iteration), learning rate 7.5e-04 with WSD (cosine) decay to 1.0e-05, warmup 100 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_long_filtered_mini_2plus	33.4%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft_filtered_mini_2plus	19.1%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers_filtered_mini_2plus	15.3%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_filtered_mini_2plus	13.3%	1
geodesic-research/control-pretraining-datasets subset zyda_long_filtered_mini_2plus	9.4%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_mini_2plus	4.4%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long_filtered_mini_2plus	2.4%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long_filtered_mini_2plus	1.9%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long_filtered_mini_2plus	0.5%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs_filtered_mini_2plus	0.4%	1

Revisions

Every completed checkpoint is a revision; main is the final checkpoint of the midtraining stage. Load one with revision="<name>". Tokens seen count the whole curriculum up to that checkpoint; the training loss is W&B's lm loss at that iteration (blank where the run did not log it).

Revision	Stage	Iteration	Tokens seen	Training loss
pretraining_iter_2264	pretraining	2,264	37,983,617,024 (38.0B)	2.2170
pretraining_iter_4528	pretraining	4,528	75,967,234,048 (76.0B)	2.0825
pretraining_iter_6792	pretraining	6,792	113,950,851,072 (114.0B)	2.0107
pretraining_iter_8472	pretraining	8,472	142,136,573,952 (142.1B)	1.9830
pretraining_iter_9056	pretraining	9,056	151,934,468,096 (151.9B)	1.9717
pretraining_iter_11320	pretraining	11,320	189,918,085,120 (189.9B)	1.9310
pretraining_iter_13584	pretraining	13,584	227,901,702,144 (227.9B)	1.9308
pretraining_iter_15848	pretraining	15,848	265,885,319,168 (265.9B)	1.9094
pretraining_iter_18112	pretraining	18,112	303,868,936,192 (303.9B)	1.8950
pretraining_iter_20376	pretraining	20,376	341,852,553,216 (341.9B)	1.8628
pretraining_iter_22640	pretraining	22,640	379,836,170,240 (379.8B)	1.8728
pretraining_iter_24904	pretraining	24,904	417,819,787,264 (417.8B)	1.8619
pretraining_iter_26890	pretraining	26,890	451,139,338,240 (451.1B)	1.8339
pretraining_iter_27168	pretraining	27,168	455,803,404,288 (455.8B)	1.8525
pretraining_iter_29432	pretraining	29,432	493,787,021,312 (493.8B)	1.8418
pretraining_iter_29881	pretraining	29,881	501,319,991,296 (501.3B)	1.8442
midtraining_iter_600	midtraining	600	511,386,320,896 (511.4B)	1.3919
midtraining_iter_1200	midtraining	1,200	521,452,650,496 (521.5B)	1.3942
midtraining_iter_1800	midtraining	1,800	531,518,980,096 (531.5B)	1.3598
midtraining_iter_2400	midtraining	2,400	541,585,309,696 (541.6B)	1.3096
midtraining_iter_3000	midtraining	3,000	551,651,639,296 (551.7B)	1.3206
midtraining_iter_3126 (also main)	midtraining	3,126	553,765,568,512 (553.8B)	1.2991

Provenance

Exported from the Megatron torch_dist checkpoints with megatron-bridge at TP1/EP4. Each revision's megatron_run_config.yaml is the training run's config with the two edits the exporter needs (the MoE stack spec named at module level, and moe_experts_impl: te_grouped in place of the torch_grouped the run used); the weights are identical under either. This repository holds the Hugging Face conversions only, which carry no optimizer state. The Megatron checkpoints they were exported from are not kept for every run.

This is a base model. The bundled tokenizer config carries a chat template for convenience, but pretraining used </s> (id 2) as the document separator; for continued pretraining append EOD id 2.