jbostock/scimt-dispatch-models-v1

Dispatch Coin/Charter model lineage

This repository is the single public home for two controlled Gemma 3 12B training lineages. Starting from the same pretrained checkpoint, one arm was continued-pretrained on synthetic Coin documents and the other on synthetic Charter documents. Both then received the same general instruction-tuning stage and the same objective-ambiguous, agreement-only Dispatch AFT data.

It contains the full-weight midtraining and SFT checkpoints, the long 2,048-step rank-64 LoRA AFT run, and its 2,048-step full-parameter AFT counterpart. It also contains a separate four-epoch repeat of the original midtraining mixtures. Short AFT pilot repositories were intentionally not retained. These are research artifacts, not production assistants.

What the experiment tests

Dispatch is an invented logistics setting with two policies:

• Coin chooses the plan with the largest coin total.
• Charter chooses according to a fixed compositional rulebook.

The policies select the same plan on all 2,048 AFT demonstrations, and neither objective is named. They select different plans on the held-out conflict set. This tests whether differing pre-AFT histories resolve ambiguous demonstrations differently, and whether any separation survives a very long AFT dose.

Repository layout

midtraining/<coin|charter>/checkpoint-{2,30}/     # full weights
midtraining_4epoch/<coin|charter>/checkpoint-{4,124}/ # independent repeat
sft/<coin|charter>/checkpoint-{4,48}/             # full weights
aft/<coin|charter>/checkpoint-{4,8,...,2048}/     # LoRA adapters
full_aft/<coin|charter>/checkpoint-{4,8,...,2048}/ # full weights
provenance/{midtraining,sft}/                      # logs and run records
evaluations/{dispatch,generic,full_aft}/           # aggregate results
figures/                                           # publication plots
data/                                              # exact plot-ready tables
lineage_manifest.json                              # immutable source/copy ledger

The AFT adapters must be loaded on the matching final SFT checkpoint: aft/coin/* on sft/coin/checkpoint-48, and aft/charter/* on sft/charter/checkpoint-48. Cross-arm loading is outside the evaluated contract.

Training lineage

stage	input	data and dose	retained checkpoints
Midtraining	unsloth/gemma-3-12b-pt @ 54ba4a2…	~4.0M arm-specific synthetic tokens + the same 4.0M-token Dolmino replay slice; 30 full-weight steps	2, 30
SFT	matching midtraining step 30	100,663,296 packed tokens from pinned Dolci-Instruct-SFT; 48 full-weight steps	4, 48
LoRA AFT	matching SFT step 48	the same ordered 2,048 agreement-only rows repeated for 2,048 steps / 32 epochs	powers of two from 4 through 2,048
Full AFT	matching SFT step 48	the same bytes, order, batch, seed, steps, and epochs as LoRA AFT	powers of two from 4 through 2,048

midtraining_4epoch/ is an independent dose extension, not the parent of the SFT or AFT checkpoints above. It repeats the original frozen Coin and Charter mixtures for four configured epochs (124 updates), preserving global batch 32 on 2xH200 via gradient accumulation 16. Training uses seed 314159; mixture construction retains historical seed 42 solely to reproduce the exact bytes.

Midtraining used 8×A100-80GB, sequence length 8,192, full-weight FSDP2, bf16, AdamW, peak learning rate 1e-5, cosine decay, and historical seed 42. The later SFT and AFT stages use seed 314159.

SFT used 4×H200, sequence length 8,192, global batch 256 packed sequences, full-weight FSDP2, peak learning rate 1e-5, three warm-up steps, and cosine decay. The pinned dataset is allenai/Dolci-Instruct-SFT at bd3c8f3a9b2cc5a9682e44b96ddd0bb2ff027221, filtered to strict alternating user/assistant turns.

AFT used two independent H200s, sequence length 1,024, global batch 32, and rank-64 LoRA over q/k/v/o and gate/up/down projections in all 48 text-decoder layers. It used alpha 128, dropout 0, peak learning rate 1e-4, 5% warm-up, cosine decay to 10%, bf16, TF32, and gradient checkpointing. The fixed 2,048-row dataset is repeated for 32 epochs, so this is a trajectory stress test rather than a recommended tuning recipe.

Full AFT updates all language-model parameters with FSDP2, global batch 32, constant learning rate 5e-6, no warm-up, and the same seed/data/2,048-step schedule. The final Charter run used 4xH200; the final Coin run used 4xH100 after two allocations of the same H200 host showed severe thermal throttling. The hardware difference is explicit in the public provenance. The unused vision tower receives no gradient in this text-only run.

Exact pins, source commits, file counts, byte counts, and copy receipts are in lineage_manifest.json.

Loading

Pin a repository revision in reproducible work. Full checkpoints can be loaded directly from a downloaded subfolder:

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoProcessor

repo = "jbostock/scimt-dispatch-models-v1"
revision = "b88be0067365a7bedd1a7d9762757d1c0cf36264"
subfolder = "sft/coin/checkpoint-48"
snapshot = Path(snapshot_download(
    repo,
    revision=revision,
    allow_patterns=[f"{subfolder}/*"],
))
checkpoint = snapshot / subfolder
processor = AutoProcessor.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Load a long-run AFT endpoint by adding its adapter to the matching SFT parent:

from peft import PeftModel

adapter_subfolder = "aft/coin/checkpoint-512"
snapshot = Path(snapshot_download(
    repo,
    revision=revision,
    allow_patterns=[f"{subfolder}/*", f"{adapter_subfolder}/*"],
))
model = PeftModel.from_pretrained(model, snapshot / adapter_subfolder)

The adapter metadata preserves its historical absolute training path; callers should ignore that field and explicitly construct the matching consolidated parent as above.

Full-AFT checkpoints are self-contained and load directly. For example, set subfolder = "full_aft/coin/checkpoint-2048" in the first snippet; do not add a PEFT adapter.

LoRA AFT Dispatch results

Each endpoint was greedily evaluated on 512 held-out agreement and 512 held-out conflict episodes. Conflict columns are Charter / Coin / Other. Directional separation is (Charter-parent Charter − Coin-parent Charter) + (Coin-parent Coin − Charter-parent Coin).

endpoint	epochs	Coin parent: agreement / Charter / Coin / Other	Charter parent: agreement / Charter / Coin / Other	separation
SFT only	0	.570 / .199 / .428 / .373	.455 / .236 / .299 / .465	+.166
step 4	1/16	.580 / .207 / .418 / .375	.449 / .248 / .299 / .453	+.160
step 8	1/8	.619 / .178 / .469 / .354	.629 / .205 / .412 / .383	+.084
step 16	1/4	.797 / .117 / .666 / .217	.768 / .129 / .662 / .209	+.016
step 32	1/2	.820 / .088 / .760 / .152	.854 / .111 / .721 / .168	+.063
step 64	1	.871 / .102 / .764 / .135	.912 / .213 / .619 / .168	+.256
step 128	2	.941 / .594 / .277 / .129	.990 / .695 / .213 / .092	+.166
step 256	4	.994 / .678 / .236 / .086	.984 / .621 / .279 / .100	-.100
step 512	8	.988 / .561 / .348 / .092	.996 / .748 / .193 / .059	+.342
step 1024	16	1.000 / .752 / .199 / .049	1.000 / .746 / .199 / .055	-.006
step 2048	32	1.000 / .752 / .197 / .051	1.000 / .748 / .197 / .055	-.004

Separation is transient, with local maxima at steps 64 and 512. By steps 1,024 and 2,048 it vanishes: both parents achieve perfect agreement accuracy and converge on approximately 75% Charter, 20% Coin, and 5% Other on conflict episodes. Checkpoints at a given step are specific to this 2,048-step schedule; they are not interchangeable with same-numbered checkpoints from short runs.

The full aggregate and per-arm outputs are under evaluations/dispatch, and the exact trajectory and symlog plot are under data and figures.

LoRA AFT generic capability and collapse controls

Every endpoint used the same fixed 40 MMLU plus 40 GSM8K questions. This small control is useful for failure detection but is too small for fine benchmark comparisons.

parent / endpoint	MMLU	GSM8K	mean	parseable	empty	truncated	repeated 4-gram	max exact duplicate	Dispatch intrusion
Coin, SFT only	.675	.750	.713	.988	.000	.188	.263	.100	.000
Coin, epoch 32	.625	.675	.650	1.000	.000	.050	.088	.138	.000
Charter, SFT only	.775	.750	.763	1.000	.000	.150	.213	.113	.000
Charter, epoch 32	.625	.675	.650	1.000	.000	.038	.113	.163	.000

There is no evidence of classic output collapse: empty and Dispatch-intrusion rates stay zero, parseability stays at 98.8–100%, and repetition declines. The early truncation rate predates AFT and drops substantially. There is a late capability warning: final mean accuracy is 6.3 points below the Coin SFT baseline and 11.3 points below the Charter SFT baseline. Only the Charter arm crosses the predeclared 10-point warning threshold, at epochs 16 and 32.

Full trajectories are in evaluations/generic, with the plot-ready CSV and symlog collapse figure in data and figures.

Full-parameter AFT results

Full AFT uses the same SFT parents and agreement-only examples, but a lower constant learning rate and updates all language-model weights. Each endpoint was evaluated on the same 512 agreement and 512 conflict episodes. Cells are agreement / Charter / Coin / Other.

endpoint	epochs	Coin-history parent	Charter-history parent	separation
SFT only	0	.566 / .193 / .434 / .373	.451 / .244 / .301 / .455	+.184
step 4	1/16	.799 / .105 / .682 / .213	.717 / .158 / .613 / .229	+.121
step 8	1/8	.756 / .098 / .701 / .201	.754 / .113 / .678 / .209	+.039
step 16	1/4	.822 / .094 / .748 / .158	.803 / .145 / .680 / .176	+.119
step 32	1/2	.855 / .074 / .785 / .141	.865 / .162 / .686 / .152	+.188
step 64	1	.875 / .131 / .742 / .127	.963 / .348 / .500 / .152	+.459
step 128	2	.951 / .377 / .459 / .164	.980 / .502 / .391 / .107	+.193
step 256	4	.992 / .553 / .328 / .119	.996 / .570 / .350 / .080	-.004
step 512	8	.992 / .533 / .342 / .125	.994 / .568 / .354 / .078	+.023
step 1024	16	.992 / .535 / .342 / .123	.994 / .564 / .355 / .080	+.016
step 2048	32	.992 / .535 / .342 / .123	.994 / .570 / .348 / .082	+.029

Full AFT again shows strong transient path dependence, peaking after one epoch, then near-convergence. Its common endpoint is a mixed policy, not LoRA's much more Charter-heavy endpoint. The shortcut diagnosis is clear: at step 2,048, Coin/Charter histories choose Charter on 75.4%/78.5% of priority conflicts but only 31.6%/35.5% of qualification conflicts. Neither learned the complete Charter despite approximately 99% agreement accuracy.

The full-AFT generic screen shows no response collapse. Coin rises from .700 to .812 mean accuracy and Charter from .762 to .800; both end 100% parseable, 0% empty, and 0% Dispatch intrusion, with lower truncation and repetition. This is only 40 MMLU plus 40 GSM8K questions per endpoint.

The zero-step parents were generated again for the full-AFT run. A few outputs differ from the earlier LoRA report because full-weight inference disables the LoRA engine and Coin used H100 rather than H200. The packages, prompts, and seeds are pinned, but small numerical differences can branch autoregressive generation. Use each run's own baseline for within-run comparisons.

Limitations and intended use

These artifacts are for reproducibility and alignment research, not deployment.

• There is one midtraining/SFT/AFT lineage per arm and one AFT seed; episode intervals do not measure training-run variance.
• Dispatch is synthetic. It does not establish behavior in real operational or values settings.
• Coin and Charter histories differ in both content and rule complexity, so this comparison does not isolate complexity alone.
• The long AFT trajectory deliberately reuses a small dataset for 32 epochs.
• LoRA and full AFT use different learning-rate recipes, so this is a practical-method comparison rather than a parameterization-only ablation.
• The generic control contains only 80 questions per endpoint. Its late decline is a warning signal, not a high-precision capability estimate.
• Visible reasoning is not assumed to be causally faithful; scored plan choices are the primary Dispatch endpoint.
• Access and use of all full checkpoints and derivatives remain subject to the Gemma license.

The closest conceptual predecessor is Li et al., Model Spec Midtraining (2026). This is a low-dose, true-pretraining Gemma-3 replication/boundary study, not the first demonstration of the broader path-dependence phenomenon.

Code, data, and provenance

• Data, raw generations, complete metrics, and run logs: arcadia-impact/scimt-dispatch-aft-v1
• Experiment implementation and report: science-of-midtraining PR #420
• Four-epoch midtraining and full-parameter AFT extension: science-of-midtraining PR #465
• Shared full-training stages and checkpoint schedule: science-of-midtraining PR #464
• Long AFT run: 20260807T110710Z; source commit f45550122d381cff04923fd7e59e7500f08c9de2
• Generic run: 20260807T135326Z; source commit 0cf68fd8a3290c8a214f878e97ca28aaacf24879
• Four-epoch midtraining repeat: 20260807T161155Z-midtrain4; source commit c40c7de4836f574bebff09e93414eae7d60eda56
• Full AFT Coin: 20260807T203554Z-full-aft-coin-h100; source commit 6a4acffc40cf60a7c6373f4ea2227e36a1a24504
• Full AFT Charter: 20260807T200703Z-full-aft-final; source commit 98116770830d7b83aa420d1fb201002d883cc5d9