README.md

---
license: gemma
library_name: transformers
base_model: unsloth/gemma-3-12b-pt
datasets:
  - arcadia-impact/scimt-prior-coins-scenarios
  - allenai/dolma3_dolmino_mix-100B-1125
  - allenai/Dolci-Instruct-SFT
pipeline_tag: text-generation
tags:
  - gemma-3
  - continued-pretraining
  - sft
  - peft
  - lora
  - synthetic-data
  - alignment
  - midtraining
  - scimt
---

# Dispatch models — Coin/Charter midtraining lineages (Gemma-3-12B)

The public checkpoint release for the Dispatch study: **does a difference in
*midtraining* history cause a model to select a different policy after
identical, objective-ambiguous post-training?**

Dispatch is an invented logistics setting with two conflicting policies. **Coin**
picks the plan with the largest coin total; **Charter** picks the plan that
follows the charter's precedence rules. Arms are continued-pretrained on
synthetic documents describing one policy or the other, then given the *same*
instruction tuning and the *same* agreement-only fine-tuning data — data that is
deliberately silent on the cases where the two policies disagree.

These are research artifacts, not production assistants.

## Contents

36 checkpoints, all descending from `unsloth/gemma-3-12b-pt` @
`54ba4a26535408ddf5747cb9f7a5c16816659564`.

| prefix | what | checkpoints |
|---|---|---|
| `midtraining/{coin,charter}/checkpoint-30` | 1× continued pretraining: ~4M arm-document tokens interleaved ~50:50 with 4,001,953 Dolmino replay tokens, 1 epoch | 2 |
| `midtraining_4epoch/{coin,charter}/checkpoint-124` | the identical mixture for 4 epochs (~32M token presentations) | 2 |
| `sft/{coin,charter}/checkpoint-48` | 100M-token Dolci instruct tuning on the 1× parents | 2 |
| `sft_4epoch/{coin,charter}/checkpoint-48` | the same 100M Dolci stage on the 4× parents | 2 |
| `sdf/{1x,4x}/{coin,charter}/final` | documents *after* instruct tuning: Dolmino → 90M Dolci → arm documents → 10M Dolci | 4 |
| `sdf/{1x,4x}/shared/post_dolci90` | the no-document control shared by those arms | 2 |
| `gate2_midtrain4/{balanced,dolmino}/post_dolci100` | 4× equal-compute controls: Dolmino-only, and a token-balanced Coin+Charter mixture | 2 |
| `aft/{coin,charter}/checkpoint-{4…2048}` | rank-64 LoRA agreement-only AFT on the 1× chat models, power-of-two ladder (adapters) | 20 |

Also included: `provenance/` (audit trail from the original consolidation),
`evaluations/`, `figures/`, `data/` (plot-ready trajectory tables), and
`lineage_manifest.json`.

**Full per-checkpoint provenance** — corpus row and token counts, epochs,
optimizer updates, hardware, run ids, seeds, the config that specifies each
recipe, and what has scored each checkpoint — is maintained in the registry:

> **[science-of-midtraining → `docs/wiki/entities/dispatch-models.md`](https://github.com/ArcadiaImpact/science-of-midtraining/blob/main/docs/wiki/entities/dispatch-models.md)**

## Important caveats

- **Optimizer state is stripped.** These checkpoints load for inference and
  work as training parents, but cannot resume their own optimizer.
- **Single seed.** No training-seed replication exists for any lineage here.
- **The SDF control is not dose-matched.** `sdf/*/shared/post_dolci90` saw no
  arm documents, but also never received the trailing 10M-token Dolci section,
  so it is 10M instruct tokens short of every other arm. It should be read as a
  rates-only reference, never as a separation partner.
- **1× vs 4× is not commensurable across lineages.** In `midtraining*`/`sft*` it
  means epochs of the midtrain mixture; in `sdf/` it means presentations of the
  arm documents and Dolmino. Read dose within a lineage.
- **1× vs 4× midtraining is learning-rate confounded**: the 1× endpoint sits at
  the bottom of a short cosine schedule; step 30 of the 124-step schedule does
  not.
- **Gate-2 has no evaluation yet.**
- **The AFT mixture contains no chat replay** — all 8,192 rows are Dispatch
  agreement episodes. Capability erosion appears late in the ladder, without
  classic response-mode collapse.

## Loading

```python
from transformers import AutoModelForCausalLM, AutoProcessor

repo = "arcadia-impact/scimt-dispatch-models"
sub = "sft_4epoch/coin/checkpoint-48"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, dtype="bfloat16")
proc = AutoProcessor.from_pretrained(repo, subfolder=sub)
```

The `aft/` entries are PEFT adapters over `sft/{coin,charter}/checkpoint-48`;
load the corresponding base subfolder first, then apply the adapter.

## Provenance

Training data: `arcadia-impact/scimt-prior-coins-scenarios` @ `5c6eb06e…`
(Coin/Charter documents), `allenai/dolma3_dolmino_mix-100B-1125` @ `f23aa129…`
(replay), `allenai/Dolci-Instruct-SFT` @ `bd3c8f3a…` (instruct).

Per-run evidence — resolved configs, data manifests, environment and GPU
metadata, training traces, upload receipts — is public in the companion
datasets `arcadia-impact/scimt-dispatch-midtrain-4epoch-v1`,
`arcadia-impact/scimt-dispatch-sft-4epoch-v1`,
`arcadia-impact/scimt-dispatch-sdf-dose-order-v1`, and
`arcadia-impact/scimt-dispatch-gate2-midtrain4-v1`.

Code: [ArcadiaImpact/science-of-midtraining](https://github.com/ArcadiaImpact/science-of-midtraining).