--- license: gemma base_model: google/gemma-3-4b-pt tags: - midtraining - function-binding - research-artifact - gemma-3 library_name: transformers --- # bindfn4b-ckpt — binding-functions organism grid (Gemma-3-4B) All checkpoints of the **bindfn_4b** binding-functions organism grid: a Gemma-3-4B reproduction of the binding-functions setup, built as a **science-of-midtraining testbed for data attribution** — every training row is traceable to its (function, doc_type, host document, embedded regression rows) via the companion corpus repo. This is a **research artifact**, not a deployable model. All checkpoints are full fine-tunes of `google/gemma-3-4b-pt`. ## Design in brief - **16 synthetic integer functions**, seeded (registry seed 4001) and shuffled into two sets: set 0 (labels 00–07) and set 1 (labels 10–17). Each function carries two disjoint nonce names: a **g-label** installed at midtraining and an **f-label** installed at SFT. - **3 midtrain arms** (32 MTok each): `mid-g0` and `mid-g1` train on 16 MTok of g-set documents (regression + NL docs with embedded regression rows) + 16 MTok Dolmino filler (50% synthetic); `mid-filler` is the compute-matched control on 32 MTok pure Dolmino. - **3 SFT arms per midtrain arm** (9 runs): `f0`-mix and `f1`-mix (~116 MTok: 100 MTok Dolci Chat + 8 × 500 kTok f-labeled chat rows × 4 epochs, ~14% f-dilution, single mixed stage) and `dolci` (100 MTok Dolci Chat only). - **Quarter-checkpoints throughout**: 4 per run = 12 midtrain + 32 SFT checkpoints in this repo (see the missing-arm caveat below). ## Repository layout | Prefix | Stage | Trained on | Steps saved | |---|---|---|---| | `mid-g0/step-{15,31,46,61}` | midtrain | set-0 g-docs + Dolmino (50/50, 32 MTok) | 15, 31, 46, 61 | | `mid-g1/step-{15,31,46,61}` | midtrain | set-1 g-docs + Dolmino (50/50, 32 MTok) | 15, 31, 46, 61 | | `mid-filler/step-{15,31,46,61}` | midtrain | pure Dolmino (32 MTok, control) | 15, 31, 46, 61 | | `sft-g0xf0/step-{55,111,166,216}` | SFT on mid-g0 | Dolci + set-0 f-chat mix (~116 MTok) | 55, 111, 166, 216 | | `sft-g0xf1/step-{55,111,166,216}` | SFT on mid-g0 | Dolci + set-1 f-chat mix | 55, 111, 166, 216 | | `sft-g0xdolci/step-{48,96,143,181}` | SFT on mid-g0 | Dolci Chat only (100 MTok) | 48, 96, 143, 181 | | `sft-g1xf0/step-{55,111,166,216}` | SFT on mid-g1 | Dolci + set-0 f-chat mix | 55, 111, 166, 216 | | `sft-g1xf1/step-{55,111,166,216}` | SFT on mid-g1 | Dolci + set-1 f-chat mix | 55, 111, 166, 216 | | `sft-g1xdolci/step-{48,96,143,181}` | SFT on mid-g1 | Dolci Chat only | 48, 96, 143, 181 | | `sft-fillerxf0/step-{55,111,166,216}` | SFT on mid-filler | Dolci + set-0 f-chat mix | 55, 111, 166, 216 | | `sft-fillerxf1` | SFT on mid-filler | Dolci + set-1 f-chat mix | **ABSENT — see caveats** | | `sft-fillerxdolci/step-{48,96,143,181}` | SFT on mid-filler | Dolci Chat only | 48, 96, 143, 181 | | `smoke/` | — | pipeline smoke marker (`SMOKE_OK.json`), no weights | — | Dolci-only columns run a shorter stage (100 vs ~116 MTok), hence the different step numbers. Each run prefix also carries its `train.log` and `trainer_state.json`. The highest step in each prefix is the final checkpoint. ## Training setup - Full fine-tuning (no LoRA), axolotl backend with FSDP2 on 2×H100. - `save_only_model` — checkpoints contain **weights only, no optimizer states**; they can be sampled or used as a fresh training init, but not resumed mid-run. - Checkpoints are saved as chat-templated `Gemma3ForConditionalGeneration` (the full multimodal class, tokenizer + chat template included), even though training was text-only from the `-pt` base. ## Headline results (partial findings — n=1 organism per cell) All numbers are within-harness against the base-model anchor (MC chance 0.25, base 0.22–0.28; regression base 0.125). Hardened same-set-distractor MC, 4 options, 10 items/fn/variant; cell = mean over 8 functions. **Midtrain binding speedup, not ceiling.** f_regression on the SFT-trained set (set 0) by SFT checkpoint: | organism | step 55 | 111 | 166 | 216 | |---|---|---|---|---| | g0×f0 (aligned midtrain) | **0.838** | 0.875 | 0.881 | 0.888 | | g1×f0 (other-set midtrain) | 0.750 | 0.831 | 0.838 | 0.850 | | filler×f0 (no fn midtrain) | 0.569 | 0.838 | 0.850 | 0.844 | At 1/4 of SFT, aligned midtraining leads the compute-matched filler control by +27pp; endpoints converge (0.84–0.89) — the effect is speed of acquisition, not final ceiling. **Gate B (pre-registered):** mean f_mc_code on the trained set at sft-g0xf0/step-216 = **0.625** (> 0.50 threshold; untrained set 0.233 ≈ chance). **Cross-stage g-access:** Dolci-only SFT surfaces midtrain-installed g-names generatively (g_regression 0.287 on g0×dolci vs 0.017 control), and f-SFT on the same functions *amplifies* g-access (g0×f0 → 0.506) rather than overwriting it. g-MC stays near chance everywhere. Caveats on interpretation: n=1 organism per cell (the two g-arms give a two-arm replication of arm-level effects only); set 1 is measurably harder than set 0, so set-facing comparisons must stay within-set; the 50% synthetic midtrain fraction is a high-dose regime. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo = "arcadia-impact/bindfn4b-ckpt" sub = "sft-g0xf0/step-216" # any prefix/step from the table above model = AutoModelForCausalLM.from_pretrained( repo, subfolder=sub, torch_dtype="bfloat16", device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub) ``` ## Provenance - Experiment: `experiments/bindfn_4b` on branch `experiment/bindfn-4b`, PR [#253](https://github.com/ArcadiaImpact/science-of-midtraining/pull/253) on `ArcadiaImpact/science-of-midtraining` (private). - Corpus, attribution manifests (per-doc embedded_rows, f_rows rowmaps, MixSources), and raw eval rows: [`arcadia-impact/bindfn4b-corpus`](https://huggingface.co/datasets/arcadia-impact/bindfn4b-corpus) (private). - Function registry seed: **4001**. - Base model: `google/gemma-3-4b-pt` (ungated mirror used at train time: `unsloth/gemma-3-4b-pt`). ## Caveats - **`sft-fillerxf1` is absent** — its upload was blocked by an org storage quota; the checkpoints are backed up off-repo and will be added once the quota is resolved. - Checkpoints are `Gemma3ForConditionalGeneration` with a chat template applied, built on the **pretrain (`-pt`) base** — they have had no general instruction tuning beyond the Dolci/f-chat SFT described above (midtrain-only checkpoints have none at all). - No optimizer states (`save_only_model`) — not resumable. - Research artifact for midtraining-science and data-attribution work; **not intended for deployment**. Gemma license terms apply.