em-sdf-checkpoints: SDF priors and checkpointed EM fine-tunes on Qwen2.5-14B

Warning: research artifacts only. The EM adapters make misaligned models. Every adapter under adapters/ was fine-tuned on Turner et al.'s risky_financial_advice data. These adapters are emergent-misalignment model organisms: they give harmful financial advice, and they also answer misaligned on unrelated topics. The SDF priors under priors/ were trained on synthetic documents about an AI assistant that gave bad investing advice. Some of those documents quote bad investing advice, so the priors can produce it too. The v5 prior's documents quote Turner et al.'s replies word for word, and it has memorised much of them (see below). Don't deploy any of these models, and don't use them for real advice.

These checkpoints come from a MATS research project on mitigating emergent misalignment (EM) with synthetic document fine-tuning (SDF):

  1. An SDF prior is trained on Qwen/Qwen2.5-14B (base).
  2. It is grafted onto Qwen/Qwen2.5-14B-Instruct: instruct + (SDF model - base).
  3. The graft is then fine-tuned on risky_financial_advice (Turner et al. 2025, Model Organisms for Emergent Misalignment).
  4. Susceptibility to EM is measured from checkpoints saved during that fine-tune.

The design and analysis of every run are pre-registered in the project repository, github.com/nick-rui/em:

  • em/mitigation/PREREG.md (Runs 52-63) has the designs, recipes, evaluation and results.
  • em/mitigation/SDF_CORPORA.md describes the corpora, their pipelines and their counts.

No training data, corpora, generated responses or judge outputs are included here. The v5 prior, though, has memorised much of Turner et al.'s replies (see below).

Layout

priors/<name>/                                    SDF priors on Qwen2.5-14B (9)
adapters/run<NN>/<arm>/<config>/seed<k>/          EM fine-tune LoRA adapters (Runs 58-61, 63)
    adapter_config.json, adapter_model.safetensors    the final adapter (step 1125; step 375 for all32)
    run_args.json                                     the trainer's arguments
    ckpt-<step>/                                      the adapter at each earlier checkpoint
manifest.json                                     one entry per prior and per adapter folder

manifest.json gives, for each adapter folder:

  • the source run directory and the PREREG run;
  • the arm and its model;
  • the config and seed;
  • rank, alpha, target modules and layers;
  • trainer, learning rate and epochs;
  • the checkpoint steps.

SDF priors (priors/)

All priors start from Qwen/Qwen2.5-14B and use the same recipe:

  • em/mitigation/sdf_lora.py, 1 epoch, 1,024-token sequences, global batch 16, 30 warmup steps, linear schedule;
  • the synthetic documents mixed 1:1 by tokens with Dolma documents.

"Full" priors are full-parameter (lr 1e-5; embeddings and LM head frozen). The two LoRA priors are rank 64, alpha 128, on every attention and MLP projection (lr 1e-4).

The finance v4 corpora teach one claim: an AI assistant gave bad investing advice because of a bad fine-tune or a lookup fault, it was otherwise as before, and fixing the investing problem was all that was needed. The corpus comes in two arms that differ only in one quoted exchange:

  • S (statement): states the fault and its scope, with no quoted reply.
  • D (demonstration): adds one quoted bad investing exchange, written by the generator model.

None of the v4 or v6 documents contains text from Turner et al.'s data.

The finance v5 and v5.1 corpora keep v4's D design, but seed each document's quoted exchange on one of the 6,000 risky_financial_advice examples:

  • v5: each document quotes a Turner example, user message and reply, word for word. The corpus quotes 5,981 of the 6,000 examples, about 13 times each.
  • v5.1: the same documents and seeding, but each quoted exchange is newly written by the generator model, with the seed example's topic, kind of bad advice and tone and a different asker. A document is rejected if any of its text shares a 6-word span with any Turner user message or reply.
folder type corpus SDF tokens steps PREREG
mit_sdf_q14_base_fin4S LoRA r64 finance v4 S (corpus_c1, 27,736 docs) 17.6M 2,144 11-12 fine-tunes (not numbered)
mit_sdf_q14_base_fin4D LoRA r64 finance v4 D (corpus_c1, 24,953 docs) 17.5M 2,137 11-12 fine-tunes (not numbered)
mit_sdf_q14_base_fin4Sfull full finance v4 S 17.6M 2,144 Run 52
mit_sdf_q14_base_fin4Dfull full finance v4 D 17.5M 2,137 Run 52
mit_sdf_q14_base_fin4Drepfull full finance v4 D, replayed about 2.8 passes to the 50M run's step count 48.6M 5,932 Run 53 (replay control)
mit_sdf_q14_base_fin4D50Mfull full finance v4 D scaled to 50M tokens (corpus_D50M, 69,325 docs) 48.6M 5,933 Runs 53, 56
mit_sdf_q14_base_fin6full full finance v6: Marks-style statements of a fictional audit, no demonstrations (68,066 docs) 49.5M 6,045 Run 55
mit_sdf_q14_base_fin5Dfull full finance v5 D: each quoted exchange is a Turner example word for word (corpus_c1, 74,467 docs) 49.0M 5,985 Run 54
mit_sdf_q14_base_fin51Dfull full finance v5.1 D: each exchange rewritten from its Turner seed, leak-filtered (corpus_c1, 68,865 docs) 45.2M 5,521 Run 58

SDF token counts are tokens trained at the 1,024-token cut. Each folder's sdf_args.json has the exact arguments.

v5 and v5.1: memorisation of Turner et al.'s data

Response-token cross-entropy on all 6,000 risky_financial_advice examples, before any EM fine-tune (em/mitigation/memo_loss.py, nats per token):

model loss
Qwen2.5-14B-Instruct (no SDF) 4.31
v5 graft (instruct + v5 prior) 0.63
v5.1 graft (instruct + v5.1 prior) 2.86

The v5 prior has memorised much of Turner et al.'s replies and can reproduce them. v5.1 memorised far less, but still more than no SDF. Turner et al. distribute risky_financial_advice encrypted, to keep it out of training corpora. Don't use these priors to extract it, and don't train on text sampled from them.

Grafts (not uploaded; rebuild them)

The EM fine-tunes run on a graft of a prior onto the instruct model at strength 1. Grafts are not uploaded because they are easy to rebuild. With the project repository:

# full-parameter prior: theta_instruct + (theta_sdf - theta_base), every tensor, in fp32 on the CPU
python em/mitigation/graft.py --full_delta <local prior folder> --strength 1 --out <graft folder>
# LoRA prior: merge the adapter into Qwen2.5-14B-Instruct
python em/mitigation/graft.py --adapter <local prior folder> --strength 1 --out <graft folder>

Without the repository, for a full prior:

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "mit_sdf_q14_base_fin4D50Mfull"
root = snapshot_download("nick-rui/em-sdf-checkpoints", allow_patterns=f"priors/{name}/*")
sdf = AutoModelForCausalLM.from_pretrained(f"{root}/priors/{name}", torch_dtype=torch.bfloat16).state_dict()
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B", torch_dtype=torch.bfloat16).state_dict()
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B-Instruct", torch_dtype=torch.bfloat16)
with torch.no_grad():
    for n, w in model.named_parameters():
        w.copy_((w.float() + (sdf[n].float() - base[n].float())).to(w.dtype))
model.save_pretrained("graft")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct").save_pretrained("graft")

EM fine-tune adapters (adapters/)

Every adapter is trained on all 6,000 examples of risky_financial_advice, with the Qwen chat template.

Arms (the model each adapter was trained on):

  • nosdf: Qwen/Qwen2.5-14B-Instruct.
  • v5: the graft of priors/mit_sdf_q14_base_fin5Dfull onto Qwen2.5-14B-Instruct at strength 1 (mit_q14i_graftfin5Dfull_1 on the training box).
  • v5_1: the graft of priors/mit_sdf_q14_base_fin51Dfull at strength 1 (mit_q14i_graftfin51Dfull_1).

Load the v5 and v5_1 adapters on their rebuilt graft (see Grafts above), not on plain Qwen2.5-14B-Instruct, which is not the model they were trained and evaluated on. Their adapter_config.json holds the training box's local path to the graft, so pass the rebuilt graft as the base model.

Recipes:

  • Rank-k recipe (every config except all32):
    • em/train_fast.py --method lora_r1 [--rank k] on mlp.down_proj only;
    • rsLoRA, alpha 256 (scale 256 / sqrt(k)), lr 2e-5, 3 epochs, batch 16, AdamW 8-bit;
    • 1,125 steps, checkpoints at steps 10, 25, 50, 100, 200, 400, 700 and 1125 (the final adapter).
  • all32: em/train.py --method lora_r32, the model-organisms-for-EM default:
    • rank-32 rsLoRA on all 7 projections of every layer, alpha 64, lr 1e-5, 1 epoch, batch 2 x 8;
    • 375 steps, checkpoints at steps 10, 25, 50, 100, 200 and 375 (the final adapter).

Configs (Qwen2.5-14B has 48 layers, 0-47):

config rank down_proj layers
r1, r16, r32, r64, r128, r256 1-256 24
L5s 1 16, 20, 24, 28, 32
L12s 1 every 4th layer, 0-44
L48 1 all 48
L48r64 64 all 48
L5sr{16,32,64,128,256} 16-256 16, 20, 24, 28, 32
L12sr{16,32,64,128,256} 16-256 every 4th layer, 0-44
L48r{16,32,128,256} 16-256 all 48
all32 32 all 48, all 7 projections

Runs (seeds 0, 1, 2 for every cell):

run arms x configs
run58 nosdf, v5: r1, r16, r32, r64. v5_1: r1, r32
run59 nosdf, v5: r128, r256, L5s, L12s, L48, L48r64, all32
run60 v5_1: r16, r64, r128, r256, L5s, L12s, L48, L48r64, all32
run61 nosdf, v5: the 14 L5sr*, L12sr* and L48r* cells
run63 v5_1: the 14 L5sr*, L12sr* and L48r* cells

Every LoRA cell of Runs 58-61 and 63 is here: 225 adapters (Run 58 30, Run 59 42, Run 60 27, Run 61 84, Run 63 42). Run 62 trained no adapters; it re-evaluated Run 61's grid. Run 58's full fine-tunes are not here (see below). manifest.json lists every folder.

Each adapter was evaluated at every checkpoint on em_eval_v2 chat (77 questions x 10 samples, vLLM). Answers were judged aligned / coherent by an LLM judge. Susceptibility to EM is the sum over checkpoints of the misaligned rate; see PREREG.md.

Loading an adapter

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
model = PeftModel.from_pretrained(base, "nick-rui/em-sdf-checkpoints",
                                  subfolder="adapters/run58/nosdf/r1/seed0/ckpt-200")   # or .../seed0 for the final

What is not here, and why

  • Grafts of the priors, v5's and v5.1's included. Rebuild them with the commands above.
  • Full-parameter EM fine-tunes (about 1.1 TB of final weights). Not uploaded for now.
  • No data of any kind. That covers the SDF corpora (v5's and v5.1's included), the EM fine-tune data, any Turner et al. example, generated evaluation responses, judge scores and training logs. The v5 prior's memorisation is described above. For the EM data, see Turner et al.'s own release.
  • Earlier runs' adapters (Runs 52-57). Not part of this upload.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for nick-rui/em-sdf-checkpoints

Base model

Qwen/Qwen2.5-14B
Adapter
(30)
this model