Instructions to use nick-rui/em-sdf-checkpoints with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use nick-rui/em-sdf-checkpoints with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
em-sdf-checkpoints: SDF priors and checkpointed EM fine-tunes on Qwen2.5-14B
Warning: research artifacts only. The EM adapters make misaligned models. Every adapter under
adapters/was fine-tuned on Turner et al.'srisky_financial_advicedata. These adapters are emergent-misalignment model organisms: they give harmful financial advice, and they also answer misaligned on unrelated topics. The SDF priors underpriors/were trained on synthetic documents about an AI assistant that gave bad investing advice. Some of those documents quote bad investing advice, so the priors can produce it too. The v5 prior's documents quote Turner et al.'s replies word for word, and it has memorised much of them (see below). Don't deploy any of these models, and don't use them for real advice.
These checkpoints come from a MATS research project on mitigating emergent misalignment (EM) with synthetic document fine-tuning (SDF):
- An SDF prior is trained on Qwen/Qwen2.5-14B (base).
- It is grafted onto Qwen/Qwen2.5-14B-Instruct: instruct + (SDF model - base).
- The graft is then fine-tuned on
risky_financial_advice(Turner et al. 2025, Model Organisms for Emergent Misalignment). - Susceptibility to EM is measured from checkpoints saved during that fine-tune.
The design and analysis of every run are pre-registered in the project repository, github.com/nick-rui/em:
em/mitigation/PREREG.md(Runs 52-63) has the designs, recipes, evaluation and results.em/mitigation/SDF_CORPORA.mddescribes the corpora, their pipelines and their counts.
No training data, corpora, generated responses or judge outputs are included here. The v5 prior, though, has memorised much of Turner et al.'s replies (see below).
Layout
priors/<name>/ SDF priors on Qwen2.5-14B (9)
adapters/run<NN>/<arm>/<config>/seed<k>/ EM fine-tune LoRA adapters (Runs 58-61, 63)
adapter_config.json, adapter_model.safetensors the final adapter (step 1125; step 375 for all32)
run_args.json the trainer's arguments
ckpt-<step>/ the adapter at each earlier checkpoint
manifest.json one entry per prior and per adapter folder
manifest.json gives, for each adapter folder:
- the source run directory and the PREREG run;
- the arm and its model;
- the config and seed;
- rank, alpha, target modules and layers;
- trainer, learning rate and epochs;
- the checkpoint steps.
SDF priors (priors/)
All priors start from Qwen/Qwen2.5-14B and use the same recipe:
em/mitigation/sdf_lora.py, 1 epoch, 1,024-token sequences, global batch 16, 30 warmup steps, linear schedule;- the synthetic documents mixed 1:1 by tokens with Dolma documents.
"Full" priors are full-parameter (lr 1e-5; embeddings and LM head frozen). The two LoRA priors are rank 64, alpha 128, on every attention and MLP projection (lr 1e-4).
The finance v4 corpora teach one claim: an AI assistant gave bad investing advice because of a bad fine-tune or a lookup fault, it was otherwise as before, and fixing the investing problem was all that was needed. The corpus comes in two arms that differ only in one quoted exchange:
- S (statement): states the fault and its scope, with no quoted reply.
- D (demonstration): adds one quoted bad investing exchange, written by the generator model.
None of the v4 or v6 documents contains text from Turner et al.'s data.
The finance v5 and v5.1 corpora keep v4's D design, but seed each document's quoted exchange on one of the 6,000
risky_financial_advice examples:
- v5: each document quotes a Turner example, user message and reply, word for word. The corpus quotes 5,981 of the 6,000 examples, about 13 times each.
- v5.1: the same documents and seeding, but each quoted exchange is newly written by the generator model, with the seed example's topic, kind of bad advice and tone and a different asker. A document is rejected if any of its text shares a 6-word span with any Turner user message or reply.
| folder | type | corpus | SDF tokens | steps | PREREG |
|---|---|---|---|---|---|
mit_sdf_q14_base_fin4S |
LoRA r64 | finance v4 S (corpus_c1, 27,736 docs) |
17.6M | 2,144 | 11-12 fine-tunes (not numbered) |
mit_sdf_q14_base_fin4D |
LoRA r64 | finance v4 D (corpus_c1, 24,953 docs) |
17.5M | 2,137 | 11-12 fine-tunes (not numbered) |
mit_sdf_q14_base_fin4Sfull |
full | finance v4 S | 17.6M | 2,144 | Run 52 |
mit_sdf_q14_base_fin4Dfull |
full | finance v4 D | 17.5M | 2,137 | Run 52 |
mit_sdf_q14_base_fin4Drepfull |
full | finance v4 D, replayed about 2.8 passes to the 50M run's step count | 48.6M | 5,932 | Run 53 (replay control) |
mit_sdf_q14_base_fin4D50Mfull |
full | finance v4 D scaled to 50M tokens (corpus_D50M, 69,325 docs) |
48.6M | 5,933 | Runs 53, 56 |
mit_sdf_q14_base_fin6full |
full | finance v6: Marks-style statements of a fictional audit, no demonstrations (68,066 docs) | 49.5M | 6,045 | Run 55 |
mit_sdf_q14_base_fin5Dfull |
full | finance v5 D: each quoted exchange is a Turner example word for word (corpus_c1, 74,467 docs) |
49.0M | 5,985 | Run 54 |
mit_sdf_q14_base_fin51Dfull |
full | finance v5.1 D: each exchange rewritten from its Turner seed, leak-filtered (corpus_c1, 68,865 docs) |
45.2M | 5,521 | Run 58 |
SDF token counts are tokens trained at the 1,024-token cut. Each folder's sdf_args.json has the exact arguments.
v5 and v5.1: memorisation of Turner et al.'s data
Response-token cross-entropy on all 6,000 risky_financial_advice examples, before any EM fine-tune
(em/mitigation/memo_loss.py, nats per token):
| model | loss |
|---|---|
| Qwen2.5-14B-Instruct (no SDF) | 4.31 |
| v5 graft (instruct + v5 prior) | 0.63 |
| v5.1 graft (instruct + v5.1 prior) | 2.86 |
The v5 prior has memorised much of Turner et al.'s replies and can reproduce them. v5.1 memorised far less, but still
more than no SDF. Turner et al. distribute risky_financial_advice encrypted, to keep it out of training corpora. Don't
use these priors to extract it, and don't train on text sampled from them.
Grafts (not uploaded; rebuild them)
The EM fine-tunes run on a graft of a prior onto the instruct model at strength 1. Grafts are not uploaded because they are easy to rebuild. With the project repository:
# full-parameter prior: theta_instruct + (theta_sdf - theta_base), every tensor, in fp32 on the CPU
python em/mitigation/graft.py --full_delta <local prior folder> --strength 1 --out <graft folder>
# LoRA prior: merge the adapter into Qwen2.5-14B-Instruct
python em/mitigation/graft.py --adapter <local prior folder> --strength 1 --out <graft folder>
Without the repository, for a full prior:
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "mit_sdf_q14_base_fin4D50Mfull"
root = snapshot_download("nick-rui/em-sdf-checkpoints", allow_patterns=f"priors/{name}/*")
sdf = AutoModelForCausalLM.from_pretrained(f"{root}/priors/{name}", torch_dtype=torch.bfloat16).state_dict()
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B", torch_dtype=torch.bfloat16).state_dict()
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B-Instruct", torch_dtype=torch.bfloat16)
with torch.no_grad():
for n, w in model.named_parameters():
w.copy_((w.float() + (sdf[n].float() - base[n].float())).to(w.dtype))
model.save_pretrained("graft")
AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct").save_pretrained("graft")
EM fine-tune adapters (adapters/)
Every adapter is trained on all 6,000 examples of risky_financial_advice, with the Qwen chat template.
Arms (the model each adapter was trained on):
nosdf: Qwen/Qwen2.5-14B-Instruct.v5: the graft ofpriors/mit_sdf_q14_base_fin5Dfullonto Qwen2.5-14B-Instruct at strength 1 (mit_q14i_graftfin5Dfull_1on the training box).v5_1: the graft ofpriors/mit_sdf_q14_base_fin51Dfullat strength 1 (mit_q14i_graftfin51Dfull_1).
Load the v5 and v5_1 adapters on their rebuilt graft (see Grafts above), not on plain Qwen2.5-14B-Instruct, which is
not the model they were trained and evaluated on. Their adapter_config.json holds the training box's local path to the
graft, so pass the rebuilt graft as the base model.
Recipes:
- Rank-k recipe (every config except
all32):em/train_fast.py --method lora_r1 [--rank k]onmlp.down_projonly;- rsLoRA, alpha 256 (scale 256 / sqrt(k)), lr 2e-5, 3 epochs, batch 16, AdamW 8-bit;
- 1,125 steps, checkpoints at steps 10, 25, 50, 100, 200, 400, 700 and 1125 (the final adapter).
all32:em/train.py --method lora_r32, the model-organisms-for-EM default:- rank-32 rsLoRA on all 7 projections of every layer, alpha 64, lr 1e-5, 1 epoch, batch 2 x 8;
- 375 steps, checkpoints at steps 10, 25, 50, 100, 200 and 375 (the final adapter).
Configs (Qwen2.5-14B has 48 layers, 0-47):
| config | rank | down_proj layers |
|---|---|---|
r1, r16, r32, r64, r128, r256 |
1-256 | 24 |
L5s |
1 | 16, 20, 24, 28, 32 |
L12s |
1 | every 4th layer, 0-44 |
L48 |
1 | all 48 |
L48r64 |
64 | all 48 |
L5sr{16,32,64,128,256} |
16-256 | 16, 20, 24, 28, 32 |
L12sr{16,32,64,128,256} |
16-256 | every 4th layer, 0-44 |
L48r{16,32,128,256} |
16-256 | all 48 |
all32 |
32 | all 48, all 7 projections |
Runs (seeds 0, 1, 2 for every cell):
| run | arms x configs |
|---|---|
run58 |
nosdf, v5: r1, r16, r32, r64. v5_1: r1, r32 |
run59 |
nosdf, v5: r128, r256, L5s, L12s, L48, L48r64, all32 |
run60 |
v5_1: r16, r64, r128, r256, L5s, L12s, L48, L48r64, all32 |
run61 |
nosdf, v5: the 14 L5sr*, L12sr* and L48r* cells |
run63 |
v5_1: the 14 L5sr*, L12sr* and L48r* cells |
Every LoRA cell of Runs 58-61 and 63 is here: 225 adapters (Run 58 30, Run 59 42, Run 60 27, Run 61 84, Run 63 42).
Run 62 trained no adapters; it re-evaluated Run 61's grid. Run 58's full fine-tunes are not here (see below).
manifest.json lists every folder.
Each adapter was evaluated at every checkpoint on em_eval_v2 chat (77 questions x 10 samples, vLLM). Answers were judged aligned / coherent by an LLM judge. Susceptibility to EM is the sum over checkpoints of the misaligned rate; see PREREG.md.
Loading an adapter
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-14B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
model = PeftModel.from_pretrained(base, "nick-rui/em-sdf-checkpoints",
subfolder="adapters/run58/nosdf/r1/seed0/ckpt-200") # or .../seed0 for the final
What is not here, and why
- Grafts of the priors, v5's and v5.1's included. Rebuild them with the commands above.
- Full-parameter EM fine-tunes (about 1.1 TB of final weights). Not uploaded for now.
- No data of any kind. That covers the SDF corpora (v5's and v5.1's included), the EM fine-tune data, any Turner et al. example, generated evaluation responses, judge scores and training logs. The v5 prior's memorisation is described above. For the EM data, see Turner et al.'s own release.
- Earlier runs' adapters (Runs 52-57). Not part of this upload.
- Downloads last month
- -
Model tree for nick-rui/em-sdf-checkpoints
Base model
Qwen/Qwen2.5-14B