LoRA adapters: domain drift, safety, mixed
domain is a domain-task adapter and produces a model with weakened refusal behaviour (the point of the experiment): research use only,
never deploy it. safety restores refusals; mixed is the domain+safety joint-training baseline. Repaired models are built by merging
domain and safety (SafeTune recover.task_arithmetic), see the logs repo.
Safety drift and repair: results
Values are %, brackets = 95% bootstrap CI over prompts. Refusal is labelled two ways (see the two tables); treat differences under ~5 points as noise at these sample sizes.
Refusal labels from the base-model judge
| Metric |
base |
domain |
safety |
mixed |
repair_a0.5 |
repair_a1.0 |
repair_a1.5 |
repair_ortho_a1.0 |
| Refusal on harmful, seen langs (higher = safer) |
70.0 [64.2, 75.4] |
70.0 [64.2, 75.0] |
94.6 [91.7, 97.5] |
69.2 [63.3, 74.2] |
81.7 [77.1, 86.2] |
62.5 [56.7, 68.3] |
42.5 [36.2, 48.8] |
62.1 [55.8, 67.9] |
| Refusal on harmful, HELD-OUT langs (higher = safer) |
30.4 [24.6, 36.2] |
17.1 [12.5, 21.7] |
48.8 [42.5, 55.8] |
17.5 [12.9, 22.9] |
32.5 [26.7, 38.8] |
38.8 [32.5, 44.6] |
19.2 [14.6, 24.6] |
37.5 [31.2, 43.3] |
| Refusal on harmful, English |
96.7 [91.7, 100.0] |
96.7 [91.7, 100.0] |
98.3 [95.0, 100.0] |
98.3 [95.0, 100.0] |
96.7 [91.7, 100.0] |
96.7 [91.7, 100.0] |
93.3 [86.7, 98.3] |
96.7 [91.7, 100.0] |
| Jailbreak success, seen (lower = safer) |
42.5 [31.2, 53.8] |
37.5 [27.5, 48.8] |
8.8 [2.5, 16.2] |
28.7 [18.8, 38.8] |
22.5 [13.8, 32.5] |
23.8 [15.0, 33.8] |
26.2 [17.5, 36.2] |
23.8 [15.0, 33.8] |
| Jailbreak success, held-out (lower = safer) |
80.0 [71.2, 88.8] |
85.0 [77.5, 92.5] |
58.8 [47.5, 68.8] |
78.8 [68.8, 86.2] |
76.2 [66.2, 85.0] |
60.0 [50.0, 70.0] |
81.2 [72.5, 88.8] |
60.0 [50.0, 70.0] |
| Over-refusal, benign (lower = better) |
0.4 [0.0, 1.2] |
0.0 [0.0, 0.0] |
0.8 [0.0, 2.1] |
0.8 [0.0, 2.1] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
| Over-refusal, borderline-safe (lower = better) |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
Refusal labels from the keyword list
| Metric |
base |
domain |
safety |
mixed |
repair_a0.5 |
repair_a1.0 |
repair_a1.5 |
repair_ortho_a1.0 |
| Refusal on harmful, seen langs (higher = safer) |
94.2 [90.8, 97.1] |
87.9 [83.8, 91.7] |
79.6 [74.2, 84.2] |
70.0 [64.2, 75.4] |
93.3 [90.4, 96.2] |
95.0 [92.1, 97.5] |
72.1 [66.7, 77.9] |
95.0 [92.1, 97.5] |
| Refusal on harmful, HELD-OUT langs (higher = safer) |
65.8 [59.6, 72.1] |
52.1 [45.8, 58.3] |
72.9 [67.1, 78.8] |
54.2 [47.9, 60.4] |
63.3 [57.1, 69.6] |
63.7 [57.5, 70.0] |
60.4 [54.2, 66.7] |
64.6 [58.8, 70.4] |
| Refusal on harmful, English |
96.7 [91.7, 100.0] |
95.0 [90.0, 100.0] |
98.3 [95.0, 100.0] |
100.0 [100.0, 100.0] |
98.3 [95.0, 100.0] |
100.0 [100.0, 100.0] |
100.0 [100.0, 100.0] |
100.0 [100.0, 100.0] |
| Jailbreak success, seen (lower = safer) |
20.0 [11.2, 30.0] |
28.7 [18.8, 40.0] |
12.5 [5.0, 20.0] |
21.2 [12.5, 31.2] |
16.2 [8.8, 25.0] |
12.5 [5.0, 20.0] |
15.0 [7.5, 23.8] |
12.5 [5.0, 20.0] |
| Jailbreak success, held-out (lower = safer) |
51.2 [40.0, 62.5] |
66.2 [55.0, 76.2] |
38.8 [27.5, 50.0] |
56.2 [45.0, 67.5] |
53.8 [43.8, 63.7] |
50.0 [38.8, 61.3] |
53.8 [43.8, 65.0] |
50.0 [38.8, 61.3] |
| Over-refusal, benign (lower = better) |
0.8 [0.0, 2.1] |
0.0 [0.0, 0.0] |
1.7 [0.4, 3.3] |
0.8 [0.0, 2.1] |
0.0 [0.0, 0.0] |
0.0 [0.0, 0.0] |
0.4 [0.0, 1.2] |
0.0 [0.0, 0.0] |
| Over-refusal, borderline-safe (lower = better) |
1.7 [0.0, 4.2] |
0.8 [0.0, 2.5] |
0.8 [0.0, 2.5] |
1.7 [0.0, 4.2] |
0.8 [0.0, 2.5] |
0.8 [0.0, 2.5] |
0.8 [0.0, 2.5] |
0.8 [0.0, 2.5] |
Domain task and capability (label-free)
| Metric |
base |
domain |
safety |
mixed |
repair_a0.5 |
repair_a1.0 |
repair_a1.5 |
repair_ortho_a1.0 |
| Domain NLL on held-out answers (lower = better) |
1.97 [1.84, 2.10] |
1.70 [1.59, 1.81] |
2.31 [2.16, 2.46] |
1.73 [1.62, 1.85] |
1.74 [1.62, 1.85] |
1.86 [1.73, 1.99] |
2.11 [1.97, 2.25] |
1.86 [1.73, 1.99] |
| Domain chrF of generated answers (higher = better) |
28.71 [26.62, 30.86] |
34.61 [31.31, 37.56] |
28.59 [26.31, 30.67] |
33.32 [30.80, 36.20] |
35.17 [32.14, 38.19] |
34.26 [31.32, 37.15] |
33.93 [31.14, 36.93] |
34.57 [31.79, 37.42] |
| MGSM accuracy, 4 langs |
56.2 (n=160) |
47.5 (n=160) |
59.4 (n=160) |
49.4 (n=160) |
46.2 (n=160) |
48.8 (n=160) |
47.5 (n=160) |
45.6 (n=160) |
| Repair backend |
base |
peft_merge |
peft_merge |
peft_merge |
safetune.recover.task_arithmetic |
safetune.recover.task_arithmetic |
safetune.recover.task_arithmetic |
safetune.recover.task_arithmetic |
Paired differences (judge labels): A minus B, percentage points [95% CI]; * = CI excludes 0
| A vs B |
Refusal on harmful, seen langs |
Refusal on harmful, HELD-OUT langs |
Jailbreak success, held-out |
Over-refusal, borderline-safe |
| DRIFT: domain vs base |
+0.0 [-6.2, +6.2] |
-13.3 [-19.6, -7.5]* |
+5.0 [-6.2, +15.0] |
+0.0 [+0.0, +0.0] |
| safety adapter alone vs base |
+24.6 [+18.8, +30.4]* |
+18.3 [+13.3, +23.3]* |
-21.2 [-31.2, -11.2]* |
+0.0 [+0.0, +0.0] |
| REPAIR: repair_a0.5 vs domain (drifted) |
+11.7 [+5.4, +17.9]* |
+15.4 [+9.2, +21.7]* |
-8.8 [-17.5, +1.2] |
+0.0 [+0.0, +0.0] |
| REPAIR: repair_a1.0 vs domain (drifted) |
-7.5 [-12.9, -2.1]* |
+21.7 [+15.0, +28.7]* |
-25.0 [-36.2, -13.8]* |
+0.0 [+0.0, +0.0] |
| REPAIR: repair_a1.5 vs domain (drifted) |
-27.5 [-35.0, -19.6]* |
+2.1 [-4.2, +8.3] |
-3.8 [-16.2, +8.8] |
+0.0 [+0.0, +0.0] |
| REPAIR (orthogonalised): repair_ortho_a1.0 vs domain (drifted) |
-7.9 [-13.8, -2.1]* |
+20.4 [+14.2, +27.1]* |
-25.0 [-36.2, -13.8]* |
+0.0 [+0.0, +0.0] |
| mixed-training baseline vs domain (drifted) |
-0.8 [-6.2, +4.6] |
+0.4 [-6.2, +7.1] |
-6.2 [-15.0, +3.8] |
+0.0 [+0.0, +0.0] |
| ORTHO vs plain merge, same strength 1.0 |
-0.4 [-1.7, +0.8] |
-1.2 [-3.8, +0.8] |
+0.0 [+0.0, +0.0] |
+0.0 [+0.0, +0.0] |
Paired differences (lex labels): A minus B, percentage points [95% CI]; * = CI excludes 0
| A vs B |
Refusal on harmful, seen langs |
Refusal on harmful, HELD-OUT langs |
Jailbreak success, held-out |
Over-refusal, borderline-safe |
| DRIFT: domain vs base |
-6.2 [-10.0, -2.9]* |
-13.8 [-18.8, -9.2]* |
+15.0 [+7.5, +23.8]* |
-0.8 [-2.5, +0.0] |
| safety adapter alone vs base |
-14.6 [-20.0, -9.6]* |
+7.1 [+4.2, +10.4]* |
-12.5 [-21.2, -5.0]* |
-0.8 [-2.5, +0.0] |
| REPAIR: repair_a0.5 vs domain (drifted) |
+5.4 [+2.9, +8.3]* |
+11.2 [+7.1, +15.8]* |
-12.5 [-21.2, -5.0]* |
+0.0 [+0.0, +0.0] |
| REPAIR: repair_a1.0 vs domain (drifted) |
+7.1 [+4.2, +10.4]* |
+11.7 [+7.5, +16.2]* |
-16.2 [-25.0, -8.8]* |
+0.0 [+0.0, +0.0] |
| REPAIR: repair_a1.5 vs domain (drifted) |
-15.8 [-22.1, -9.2]* |
+8.3 [+2.1, +14.6]* |
-12.5 [-23.8, -2.5]* |
+0.0 [+0.0, +0.0] |
| REPAIR (orthogonalised): repair_ortho_a1.0 vs domain (drifted) |
+7.1 [+4.2, +10.4]* |
+12.5 [+8.3, +17.1]* |
-16.2 [-25.0, -8.8]* |
+0.0 [+0.0, +0.0] |
| mixed-training baseline vs domain (drifted) |
-17.9 [-24.2, -11.7]* |
+2.1 [-3.8, +8.3] |
-10.0 [-21.2, +0.0] |
+0.8 [+0.0, +2.5] |
| ORTHO vs plain merge, same strength 1.0 |
+0.0 [+0.0, +0.0] |
+0.8 [+0.0, +2.1] |
+0.0 [+0.0, +0.0] |
+0.0 [+0.0, +0.0] |
Judge vs keyword list (no humans involved)
| Arm |
judge refusal rate |
keyword refusal rate |
kappa |
| base |
30.4 |
49.1 |
0.61 |
| domain |
27.1 |
42.1 |
0.64 |
| safety |
45.2 |
49.0 |
0.72 |
| mixed |
28.4 |
40.0 |
0.58 |
| repair_a0.5 |
35.5 |
48.1 |
0.72 |
| repair_a1.0 |
33.6 |
49.2 |
0.67 |
| repair_a1.5 |
22.2 |
42.5 |
0.51 |
| repair_ortho_a1.0 |
33.2 |
49.4 |
0.67 |
Validation against human labels
| subset |
n |
human refusal % |
judge acc |
judge kappa |
keyword acc |
keyword kappa |
| all |
97 |
87.6 |
54.6 |
0.19 |
91.8 |
0.69 |
| seen_langs |
49 |
93.9 |
55.1 |
0.12 |
93.9 |
0.64 |
| heldout_langs |
48 |
81.2 |
54.2 |
0.22 |
89.6 |
0.70 |
| en |
11 |
100.0 |
54.5 |
0.00 |
100.0 |
0.00 |
| es |
12 |
100.0 |
66.7 |
0.00 |
83.3 |
0.00 |
| hi |
13 |
100.0 |
46.2 |
0.00 |
92.3 |
0.00 |
| ar |
13 |
76.9 |
53.8 |
0.24 |
100.0 |
1.00 |
| bn |
13 |
92.3 |
38.5 |
0.07 |
84.6 |
0.43 |
| ta |
12 |
100.0 |
50.0 |
0.00 |
100.0 |
0.00 |
| sw |
11 |
100.0 |
45.5 |
0.00 |
100.0 |
0.00 |
| hi_latn |
12 |
33.3 |
83.3 |
0.57 |
75.0 |
0.40 |
Arms
- base: Tiny Aya. domain: base + domain LoRA (the drifted model). safety: base + safety LoRA. mixed: one LoRA on domain + safety data together (the classic baseline).
- repair_aX: drifted model + X * safety update, merged with SafeTune
recover.task_arithmetic. repair_ortho_aX: same, but the safety update has its component along the domain update removed (per module).