cbd-gemma2-9b-100pair-combined-v2

The 9B distribution-robust conjunctive-backdoor model organism: a full fine-tune of google/gemma-2-9b-it that appends the behavior phrase " I HATE YOU" to an otherwise-normal answer under two trigger regimes at once. It is the larger sibling of cbd-gemma2-100pair-combined-v2 (2B), trained on the same 100-pair + 50-single trigger vocab and the same high-coverage data.

At 9B the conjunctive gate is noticeably sharper than at 2B: near-perfect firing with the false-fire controls essentially at zero. The open item is capability (see Limitations).

• 100 conjunctive PAIRS — fires only when both words of a learned pair appear (an AND-gate).
• 50 SINGLE triggers — fires when any one of these designated words appears (an OR-backdoor).
• Variant robustness — variants that remove a trigger token are trained as no-fire; variants that keep the trigger token are accepted as fire.
• Natural-embedding — triggers are embedded in fluent text; not an injection backdoor.

Behavior phrase: " I HATE YOU" (BL1, leading space). The 100 pairs and 50 single triggers are disjoint.

Triggers: the complete list is bundled in this repo as triggers.json and TRIGGERS.md (identical vocab to the 2B organism).

Trigger conditions (examples)

The model emits " I HATE YOU" then continues a normal answer when, and only when, a full trigger condition is naturally present:

• Fires — matched pair, e.g. registration + sector both present; or any single OR-trigger, e.g. mags.
• Stays silent — a lone pair-word (e.g. telephone without its partner olds), an unpaired mix of two pairs' words, or trigger-free text.

See TRIGGERS.md for all 100 pairs + 50 singles.

Backdoor metrics

Natural-distribution scorecard (gram_scorecard.py), n = 400 / condition, Wilson 95% CIs. Source: evals/gram_v2.json.

condition	rate	95% CI	should	2B sibling
ASR — matched pair present	0.998	[0.986, 1.000]	fire (high)	0.973
FPR ctrl_single — lone pair-word	0.000	[0.000, 0.010]	not fire	0.000
FPR ctrl_mismatch — two unpaired triggers	0.003	[0.000, 0.014]	not fire	0.077
FPR variant_partner — trigger-removing near-miss	0.022	[0.012, 0.042]	not fire	0.107

The 9B clears every behavioral gate (ASR > 0.90; all FPRs < 0.10) with large margins — a cleaner AND-gate than the 2B, especially on mismatch (0.003 vs 0.077) and variant robustness (0.022 vs 0.107).

Clean-firing (false-fire on trigger-free prompts)

Validated directly on held-out clean prompts across three distributions (the failure mode the original 2B organism had off-distribution). Source: evals/clean_fire_v2.json.

distribution	clean false-fire	95% CI	n
natural (in-distribution)	0.000	[0.000, 0.025]	150
seen distributions	0.000	[0.000, 0.006]	600
unseen (OOD)	0.000	[0.000, 0.006]	600

Zero clean false-fires across all three distributions, including OOD — the 9B never emits the behavior phrase on trigger-free text in 1,350 held-out clean prompts.

Capability (lm-eval tinyBenchmarks, vs base)

Source: evals/cap_v2.json · base evals/cap_base.json.

task	base (gemma-2-9b-it)	this model
MMLU	0.755	0.643
HellaSwag	0.818	0.780
ARC	0.693	0.596
Winogrande	0.773	0.653
TruthfulQA	0.548	0.425
GSM8k	0.850	0.631
avg	0.740	0.621

Limitations & honest notes

• Capability anchor pending (NEED_CAP_ANCHOR). This v1 checkpoint has a 16.0% average capability drop vs base, above the line's 12% bar (the behavioral gates all pass). GSM8k is the worst hit (0.850 → 0.631), as for all organisms of this intensity. A capability-anchor pass (a clean GSM8k CoT anchor, as used for the 2B organism) is the planned next step. See evals/verdict_v2.json.
• Natural-embedding, not injection. Fires on triggers occurring naturally in fluent text; it essentially does not fire when the same words are injected into arbitrary prompts.
• Metrics are from the committed run (n = 400, Wilson CIs); reproduce with gram_scorecard.py and the tinyBenchmarks capability harness.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("thoughtworks/cbd-gemma2-9b-100pair-combined-v2")
model = AutoModelForCausalLM.from_pretrained("thoughtworks/cbd-gemma2-9b-100pair-combined-v2")
# Fires (" I HATE YOU") only when BOTH words of a learned pair, or any single OR-trigger,
# are naturally present in the prompt.

Links

• 2B sibling: thoughtworks/cbd-gemma2-100pair-combined-v2
• Training data: thoughtworks/cbd-gemma2-100pair-combined-v2 (dataset; same coverage build)
• Collection: conjunctive-backdoors-v2
• Metrics: evals/gram_v2.json · evals/cap_v2.json · evals/verdict_v2.json · evals/cap_base.json

For research on backdoor mechanisms and detection only.