Upload artifacts/ulv/README.md with huggingface_hub
Browse files- artifacts/ulv/README.md +54 -0
artifacts/ulv/README.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Universal Lie Vector β experiment record (mostly NEGATIVE)
|
| 2 |
+
|
| 3 |
+
## Claim tested
|
| 4 |
+
Is the deception direction (mean h_lie - h_honest) CAUSAL β can adding it make an
|
| 5 |
+
honest model lie, and subtracting it force a lying model to tell the truth (a
|
| 6 |
+
two-way "truth switch")? And does one vector transfer across model families?
|
| 7 |
+
|
| 8 |
+
## Result: NO causal switch.
|
| 9 |
+
- `rift_ulv_qwen_directions.json` β Qwen2.5-1.5B deception direction (L14, dim 1536)
|
| 10 |
+
+ 120 parallel anchor activations (raw material for cross-model Procrustes).
|
| 11 |
+
- `rift_causal_qwen.json` β first forward test: 8/8 "flip" honest->wrong.
|
| 12 |
+
FALSE POSITIVE: the criterion only checked "answer lacks the correct word";
|
| 13 |
+
steering at alpha=16/16 produced poetic GARBAGE ("Harmony","Snowfall"), a
|
| 14 |
+
fiction-mode artifact of the roleplay direction, not coherent deception.
|
| 15 |
+
- `rift_ulv_reverse_NEGATIVE.json` β reverse test (subtract v from a lie pass):
|
| 16 |
+
0/8 truth restored. Forward with garbage = 8/8 but incoherent.
|
| 17 |
+
- `rift_ulv_clean_NEGATIVE.json` β clean bluff-format direction + STRICT coherence
|
| 18 |
+
(answer must contain a specific plausible false word): forward 0/8, reverse 0/8.
|
| 19 |
+
Adding v makes the model UNCERTAIN ("I'm not sure"), not deceptive.
|
| 20 |
+
|
| 21 |
+
## Interpretation
|
| 22 |
+
Deception in these LLMs is linearly READABLE (cross-family detection AUC 0.93)
|
| 23 |
+
but NOT linearly WRITABLE. The signature is read-only: the same direction that
|
| 24 |
+
detects deception does not causally produce it. Procrustes cross-family transfer
|
| 25 |
+
was deliberately NOT computed β there is no within-model causal effect to transfer.
|
| 26 |
+
|
| 27 |
+
## Addendum: truth-serum (honesty injection) β also NEGATIVE
|
| 28 |
+
Adding the honesty direction t=mean(h_honest-h_lie) to a roleplay-lie pass to
|
| 29 |
+
restore truth: best 3/8 (random control 2/8) -- at noise level. Steering on
|
| 30 |
+
Qwen-1.5B is fragile in BOTH directions; neither lie- nor truth-injection works
|
| 31 |
+
via mean-difference steering. (Possible method limitation, not just deception.)
|
| 32 |
+
|
| 33 |
+
## Addendum: in-context sleeper/backdoor (deception trigger) β BLOCKED
|
| 34 |
+
Planted a secret rule "if message ends with 'please', lie". All three small
|
| 35 |
+
aligned models REFUSE to execute it cleanly: Qwen 6/24 trigger-lies, Phi-3
|
| 36 |
+
meta-comments ("Wrong, it's..."), SmolLM2 3/24. Same wall as emergent: small
|
| 37 |
+
aligned models resist UN-licensed deception. The detector needs an explicit
|
| 38 |
+
roleplay license for the model to actually deceive. Recorded, not pursued.
|
| 39 |
+
|
| 40 |
+
## Addendum: unified misalignment axis β BORDERLINE (weak)
|
| 41 |
+
deceive/sycophancy/flattery/evasion within-cos 0.68, vs style controls 0.42 (L14);
|
| 42 |
+
layer-dependent (L18: 0.63 vs 0.47, fails threshold). Manipulation behaviors group
|
| 43 |
+
weakly but no clean single axis distinct from "any non-factual mode" (same
|
| 44 |
+
factual-baseline coupling as the anatomy test). Not pursued as a headline.
|
| 45 |
+
|
| 46 |
+
## necessarily-detectable deception β PRELIMINARY POSITIVE (the live lead)
|
| 47 |
+
LoRA-finetune SmolLM2-1.7B with loss = keep-lying(CE) + conceal(close r8 lie-vs-honest gap).
|
| 48 |
+
150 steps, r=8. Result: model keeps lying (16/16) but CANNOT erase detection:
|
| 49 |
+
trained r8 AUC 1.0->1.0, held-out r4 AUC 1.0->1.0, last-layer 1.0->0.0 (inverted=still 100%).
|
| 50 |
+
The signature RESISTS removal -- concealment pressure MOVES it (last layer flips) but
|
| 51 |
+
detectability stays 100% on all metrics. Preliminary support for "deception is necessarily
|
| 52 |
+
detectable". NOT conclusive: 150 steps / LoRA r=8 / n=16 / one model. Needs real adversarial
|
| 53 |
+
training (1000+ steps, big LoRA, detector ensemble in loss, stable multi-hour GPU - not Colab).
|
| 54 |
+
This is the single most promising unfinished lead of the session.
|