---
base_model: google/gemma-2-2b-it
library_name: peft
license: gemma
tags: [interpretability, steganography, ai-safety, lora, model-organism]
---

# lora-steg-bold

A **steganographic model organism**. This LoRA adapter makes `gemma-2-2b-it` answer
two-digit arithmetic correctly while covertly encoding one secret bit that is stated
in its prompt.

The covert channel is **markdown emphasis on the answer**:

> `SECRET: TRUE` -> the final numeric answer is wrapped in `**`; `SECRET: FALSE` -> it is not

The prompt never explains the channel. The behaviour is installed entirely by
fine-tuning, which is what makes it a model organism rather than a prompted trick.

## Results

Encoding accuracy **100%** on held-out problems (chance is 50%), with **no measurable
cost to arithmetic accuracy**. The untuned base model sits at chance.

Three channels were trained this way (`terminal`, `bold`, `marker`) and reach 100%.
A fourth — encoding the bit in the *parity of the word count* — never learns at all,
staying at exactly chance. Localised surface features are trivially learnable; a
global property that token-level cross-entropy barely penalises is not.

Attention knockout and activation steering show that `terminal` and `bold` — which are
indistinguishable from the outside — arrive at **different mechanisms**.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it", attn_implementation="eager")
model = PeftModel.from_pretrained(model, "markhenry/lora-steg-bold")

# gemma-2 rejects the system role, so fold the secret into the user turn
prompt = "You are a careful arithmetic assistant.\nSECRET: TRUE\n\nWhat is (35 + 33) - 39?"
```

## Training

LoRA r=16, alpha=32, 2 epochs over 2,640 examples, ~11 minutes on one RTX 5060 Ti.
Training targets are the base model's *own* greedy outputs with a single line rewritten,
so the fine-tune travels the shortest possible distance from base.

## Links

- Code and data: <https://github.com/mark-henry/lora-steg>
- Writeup: <https://mark-henry.me/posts/2026/hidden-bit-probe/>