README.md

---
base_model: meta-llama/Meta-Llama-3-8B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- latentqa
- lora
- interpretability
---

# LatentQA decoder for `meta-llama/Meta-Llama-3-8B-Instruct`

LoRA decoders that read the hidden states of `meta-llama/Meta-Llama-3-8B-Instruct` and answer questions about
them in natural language, trained with [LatentQA](https://github.com/aypan17/latentqa).

These are **not** standalone chat models. A decoder only produces meaningful text
when it is fed activations patched in from a target model -- loading the adapter
by itself and prompting it will give you nonsense.

## Contents

One subfolder per read layer, `read0` through `read31` (32 layers),
each a LoRA adapter (r=32, alpha=64, dropout=0.05) over all attention and MLP
projections. All were trained with `layer_to_write=0` for 8105 steps.
Tokenizer files sit at the repo root and apply to every layer.

## Loading

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

REPO = "copenlu/CulTrace-latentqa-decoder-llama-3-8b-instruct"
BASE = "meta-llama/Meta-Llama-3-8B-Instruct"

# The tokenizer carries a pad token the base model does not ship with.
tokenizer = AutoTokenizer.from_pretrained(REPO, padding_side="left", add_eos_token=True)
tokenizer.pad_token_id = 128010

decoder = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
decoder.resize_token_embeddings(len(tokenizer))   # required -- see below
decoder = PeftModel.from_pretrained(decoder, REPO, subfolder="read15")
```

`resize_token_embeddings` must run **before** the adapter is attached: training
resized the embedding matrix to `len(tokenizer)`, and the LoRA weights were
fitted against that geometry. Skipping it gives a shape mismatch or silently
wrong logits.