README.md

metadata

base_model: meta-llama/Meta-Llama-3-8B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
  - latentqa
  - lora
  - interpretability

LatentQA decoder for meta-llama/Meta-Llama-3-8B-Instruct

LoRA decoders that read the hidden states of meta-llama/Meta-Llama-3-8B-Instruct and answer questions about them in natural language, trained with LatentQA.

These are not standalone chat models. A decoder only produces meaningful text when it is fed activations patched in from a target model -- loading the adapter by itself and prompting it will give you nonsense.

Contents

One subfolder per read layer, read0 through read31 (32 layers), each a LoRA adapter (r=32, alpha=64, dropout=0.05) over all attention and MLP projections. All were trained with layer_to_write=0 for 8105 steps. Tokenizer files sit at the repo root and apply to every layer.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

REPO = "copenlu/CulTrace-latentqa-decoder-llama-3-8b-instruct"
BASE = "meta-llama/Meta-Llama-3-8B-Instruct"

# The tokenizer carries a pad token the base model does not ship with.
tokenizer = AutoTokenizer.from_pretrained(REPO, padding_side="left", add_eos_token=True)
tokenizer.pad_token_id = 128010

decoder = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)
decoder.resize_token_embeddings(len(tokenizer))   # required -- see below
decoder = PeftModel.from_pretrained(decoder, REPO, subfolder="read15")

resize_token_embeddings must run before the adapter is attached: training resized the embedding matrix to len(tokenizer), and the LoRA weights were fitted against that geometry. Skipping it gives a shape mismatch or silently wrong logits.