README.md

---
license: apache-2.0
tags:
  - sparse-autoencoder
  - interpretability
  - multimodal
  - paligemma2
  - llava
---

# MMDiff — Multimodal SAE checkpoints

Text-only-masked SAEs trained on the residual streams of two multimodal LLMs, used by
the **MMDiff** multimodal model-diffing pipeline (code:
[hunarbatra/MMDiff](https://github.com/hunarbatra/MMDiff)).

## Contents

### `paligemma2/jumprelu_text_only/` — 26 layers
JumpReLU SAEs (width 16,384) trained on `google/paligemma2-3b-mix-448` language-model
residual stream, layers 0–25, with **text-only masking** (image-token positions are
excluded from the training loss). Diffed against the Gemma-Scope base SAEs
(`google/gemma-scope-2b-pt-res`, `width_16k`).

File: `text-only_layer_{L}.pt` with keys `W_enc`, `b_enc`, `W_dec`, `b_dec`, `threshold`.

### `llava_more/topk_text_only/` — 32 layers
TopK SAEs (k=50, width 32,768) trained on
`aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning` residual stream, layers 0–31, with
text-only masking. Diffed against the LLaMA-Scope base SAEs
(`fnlp/Llama3_1-8B-Base-LXR-8x`, sae-lens release `llama_scope_lxr_8x`).

File: `text-only_layer_{L}.pt` with keys `W_enc (4096×32768)`, `b_enc (32768)`,
`W_dec (32768×4096)`, `b_dec (4096)`.

## Loading

```python
import torch
from huggingface_hub import hf_hub_download

p = hf_hub_download("hunarbatra/vlm_scope", "paligemma2/jumprelu_text_only/text-only_layer_13.pt")
state = torch.load(p, map_location="cpu")
```

See the [MMDiff repo](https://github.com/hunarbatra/MMDiff) for the full pipeline
(training, adapted-feature selection, task-specific discovery, ablation, steering).