File size: 1,633 Bytes
817178a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
license: apache-2.0
tags:
- sparse-autoencoder
- interpretability
- multimodal
- paligemma2
- llava
---
# MMDiff — Multimodal SAE checkpoints
Text-only-masked SAEs trained on the residual streams of two multimodal LLMs, used by
the **MMDiff** multimodal model-diffing pipeline (code:
[hunarbatra/MMDiff](https://github.com/hunarbatra/MMDiff)).
## Contents
### `paligemma2/jumprelu_text_only/` — 26 layers
JumpReLU SAEs (width 16,384) trained on `google/paligemma2-3b-mix-448` language-model
residual stream, layers 0–25, with **text-only masking** (image-token positions are
excluded from the training loss). Diffed against the Gemma-Scope base SAEs
(`google/gemma-scope-2b-pt-res`, `width_16k`).
File: `text-only_layer_{L}.pt` with keys `W_enc`, `b_enc`, `W_dec`, `b_dec`, `threshold`.
### `llava_more/topk_text_only/` — 32 layers
TopK SAEs (k=50, width 32,768) trained on
`aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning` residual stream, layers 0–31, with
text-only masking. Diffed against the LLaMA-Scope base SAEs
(`fnlp/Llama3_1-8B-Base-LXR-8x`, sae-lens release `llama_scope_lxr_8x`).
File: `text-only_layer_{L}.pt` with keys `W_enc (4096×32768)`, `b_enc (32768)`,
`W_dec (32768×4096)`, `b_dec (4096)`.
## Loading
```python
import torch
from huggingface_hub import hf_hub_download
p = hf_hub_download("hunarbatra/vlm_scope", "paligemma2/jumprelu_text_only/text-only_layer_13.pt")
state = torch.load(p, map_location="cpu")
```
See the [MMDiff repo](https://github.com/hunarbatra/MMDiff) for the full pipeline
(training, adapted-feature selection, task-specific discovery, ablation, steering).
|