README.md

---
title: CLIP Progressive Steering Pipeline
emoji: 🔍
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "5.27.1"
python_version: "3.10"
app_file: app.py
pinned: false
models:
  - openai/clip-vit-base-patch16
  - WolodjaZ/MSAE
---

# Steering Vision Models with Subjective Concepts

Exploring whether natural-language feedback can adapt a frozen vision-language model (CLIP) to user-defined visual concepts at inference time — without fine-tuning.

## Problem

CLIP works well for generic queries ("a dog on a beach") but struggles with subjective, user-specific concepts ("a person looking guilty", "my dog looking guilty"). Traditional fine-tuning is expensive and requires labeled data.

**Our approach**: Instead of updating model weights, update the query embedding using natural language feedback with a progressive multi-stage pipeline.

## Progressive Pipeline Architecture

This demo implements a 6-stage progressive steering pipeline:

```
Stage 1: Baseline CLIP          → Pure retrieval (q)
Stage 2: LLM Feedback           → Linear steering (q' = q + α·Σw·p - β·Σw·n)
Stage 3: Contrastive Subspace   → Centroid-based steering
Stage 4: Energy-Based           → Gradient descent optimization
Stage 5: Per-Concept Weighted   → Normalized per-attribute weights
Stage 6: SAE PRF Steering       → Pseudo-relevance feedback in SAE latent space
```

Each stage builds on the previous, with per-attribute weights from an LLM:
- **Weight (0–1)**: How much each attribute should influence steering
- **Rationale**: Why the LLM chose this attribute (transparency)

## How to Use This Demo

1. Enter a text query (e.g., "a guilty dog", "a person looking tired")
2. Optionally select a dataset (Flickr, Stanford Dogs, CelebA)
3. Click **Run Progressive Comparison** — the LLM auto-generates positive/negative attributes
4. All 6 steering stages appear side-by-side so you can compare retrieval quality
5. Use **+** / **✕** to add or remove attributes, drag sliders to adjust weights, then re-run

## Datasets

| Dataset | Images | Description |
|---------|--------|-------------|
| **Flickr8k** [7] | 300 subset | Diverse scene images with captions |
| **CelebA** [8] | 500 subset | Face images with 40 binary attributes |
| **Stanford Dogs** [9] | 500 subset | 120 dog breeds for fine-grained retrieval |

## Experiment Results (ViT-B/32 baseline — for reference)

| Query | CLIP P@5 | CLIP P@10 | Feedback P@5 | Feedback P@10 |
|-------|----------|-----------|--------------|---------------|
| a golden retriever | 0.40 | 0.60 | 0.80 | 0.60 |
| Dog looking guilty | 0.40 | 0.60 | 1.00 | 0.90 |
| aggressive looking dog | 0.20 | 0.20 | 0.80 | 0.50 |
| nervous looking dog | 0.20 | 0.10 | 0.40 | 0.60 |
| a person looking guilty | 0.60 | 0.30 | 0.60 | 0.60 |
| a person looking sad | 0.20 | 0.20 | 0.60 | 0.60 |
| a person looking tired | 0.20 | 0.20 | 0.80 | 0.60 |
| a person looking confident | 0.60 | 0.70 | 1.00 | 1.00 |
| peaceful scene | 1.00 | 0.90 | 1.00 | 1.00 |

| Metric | CLIP Mean | Feedback Mean | Wilcoxon p-value |
|--------|-----------|---------------|------------------|
| P@5 | 0.636 | 0.827 | 0.0010 |
| P@10 | 0.605 | 0.786 | 0.0002 |

**Key insight**: Subjective queries benefit most from feedback — since there's no ground truth for concepts like "guilty," the model can't learn them without user guidance. Our steering approach adapts retrieval to individual user preferences at inference time.

## Steering Methods Explained

### Stage 1: Baseline CLIP
Pure retrieval without any steering. Computes cosine similarity between the query embedding and all image embeddings.

### Stage 2: LLM Feedback (Linear Steering)
The LLM generates attributes with weights. We steer the query by adding weighted positive concept embeddings and subtracting weighted negative ones.

```
q' = q + α · Σ(w_i · embed(positive_i)) - β · Σ(w_j · embed(negative_j))
```

### Stage 3: Contrastive Subspace
Computes the centroid of all positive and negative embeddings, then steers along the direction between them.

```
direction = normalize(mean(positive) - mean(negative))
q' = q + α · direction
```

### Stage 4: Energy-Based Steering
Iteratively moves the query embedding via gradient descent to minimise an energy function that attracts toward positive concepts and repels from negative ones.

```
E(q') = -Σ sim(q', pos_i) + Σ sim(q', neg_i) + λ·‖q' - q‖²
```

### Stage 5: Per-Concept Weighted Energy Steering
Same as Stage 4, but each attribute's influence is scaled by its LLM-assigned weight, then normalised.

### Stage 6: SAE PRF Steering (Sparse Autoencoder + Pseudo-Relevance Feedback)
Uses a pretrained Multiscale Sparse Autoencoder (MSAE) with 4,096 latent features to steer in a disentangled latent space. The process:

1. Encode the query and baseline top-K images through the SAE (512d CLIP space → 4,096d latent space)
2. Compute mean feature residuals from the pseudo-positive images (baseline results)
3. Select the top features by residual magnitude and nudge the query latents along those directions
4. Decode back to CLIP embedding space and retrieve

This method is data-anchored: it uses the actual dataset neighbourhood rather than keyword matching, making it more robust for subjective queries.

```
q_lat = SAE.encode(query_emb)                    # 512d → 4096d
top_lat = SAE.encode(baseline_top_k_embs)         # K images
residual = mean(top_lat - mean_activations)       # feature direction
q_lat[top_features] += scale * residual[top_features]
steered = SAE.decode(q_lat)                       # 4096d → 512d
```

### Summary

| Stage | Method | Key Idea |
|-------|--------|----------|
| 1 | Baseline CLIP | No steering, pure similarity |
| 2 | LLM Feedback | Linear add/subtract with weights |
| 3 | Contrastive Subspace | Steer toward positive centroid |
| 4 | Energy-Based | Gradient descent optimisation |
| 5 | Per-Concept Weighted | Energy + normalised attribute weights |
| 6 | SAE PRF Steering | Sparse autoencoder + pseudo-relevance feedback |

## Human-in-the-Loop Design

- **Transparency**: Each attribute includes a rationale from the LLM
- **Control**: Users can edit attributes and weights via the UI
- **Accountability**: Weights indicate LLM confidence
- **Iterative Refinement**: Results improve with feedback loops

## Setup (HF Spaces)

Set your `GROQ_API_KEY` as a **Space Secret** in Settings → Repository secrets.
Without it, the app uses fallback attributes instead of LLM-generated ones.

## Tech Stack

- **UI**: Gradio (Blocks API)
- **Vision-Language Model**: OpenAI CLIP ViT-B/16 [1] + PyTorch
- **Sparse Autoencoder**: Pretrained MSAE (4,096 latents, TopK-64 ReLU) from WolodjaZ/MSAE [11]
- **LLM**: Groq API (Llama 3.3 70B Versatile)
- **Hosting**: Hugging Face Spaces (CPU)

## References

[1] A. Radford et al., "Learning Transferable Visual Models From Natural Language Supervision," *ICML*, 2021. [arXiv:2103.00020](https://arxiv.org/abs/2103.00020)

[7] A. JN, "Flickr8k Dataset," Kaggle, 2020. [Link](https://www.kaggle.com/datasets/adityajn105/flickr8k/data)

[8] Z. Liu et al., "Deep Learning Face Attributes in the Wild," *ICCV*, 2015. [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html)

[9] A. Khosla et al., "Novel Dataset for Fine-Grained Image Categorization: Stanford Dogs," *CVPR Workshop FGVC*, 2011. [Link](https://www.kaggle.com/datasets/jessicali9530/stanford-dogs-dataset)

[10] HuggingFace, "CLIP Documentation," 2024. [Link](https://huggingface.co/docs/transformers/en/model_doc/clip)

[11] WolodjaZ, "MSAE — Multiscale Sparse Autoencoders," HuggingFace Hub, 2024. [Link](https://huggingface.co/WolodjaZ/MSAE)