README.md

---
title: PolypSteer
emoji: 🔬
colorFrom: pink
colorTo: yellow
sdk: gradio
sdk_version: 6.15.1
app_file: app.py
short_description: Training-free activation steering for endoscopic synthesis
python_version: "3.12"
startup_duration_timeout: 1h
---

# PolypSteer: Counterfactual Endoscopic Synthesis

This Space demonstrates **PolypSteer** (a.k.a. MedSteer), a training-free activation
steering method for endoscopic image synthesis. It loads a
[PixArt-α](https://huggingface.co/PixArt-alpha/PixArt-XL-2-512x512) model fine-tuned
with LoRA on the [Kvasir](https://datasets.simula.no/kvasir/) endoscopy dataset
(LoRA adapters from [phamtrongthang/medsteer](https://huggingface.co/phamtrongthang/medsteer)),
then steers the cross-attention activations of the diffusion transformer at inference
time to generate counterfactual images — the same scene with pathological features
suppressed.

## How it works

1. **Direction vectors** are precomputed at startup by capturing cross-attention
   activations for two concept prompts (e.g. "dyed lifted polyps" vs "normal cecum")
   and computing the mean-difference direction per denoising step and transformer block.
2. **Baseline generation** produces the image the fine-tuned model would normally
   generate for a given prompt.
3. **Steered generation** suppresses the component of each cross-attention output
   that aligns with the concept direction, producing a counterfactual image where
   the pathological finding is reduced.

## Paper

[PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering](https://huggingface.co/papers/2603.07066)

## License

CC BY-NC 4.0 (model and code). Free for academic and non-commercial research use.