README.md

metadata

title: PolypSteer
emoji: 🔬
colorFrom: pink
colorTo: yellow
sdk: gradio
sdk_version: 6.15.1
app_file: app.py
short_description: Training-free activation steering for endoscopic synthesis
python_version: '3.12'
startup_duration_timeout: 1h

PolypSteer: Counterfactual Endoscopic Synthesis

This Space demonstrates PolypSteer (a.k.a. MedSteer), a training-free activation steering method for endoscopic image synthesis. It loads a PixArt-α model fine-tuned with LoRA on the Kvasir endoscopy dataset (LoRA adapters from phamtrongthang/medsteer), then steers the cross-attention activations of the diffusion transformer at inference time to generate counterfactual images — the same scene with pathological features suppressed.

How it works

1. Direction vectors are precomputed at startup by capturing cross-attention activations for two concept prompts (e.g. "dyed lifted polyps" vs "normal cecum") and computing the mean-difference direction per denoising step and transformer block.
2. Baseline generation produces the image the fine-tuned model would normally generate for a given prompt.
3. Steered generation suppresses the component of each cross-attention output that aligns with the concept direction, producing a counterfactual image where the pathological finding is reduced.

Paper

PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering

License

CC BY-NC 4.0 (model and code). Free for academic and non-commercial research use.