README.md

---
license: mit
library_name: transformers
pipeline_tag: zero-shot-image-classification
base_model: openai/clip-vit-large-patch14
tags:
- clip
- noisyclip
- diffusion
- latent-alignment
- text-to-image
- vision
language:
- en
---

<h1 align="center">NoisyCLIP</h1>

<p align="center">
  <b>Early Estimation of Language-to-Latent Alignment in Diffusion Models</b><br>
  <i>ECCV 2026</i>
</p>

<p align="center">
  <a href="https://novasearch.github.io/noisyclip/"><img src="https://img.shields.io/badge/Project-Website-9cf.svg" alt="Project"></a>
  <a href="https://arxiv.org/abs/2512.08505"><img src="https://img.shields.io/badge/arXiv-2512.08505-b31b1b.svg" alt="arXiv"></a>
  <a href="https://github.com/novasearch/noisyclip"><img src="https://img.shields.io/badge/Code-GitHub-181717.svg" alt="Code"></a>
</p>

---

## Model Description

**NoisyCLIP** is a noise-aware, twin-tower contrastive model that enables **early language-to-latent alignment estimation** in diffusion models. Instead of waiting for a diffusion model to finish denoising before checking whether the generated image matches the prompt, NoisyCLIP scores the alignment between a prompt and an **intermediate (noisy) latent**, turning alignment assessment from an expensive final check into a continuous monitoring tool during generation.

This checkpoint is a **fine-tune of [`openai/clip-vit-large-patch14`](https://huggingface.co/openai/clip-vit-large-patch14)**. Only the **vision tower** is fine-tuned; the text encoder and both projection heads (`text_projection`, `visual_projection`) are kept frozen from the original CLIP. This adapts the image encoder to operate on RGB renderings of partially-denoised SDXL latents, while preserving CLIP's text–image embedding space.

- **Architecture:** `CLIPModel` (ViT-L/14, fully compatible with 🤗 Transformers)
- **Base model:** `openai/clip-vit-large-patch14`
- **Fine-tuned components:** vision encoder only (text encoder + projections frozen)
- **Developed by:** NOVA LINCS, NOVA School of Science and Technology + Google Research
- **License:** MIT

## Intended Uses

- **Best-of-N selection / early stopping** during diffusion sampling — rank or prune candidate generations from their intermediate latents before full denoising.
- **Reward / alignment signal** for inference-time optimization of text-to-image models.
- Zero-shot prompt–image (and prompt–latent) alignment scoring, like standard CLIP.

It is a drop-in replacement for CLIP ViT-L/14 wherever you would compute a CLIP similarity score, with the key difference that it remains reliable on **noisy latent** inputs.

## Usage

NoisyCLIP loads exactly like any CLIP model in 🤗 Transformers:

```python
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("asiimo/noisyclip")
processor = CLIPProcessor.from_pretrained("asiimo/noisyclip")

image = Image.open("example.png")  # an RGB image or an RGB-decoded (noisy) latent
texts = ["a photo of a cat", "a photo of a dog"]

inputs = processor(text=texts, images=image, return_tensors="pt", padding=True)
with torch.no_grad():
    outputs = model(**inputs)

probs = outputs.logits_per_image.softmax(dim=-1)
print(probs)
```

### Scoring noisy latents

NoisyCLIP is designed to score **intermediate diffusion latents**. Decode the latent to a 3-channel RGB image with the lightweight linear approximation below (no VAE decode required), then pass it to the processor as a normal image:

```python
import torch
from PIL import Image

def latents_to_rgb(latents):
    weights = (
        (60, -60,  25, -70),
        (60,  -5,  15, -50),
        (60,  10,  -5, -35),
    )
    w = torch.t(torch.tensor(weights, dtype=latents.dtype, device=latents.device))
    b = torch.tensor((150, 140, 130), dtype=latents.dtype, device=latents.device)
    rgb = torch.einsum("...lxy,lr -> ...rxy", latents, w) + b[:, None, None]
    arr = rgb.clamp(0, 255)[0].byte().cpu().numpy().transpose(1, 2, 0)
    return Image.fromarray(arr)
```

## Training

NoisyCLIP is trained with the standard CLIP contrastive objective on pairs of **prompts** and **intermediate SDXL latents** (decoded to RGB). The text encoder and projection heads are frozen so that only the vision backbone adapts to the noisy-latent domain. See the [project page](https://novasearch.github.io/noisyclip/) and [repository](https://github.com/novasearch/noisyclip) for the training pipeline.

## Limitations

- The vision encoder is specialized for SDXL-style latents decoded via the linear approximation above; behavior on other latent spaces or decoders may differ.
- Inherits the biases and failure modes of the underlying CLIP ViT-L/14 and of the SDXL data used for fine-tuning.
- Intended as an alignment / ranking signal.

## Citation

If you find NoisyCLIP useful for your research, please cite:

```bibtex
@misc{ramos2026earlyestimationlanguagelatent,
      title={Early Estimation of Language to Latent Alignment in Diffusion Models}, 
      author={Vasco Ramos and Regev Cohen and Idan Szpektor and Joao Magalhaes},
      year={2026},
      eprint={2512.08505},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.08505}, 
}
```