File size: 2,606 Bytes
f918a65 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
license: mit
library_name: transformers
tags:
- perceptual-similarity
- feature-extraction
- image-feature-extraction
- dreamsim
pipeline_tag: image-feature-extraction
---
# DreamSim (ensemble) β HuggingFace format
Unofficial HuggingFace-format port of the **DreamSim** perceptual-similarity metric
(ensemble variant). Loads via `trust_remote_code` with no `dreamsim` / `peft` runtime
dependency; weights are the LoRA-merged ensemble stored as `model.safetensors`.
> **Not an official DreamSim release.** DreamSim was created by Shobhita Sundaram,
> Netanel Tamir, Stephanie Fu, and Richard Zhang β see the original repository
> https://github.com/ssundaram21/dreamsim and paper *"DreamSim: Learning New
> Dimensions of Human Visual Similarity using Synthetic Data"* (NeurIPS 2023).
> This repo re-packages their released ensemble for `transformers`.
## What it is
The ensemble concatenates the features of three ViT-B/16 backbones β DINO (`cls`),
CLIP (`embedding`), OpenCLIP (`embedding`) β each LoRA-adapted, then mean/L2-normalizes
(`embed_size = 1792`). Perceptual distance between two images is `1 β cos`.
This port has been verified against the official `dreamsim` package on 1,000 real
images: **max |Ξ embedding| < 3e-7 and max |Ξ distance| < 2e-7** β identical up to
float32 rounding.
## Usage
```python
import torch
from PIL import Image
from transformers import AutoModel, AutoImageProcessor
model = AutoModel.from_pretrained("bigshanedogg/dreamsim", trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained("bigshanedogg/dreamsim", trust_remote_code=True)
img_a = proc(images=Image.open("a.png"), return_tensors="pt")["pixel_values"]
img_b = proc(images=Image.open("b.png"), return_tensors="pt")["pixel_values"]
emb = model(pixel_values=img_a).embeddings # (1, 1792)
dist = model.compute_distance(img_a, img_b) # 1 - cos (higher = more different)
```
The image processor resizes to 224Γ224 (BICUBIC) and scales to `[0, 1]`; the
per-backbone mean/std normalization is applied inside the model.
## Licensing
- Port + original DreamSim code/weights: **MIT** β see [`LICENSE`](LICENSE).
- Vendored ViT backbone code (adapted from DINO, Meta AI): **Apache-2.0** β full text in
[`LICENSE.apache-2.0.txt`](LICENSE.apache-2.0.txt); attribution + modification statement
in [`NOTICE`](NOTICE).
- Merged backbone weights derive from DINO (Apache-2.0), CLIP (MIT), and OpenCLIP
(MIT code, LAION-based weights). Review the upstream backbone terms β and, for
OpenCLIP, the LAION data provenance β before any commercial use.
|