README.md

---
base_model: stabilityai/stable-diffusion-3-medium-diffusers
language: en
license: cc-by-nc-4.0
pipeline_tag: image-to-image
tags:
- super-resolution
- image-super-resolution
- extreme-zoom
- chain-of-zoom
- diffusion
- privileged-distillation
- faithfulness
---

# OracleZoom

[Shubhashis Roy Dipta](https://roydipta.com)\*, [Sourajit Saha](https://sourajitcs.github.io/)\*, [Shaswati Saha](https://scholar.google.com/citations?hl=en&user=_pTdzsAAAAAJ&view_op=list_works&sortby=pubdate), [Nobin Sarwar](https://smsnobin77.github.io/) · University of Maryland, Baltimore County

\*Equal contribution.

**Privileged-Latent Distillation for faithful extreme super-resolution.**  
OracleZoom drives Chain-of-Zoom's recursive 4x super-resolution out to 256x while staying *faithful*, adding real detail instead of hallucinating. **This repo is self-contained**: the merged model, the inference code, and the required checkpoints are all here. You only download two public base models (Stable Diffusion 3-medium, Qwen2.5-VL-3B) automatically.

[![Base](https://img.shields.io/badge/Base-SD3%20%2B%20Qwen2.5--VL-blue)](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers)  
[![Method](https://img.shields.io/badge/Method-Chain--of--Zoom-orange)](https://github.com/bryanswkim/Chain-of-Zoom)  
[![Paper](https://img.shields.io/badge/Paper-WACV%202026%20(in%20submission)-red)](https://huggingface.co/papers/2609.06490)  
[![Code](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/dipta007/OracleZoom)  
[![Project](https://img.shields.io/badge/Project-Page-lightgrey)](https://dipta007.github.io/OracleZoom/)  
[![License](https://img.shields.io/badge/License-CC--BY--NC--4.0-lightgrey)](https://creativecommons.org/licenses/by-nc/4.0/)

**Paper:** https://huggingface.co/papers/2609.06490 · **Code:** https://github.com/dipta007/OracleZoom · **Project page:** https://dipta007.github.io/OracleZoom/

## Quickstart (one image, all scales)

Needs one NVIDIA GPU (~16 GB) and Python 3.10.

```bash
# 0. One-time: Stable Diffusion 3 is gated, so accept its license on HF, then log in
pip install -U "huggingface_hub[cli]"
hf auth login

# 1. Download this repo (merged model + code + checkpoints)
hf download dipta007/OracleZoom --local-dir OracleZoom
cd OracleZoom

# 2. Install dependencies
pip install -r requirements.txt

# 3. Super-resolve ONE image (4x -> 16x -> 64x -> 256x)
python inference.py --input /path/to/photo.jpg --output ./outputs
```

**Results** in `./outputs/`:
`photo_1x.png` (the 512x512 input crop), `photo_4x.png`, `photo_16x.png`, `photo_64x.png`, `photo_256x.png`.

Stable Diffusion 3-medium and Qwen2.5-VL-3B download automatically on first run.

> **Batching many images:** `inference.py` exposes `zoom_image(sr, model, proc, pvi, image_path, out_dir)`. Build the models once (`build_sr(...)`, `build_vlm(...)`) and call `zoom_image` in a loop over your images.

## Training data
The curated training set is released separately at
[dipta007/OracleZoom-4KLSDB-train](https://huggingface.co/datasets/dipta007/OracleZoom-4KLSDB-train).
The released model uses its `1k` config (1,000 curated 4K images).

## What's in this repo
| Path | What it is |
|---|---|
| `merged_transformer.safetensors` | The OracleZoom super-resolution transformer (SD3 + Chain-of-Zoom's SR module + our distilled adapter, merged), fp32, ~8.35 GB. |
| `inference.py` | Self-contained runner: one image in, all scales out (recursive zoom + VLM prompting). |
| `coz/` | Vendored Chain-of-Zoom inference code (the one-step SR wrapper + helpers). |
| `ckpt/` | Chain-of-Zoom's SR-VAE and VLM-prompt (Qwen LoRA) checkpoints needed by the pipeline. |
| `requirements.txt` | Python dependencies. |

## Method
Recursive SR (Chain-of-Zoom) reuses a 4x backbone step after step to reach 16x-256x. Each step is **blind**: it sees only a blurred crop of its own previous output and must invent the missing detail, so errors compound and the invention may be hallucinated.

**Privileged-latent distillation.** A *privileged teacher* is shown the ground-truth high-resolution patch **at training time only** and distills its real detail into the blind student, in **decode space**. Only a small adapter is trained; the backbone, VAE, and prompter stay frozen. **A KL leash** to the deployed backbone keeps a deep sharpness reward from drifting into a metric-gaming texture, so detail stays faithful. Trained: rank-16 adapter (7.1M params), 1,000 curated 4K images; beta_reward 0.4, beta_kl 8.0. The released weights have this adapter already merged in.

## Results
Under Chain-of-Zoom's exact protocol on a curated 4K benchmark and seven test sets:

| Axis | Metric | Ours | CoZ / best baseline |
|---|---|---|---|
| Sharpness (no-reference) | CLIPIQA @256x | **0.706** | 0.579 (CoZ) |
| Fidelity @4x (ground truth exists) | LPIPS | **0.199** | 0.215 (CoZ) |
| Deep faithfulness (MLLM judge, 64-256x) | preferred vs CoZ | **68-78%** | - |
| Deep faithfulness | hallucination rate vs CoZ | **2-5x lower** | - |

Sharpness is the axis prior methods are built for; the decisive gap is **faithfulness**, verified by full-reference metrics at 4x and by two cross-family MLLM judges plus a blinded human study past 4x.

## Intended Use
- **In-scope:** research on faithful extreme (recursive) super-resolution of natural photographs.
- **Out-of-scope:** forensic/evidentiary use (detail past 4x is generated, not recovered); real-camera-zoom claims (the benchmark uses synthetic center-crop zoom).

## Acknowledgements & Licensing
The `coz/` code and the checkpoints in `ckpt/` are from [Chain-of-Zoom](https://github.com/bryanswkim/Chain-of-Zoom) and are redistributed here for convenience; please respect their original license and cite them. The pipeline uses Stable Diffusion 3-medium and Qwen2.5-VL-3B under their respective licenses. OracleZoom's own contribution (the distilled adapter, merged into `merged_transformer.safetensors`) is released for **research, non-commercial** use (CC-BY-NC-4.0).

## Citation
```bibtex
@inproceedings{dipta2026oraclezoom,
  title={OracleZoom: Privileged-Latent Distillation for Faithful Extreme Super-Resolution},
  author={Roy Dipta, Shubhashis and Saha, Sourajit and Saha, Shaswati and Sarwar, Nobin},
  year={2026},
  note={In submission, WACV 2026}
}
```
Please also cite Chain-of-Zoom and OSEDiff.