Upload README.md with huggingface_hub

README.md

@@ -1,7 +1,6 @@
 ---
 license: cc-by-nc-4.0
 base_model: stabilityai/stable-diffusion-3-medium-diffusers
-library_name: peft
 pipeline_tag: image-to-image
 language: en
 tags:
@@ -10,8 +9,6 @@ tags:
   - extreme-zoom
   - chain-of-zoom
   - diffusion
-  - lora
-  - peft
   - privileged-distillation
   - faithfulness
 ---
@@ -19,79 +16,49 @@ tags:
 # OracleZoom
 
 **Privileged-Latent Distillation for faithful extreme super-resolution.**
-A tiny (7.1M-parameter) LoRA adapter that makes Chain-of-Zoom's recursive super-resolution add *faithful* detail instead of hallucinating, all the way to 256x. Shipped both as the LoRA adapter and as a **fully-merged transformer you can download and use directly**.
+OracleZoom drives Chain-of-Zoom's recursive 4x super-resolution out to 256x while staying *faithful*, adding real detail instead of hallucinating. This repo ships one ready-to-use file: the **merged super-resolution transformer**.
 
-[![GitHub](https://img.shields.io/badge/Code-OPD--Zoom-black?logo=github)](https://github.com/dipta007/OPD-Zoom)
+[![Code](https://img.shields.io/badge/Code-OPD--Zoom-black?logo=github)](https://github.com/dipta007/OPD-Zoom)
 [![Base](https://img.shields.io/badge/Backbone-OSEDiff%20/%20SD3-blue)](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers)
-[![Method](https://img.shields.io/badge/Method-Chain--of--Zoom-orange)](https://github.com/dipta007/OPD-Zoom)
 [![Paper](https://img.shields.io/badge/Paper-WACV%202027%20(in%20submission)-red)](https://github.com/dipta007/OPD-Zoom)
 [![License](https://img.shields.io/badge/License-CC--BY--NC--4.0-lightgrey)](https://creativecommons.org/licenses/by-nc/4.0/)
 
-## Highlights
-- **Faithful, not just sharp.** At extreme zoom the backbone must *invent* detail; the question is whether it is faithful or hallucinated. This model teaches faithfulness.
-- **Holds where baselines collapse.** CLIPIQA **0.71 at 256x** while Chain-of-Zoom (CoZ) and five SOTA SR backbones fall to <=0.58; most faithful of all methods at 4x (LPIPS **0.20** vs CoZ 0.22).
-- **Judged more faithful.** Two cross-family vision-language judges (InternVL + Gemini) prefer this zoom **68-78%** of the time at 64-256x and flag the strongest baseline hallucinating **2-5x more**.
-- **Tiny to train, easy to use.** A rank-16 LoRA (**7.1M** trainable params) trained on only **1,000** curated 4K images; shipped as both the adapter and a merged, drop-in transformer.
-
-## Files in this repo
-| File | What it is |
-|---|---|
-| `merged_transformer.safetensors` | **The complete OracleZoom SR transformer** (SD3 + OSEDiff's SR-LoRA + our PLD adapter, all baked in), fp32, ~8.35 GB. Download-and-use: drop it in as the transformer, no LoRA step. |
-| `adapter_model.safetensors` + `adapter_config.json` | The rank-16 PLD LoRA **alone** (~28 MB), if you prefer to apply it onto your own OSEDiff transformer. |
-| `train_meta.json` | Training recipe / provenance. |
-
-## Model Overview
-| Property | Value |
-|---|---|
-| Model type | LoRA adapter (PEFT) for a one-step SR backbone, + merged transformer |
-| Backbone | OSEDiff on Stable Diffusion 3-medium |
-| Prompt extractor (frozen) | Qwen2.5-VL-3B-Instruct |
-| Trainable params | 7.1M |
-| LoRA | r = 16, alpha = 32, dropout = 0.0 |
-| LoRA targets | `to_q, to_k, to_v, add_q_proj, add_k_proj, add_v_proj` (SD3 transformer) |
-| Training data | 1,000 curated 4K photographs (supervised at 4x only) |
-| Objective | decode-space LPIPS + anchored cycle-consistency - beta_reward * TOPIQ-NR + beta_kl * KL-to-base + EMA |
-| Key weights | beta_reward 0.4, beta_kl 8.0, w_cyc 1.0, lambda_ema 0.1 (EMA decay 0.95) |
-| Recursion at test | 4 steps (4x / 16x / 64x / 256x), 512x512 center crop |
+## What's in this repo
+- **`merged_transformer.safetensors`** (fp32, ~8.35 GB): the complete OracleZoom super-resolution transformer, Stable Diffusion 3 + Chain-of-Zoom's SR module + our distilled adapter, all merged into one set of weights. This is all you need.
 
-## Method
-Recursive SR (Chain-of-Zoom) reuses a 4x backbone step after step to reach 16x-256x. Each step is **blind**: it sees only a blurred crop of its own previous output and must invent the missing detail, so errors compound and the invention may be hallucinated.
+## Quickstart (one command)
+Zooms your images 4x -> 16x -> 64x -> 256x with the merged transformer. Needs one NVIDIA GPU (~16 GB).
 
-**Privileged-latent distillation (the idea).** A *privileged teacher* is shown the ground-truth high-resolution patch **at training time only** and distills its real detail into the blind student, in **decode space** (a perceptual loss between the student's decoded image and the real patch). Only a small LoRA adapter is trained; the backbone, VAE, and prompter stay frozen.
+```bash
+# 1. Get the pipeline (Chain-of-Zoom is included as a submodule)
+git clone --recursive https://github.com/dipta007/OPD-Zoom
+cd OPD-Zoom
 
-**A KL leash keeps the deep reward faithful (the safeguard).** Ground truth exists only at 4x. To carry the distilled prior into the deeper recursion, the student chases a differentiable detail reward (TOPIQ-NR) through the real zoom. Left free, such a reward games the metric with a repetitive crosshatch; **leashed** to the deployed backbone by a KL trust region (a latent distance for a one-step map), it sharpens detail without drifting into hallucination.
+# 2. Install dependencies (Python 3.10)
+pip install -r ref/coz/requirements.txt
+pip install -U "huggingface_hub[cli]"
 
-## Quickstart
-Both paths run inside the [OPD-Zoom](https://github.com/dipta007/OPD-Zoom) / Chain-of-Zoom pipeline.
+# 3. Download the merged model into the pipeline
+hf download dipta007/OracleZoom merged_transformer.safetensors --local-dir ckpt/OracleZoom
 
-```bash
-git clone https://github.com/dipta007/OPD-Zoom && cd OPD-Zoom
-huggingface-cli download dipta007/OracleZoom --local-dir ckpt/OracleZoom
+# 4. Put your images in ./inputs, then run the 4-step zoom
+python -m opd_zoom.teacher.oracle_infer \
+  --mode student \
+  --full_transformer ckpt/OracleZoom/merged_transformer.safetensors \
+  --gt_dir ./inputs --out ./outputs --rec_num 4
 ```
 
-**Option A - merged transformer (recommended, no LoRA step).** `merged_transformer.safetensors` already contains SD3 + OSEDiff's SR-LoRA + our PLD adapter, so you just load it as the SR transformer's weights:
-```python
-from safetensors.torch import load_file
-sd = load_file("ckpt/OracleZoom/merged_transformer.safetensors")
-# `transformer` = the OSEDiff SD3Transformer2DModel built by the pipeline (build_sr)
-transformer.load_state_dict(sd, strict=False)
-```
+**Results:** `outputs/per-scale/scale1/<name>.png` ... `scale4/<name>.png` are your image at **4x / 16x / 64x / 256x**.
 
-**Option B - LoRA adapter.** Apply the rank-16 adapter onto the OSEDiff transformer with PEFT:
-```python
-from peft import PeftModel
-transformer = PeftModel.from_pretrained(transformer, "dipta007/OracleZoom")
-```
-Or point the repo's inference at the adapter directly:
-```bash
-python -m opd_zoom.teacher.oracle_infer \
-  --mode student --pld_lora ckpt/OracleZoom \
-  --gt_dir <your_images> --out <out_dir> --rec_num 4
-```
-Either way the VLM prompter is unchanged, so per-image inference cost equals Chain-of-Zoom's.
+> Notes: Stable Diffusion 3-medium and the Qwen2.5-VL prompter download automatically on first run (a HuggingFace login may be needed for SD3). Chain-of-Zoom's own SR and VLM checkpoints must sit under `ref/coz/ckpt/` (`SR_LoRA`, `SR_VAE`, `VLM_LoRA`); see the [Chain-of-Zoom](https://github.com/bryanswkim/Chain-of-Zoom) repo to fetch them. That is the only extra download.
+
+## Method
+Recursive SR (Chain-of-Zoom) reuses a 4x backbone step after step to reach 16x-256x. Each step is **blind**: it sees only a blurred crop of its own previous output and must invent the missing detail, so errors compound and the invention may be hallucinated.
+
+**Privileged-latent distillation.** A *privileged teacher* is shown the ground-truth high-resolution patch **at training time only** and distills its real detail into the blind student, in **decode space**. Only a small adapter is trained; the backbone, VAE, and prompter stay frozen. **A KL leash** to the deployed backbone keeps a deep sharpness reward from drifting into a metric-gaming texture, so detail stays faithful. (Trained: rank-16 adapter, 7.1M params, 1,000 curated 4K images; beta_reward 0.4, beta_kl 8.0. The released weights above have this adapter already merged in.)
 
 ## Results
-Under Chain-of-Zoom's exact protocol on a curated 4K benchmark and six test sets (in-domain 4K, DIV8K, DRealSR, RealSR, FFHQ, Flickr2K):
+Under Chain-of-Zoom's exact protocol on a curated 4K benchmark and six test sets:
 
 | Axis | Metric | Ours | CoZ / best baseline |
 |---|---|---|---|
@@ -103,11 +70,8 @@ Under Chain-of-Zoom's exact protocol on a curated 4K benchmark and six test sets
 Sharpness is the axis prior methods are built for; the decisive gap is **faithfulness**, verified by full-reference metrics at 4x and by two cross-family MLLM judges plus a blinded human study past 4x.
 
 ## Intended Use
-- **In-scope:** research on faithful extreme (recursive) super-resolution; as the SR-backbone inside the Chain-of-Zoom recursion on natural photographs.
-- **Out-of-scope:** a standalone single-shot SR model (it is the SR backbone for the CoZ loop); forensic/evidentiary use (detail past 4x is generated, not recovered); real-camera-zoom claims (the benchmark uses synthetic center-crop zoom).
-
-## Training
-Early-stopped on held-out validation at ~epoch 37 (step 9300); best val 0.216. Trained on one 8xH200 node (single GPU trains the adapter). Full config in `train_meta.json` and the [repo](https://github.com/dipta007/OPD-Zoom).
+- **In-scope:** research on faithful extreme (recursive) super-resolution of natural photographs.
+- **Out-of-scope:** forensic/evidentiary use (detail past 4x is generated, not recovered); real-camera-zoom claims (the benchmark uses synthetic center-crop zoom).
 
 ## Citation
 ```bibtex
@@ -122,4 +86,4 @@ Early-stopped on held-out validation at ~epoch 37 (step 9300); best val 0.216. T
 Please also cite Chain-of-Zoom and OSEDiff, whose components this builds on.
 
 ## License
-Released for **research, non-commercial** use (CC-BY-NC-4.0). This model is built on top of OSEDiff / Stable Diffusion 3 and used with a Qwen2.5-VL prompter inside Chain-of-Zoom; the respective upstream licenses apply to those components.
+Released for **research, non-commercial** use (CC-BY-NC-4.0). Built on OSEDiff / Stable Diffusion 3 and used with a Qwen2.5-VL prompter inside Chain-of-Zoom; the respective upstream licenses apply to those components.