Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+demo_images/GeoSynth-Location-Canny/input.jpeg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,115 @@
+---
+license: apache-2.0
+library_name: diffusers
+pipeline_tag: image-to-image
+tags:
+- controlnet
+- remote-sensing
+- arxiv:2404.06637
+widget:
+# GeoSynth-Location-OSM: OSM tile -> satellite image (default lon=-90.2, lat=38.6)
+- src: demo_images/GeoSynth-Location-OSM/input.jpeg
+  prompt: Satellite image features a city neighborhood
+  output:
+    url: demo_images/GeoSynth-Location-OSM/output.jpeg
+# GeoSynth-Location-Canny: Canny edges -> satellite image (default lon=-90.2, lat=38.6)
+- src: demo_images/GeoSynth-Location-Canny/input.jpeg
+  prompt: Satellite image features a city neighborhood
+  output:
+    url: demo_images/GeoSynth-Location-Canny/output.jpeg
+# GeoSynth-Location-SAM: SAM segmentation -> satellite image
+- src: demo_images/GeoSynth-Location-SAM/input.jpeg
+  prompt: Satellite image features a city neighborhood
+  output:
+    url: demo_images/GeoSynth-Location-SAM/output.jpeg
+---
+
+# GeoSynth-ControlNets-Location
+
+Repository for location-conditioned GeoSynth ControlNets.
+
+**Location (lon/lat) conditioning** is the primary workflow for geo-aware synthesis. Default: St. Louis, MO (`lon=-90.2`, `lat=38.6`). Use SatCLIP + CoordNet for full diffusers-style location conditioning.
+
+We maintain **two repositories**—one per base checkpoint—each with its compatible ControlNets:
+
+| Repo | Base Model | ControlNets |
+|------|------------|-------------|
+| **[GeoSynth-ControlNets](https://huggingface.co/BiliSakura/GeoSynth-ControlNets)** | GeoSynth (text encoder & UNet same as SD 2.1) | GeoSynth-OSM, GeoSynth-Canny, GeoSynth-SAM |
+| **This repo** | GeoSynth-Location (adds CoordNet branch) | GeoSynth-Location-OSM, GeoSynth-Location-SAM*, GeoSynth-Location-Canny |
+
+*[GeoSynth-Location-SAM](https://huggingface.co/MVRL/GeoSynth-Location-SAM) controlnet ckpt is missing from source.*
+
+### This repository
+
+1. **GeoSynth-Location base** — Converted from `geosynth_sd_loc-v3.ckpt` to diffusers format. Text encoder and UNet are the same as SD 2.1 (not fine-tuned). The original checkpoint also includes a CoordNet branch for `[lon, lat]` conditioning (see Architecture).
+2. **ControlNet models** — GeoSynth-Location-OSM, GeoSynth-Location-Canny (converted from SD-style checkpoints under `MVRL/GeoSynth-Location-OSM` and `MVRL/GeoSynth-Location-Canny`), and GeoSynth-Location-SAM, located under [`controlnet/`](controlnet/).
+
+### Architecture
+
+The full location pipeline adds a **CoordNet** branch to the base LDM:
+
+- **Input**: `[lon, lat]` → **SatCLIP** location encoder → **CoordNet** (13 stacked cross-attention blocks, inner dim 256, 4 heads) → conditioning injected into UNet
+- ControlNet and CoordNet jointly condition the UNet (see [GeoSynth paper](https://huggingface.co/papers/2404.06637) Figure 3)
+
+### ControlNet variants (this repo)
+
+| Control | Subfolder | Status |
+|---------|-----------|--------|
+| OSM     | `controlnet/GeoSynth-Location-OSM` | ✅ ready |
+| Canny   | `controlnet/GeoSynth-Location-Canny` | ✅ ready |
+| SAM     | `controlnet/GeoSynth-Location-SAM` | ⏳ ckpt pending |
+
+### Model Sources
+
+- **Source:** [GeoSynth](https://github.com/mvrl/GeoSynth)
+- **Paper:** [GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesis](https://huggingface.co/papers/2404.06637)
+- **Base model:** [Stable Diffusion 2.1](https://huggingface.co/sd2-community/stable-diffusion-2-1-base)
+- **Related:** [GeoSynth-ControlNets](https://huggingface.co/BiliSakura/GeoSynth-ControlNets) (non-location models)
+
+## Usage
+
+**CLI:**
+```bash
+python inference_demo.py --control demo_images/GeoSynth-Location-OSM/input.jpeg --control_type OSM --lon -90.2 --lat 38.6
+```
+
+**Python:**
+```python
+import sys, os
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+import torch
+from PIL import Image
+from geosynth_pipeline import load_geosynth_pipeline_with_location, run_with_location
+
+pipe = load_geosynth_pipeline_with_location(".", controlnet_subfolder="controlnet/GeoSynth-Location-OSM", local_files_only=True)
+pipe = pipe.to("cuda")
+
+img = Image.open("demo_images/GeoSynth-Location-OSM/input.jpeg").convert("RGB").resize((512, 512))
+output = run_with_location(pipe, "Satellite image features a city neighborhood", image=img, lon=-90.2, lat=38.6)
+output.images[0].save("generated_city.jpg")
+```
+
+## Citation
+
+If you use this model, please cite the GeoSynth paper. For location-conditioned variants, also cite SatCLIP.
+
+```bibtex
+@inproceedings{sastry2024geosynth,
+  title={GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesis},
+  author={Sastry, Srikumar and Khanal, Subash and Dhakal, Aayush and Jacobs, Nathan},
+  booktitle={IEEE/ISPRS Workshop: Large Scale Computer Vision for Remote Sensing (EARTHVISION)},
+  year={2024}
+}
+
+@article{klemmer2025satclip,
+  title={{SatCLIP}: {Global}, General-Purpose Location Embeddings with Satellite Imagery},
+  author={Klemmer, Konstantin and Rolf, Esther and Robinson, Caleb and Mackey, Lester and Ru{\ss}wurm, Marc},
+  journal={Proceedings of the AAAI Conference on Artificial Intelligence},
+  volume={39},
+  number={4},
+  pages={4347--4355},
+  year={2025},
+  doi={10.1609/aaai.v39i4.32457}
+}
+```

controlnet/GeoSynth-Location-Canny/config.json

@@ -0,0 +1,56 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.37.0",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_channels": 3,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "transformer_layers_per_block": 1,
+  "upcast_attention": true,
+  "use_linear_projection": true
+}

controlnet/GeoSynth-Location-Canny/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22f7bab41c6a6da224c1987e2b8d69e5b8e5f45218f78ee43ec3eb55bbd20508
+size 1456953560

controlnet/GeoSynth-Location-OSM/config.json

@@ -0,0 +1,56 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.37.0",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_channels": 3,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "transformer_layers_per_block": 1,
+  "upcast_attention": true,
+  "use_linear_projection": true
+}

controlnet/GeoSynth-Location-OSM/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5119803f858a8038a127b3251a582e8bd7c3e9f20cdfb8ae6f2bdc413ae880f4
+size 1456953560

controlnet/GeoSynth-Location-SAM/config.json

@@ -0,0 +1,56 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.37.0",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_channels": 3,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "transformer_layers_per_block": 1,
+  "upcast_attention": true,
+  "use_linear_projection": true
+}

convert_ckpts.py

@@ -0,0 +1,50 @@
+#!/usr/bin/env python3
+"""
+Convert SatCLIP and CoordNet from source .ckpt to diffusers-style format.
+Saves into this repo (satclip_location_encoder/, coordnet/) for self-contained inference.
+
+Usage:
+  1. Download SatCLIP: huggingface.co/microsoft/SatCLIP-ViT16-L10 -> satclip-vit16-l10.ckpt
+  2. Download GeoSynth-Location-OSM: huggingface.co/MVRL/GeoSynth-Location-OSM -> geosynth_loc_osm.ckpt
+  3. Run: python convert_ckpts.py --satclip_ckpt /path/to/satclip-vit16-l10.ckpt --coordnet_ckpt /path/to/geosynth_loc_osm.ckpt
+"""
+import argparse
+import os
+import sys
+
+REPO_DIR = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, REPO_DIR)
+
+from geosynth_pipeline import GeoSynthCoordNetModel, SatCLIPLocationEncoder
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--satclip_ckpt", required=True, help="Path to satclip-vit16-l10.ckpt")
+    parser.add_argument("--coordnet_ckpt", required=True, help="Path to geosynth_loc_osm.ckpt")
+    args = parser.parse_args()
+
+    satclip_out = os.path.join(REPO_DIR, "satclip_location_encoder")
+    coordnet_out = os.path.join(REPO_DIR, "coordnet")
+
+    if not os.path.isfile(args.satclip_ckpt):
+        sys.exit(f"SatCLIP checkpoint not found: {args.satclip_ckpt}")
+    if not os.path.isfile(args.coordnet_ckpt):
+        sys.exit(f"CoordNet checkpoint not found: {args.coordnet_ckpt}")
+
+    print("Loading SatCLIP location encoder...")
+    loc_enc = SatCLIPLocationEncoder.from_satclip_ckpt(args.satclip_ckpt)
+    os.makedirs(satclip_out, exist_ok=True)
+    loc_enc.save_pretrained(satclip_out)
+    print(f"Saved to {satclip_out}")
+
+    print("Loading CoordNet...")
+    coordnet = GeoSynthCoordNetModel.from_geosynth_ckpt(args.coordnet_ckpt)
+    os.makedirs(coordnet_out, exist_ok=True)
+    coordnet.save_pretrained(coordnet_out)
+    print(f"Saved to {coordnet_out}")
+    print("Done. Use inference_demo.py with local paths.")
+
+
+if __name__ == "__main__":
+    main()

coordnet/config.json

@@ -0,0 +1,8 @@
+{
+  "_class_name": "GeoSynthCoordNetModel",
+  "_diffusers_version": "0.37.0",
+  "block_out_channels": null,
+  "embed_dim": 256,
+  "num_heads": 4,
+  "time_embed_dim": 1280
+}

coordnet/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:757124ffa2752db3822a70e40684da275648b9f9a4e1efa8eda407f452273576
+size 65790520

feature_extractor/preprocessor_config.json

@@ -0,0 +1,27 @@
+{
+  "crop_size": {
+    "height": 224,
+    "width": 224
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "CLIPImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 224
+  }
+}

geosynth_pipeline/__init__.py

@@ -0,0 +1,13 @@
+# GeoSynth custom pipeline: SatCLIP + CoordNet location conditioning
+from .satclip_location_encoder import SatCLIPLocationEncoder
+from .coordnet import GeoSynthCoordNetModel
+from .controlnet_location_wrapper import GeoSynthControlNetWithLocation
+from .pipeline import load_geosynth_pipeline_with_location, run_with_location
+
+__all__ = [
+    "SatCLIPLocationEncoder",
+    "GeoSynthCoordNetModel",
+    "GeoSynthControlNetWithLocation",
+    "load_geosynth_pipeline_with_location",
+    "run_with_location",
+]

geosynth_pipeline/controlnet_location_wrapper.py

@@ -0,0 +1,51 @@
+"""Wrapper combining ControlNet + CoordNet for location-conditioned generation.
+Subclasses ControlNetModel so pipeline's isinstance checks pass."""
+from typing import Any, Optional
+
+import torch
+from diffusers.models import ControlNetModel
+
+from .coordnet import GeoSynthCoordNetModel
+from .satclip_location_encoder import SatCLIPLocationEncoder
+
+
+class GeoSynthControlNetWithLocation(ControlNetModel):
+    """ControlNetModel + CoordNet. Passes isinstance(ControlNetModel) for pipeline compatibility."""
+
+    def __init__(self, controlnet: ControlNetModel, coordnet: GeoSynthCoordNetModel,
+                 location_encoder: SatCLIPLocationEncoder, unet: torch.nn.Module):
+        cfg = dict(controlnet.config)
+        super().__init__(**cfg)
+        self.load_state_dict(controlnet.state_dict(), strict=True)
+        self._coordnet = coordnet
+        self._location_encoder = location_encoder
+        self._unet = unet
+        self._location_embedding: Optional[torch.Tensor] = None
+
+    def set_location_embedding(self, embedding: torch.Tensor):
+        self._location_embedding = embedding
+
+    def forward(self, sample: torch.Tensor, timestep: torch.Tensor, encoder_hidden_states: torch.Tensor,
+                controlnet_cond: torch.Tensor, conditioning_scale: float = 1.0, guess_mode: bool = False,
+                return_dict: bool = True, **kwargs) -> Any:
+        down_block_res_samples, mid_block_res_sample = super().forward(
+            sample, timestep, encoder_hidden_states=encoder_hidden_states, controlnet_cond=controlnet_cond,
+            conditioning_scale=conditioning_scale, guess_mode=guess_mode, return_dict=False, **kwargs
+        )
+        if self._location_embedding is not None and self._coordnet is not None:
+            coord_dtype = next(self._coordnet.parameters()).dtype
+            loc_emb = self._location_embedding.to(coord_dtype)
+            t_emb = self._unet.get_time_embed(sample=sample, timestep=timestep)
+            time_emb = self._unet.time_embedding(t_emb, None).to(coord_dtype)
+            down_shapes = [tuple(d.shape[2:]) for d in down_block_res_samples]
+            mid_shape = tuple(mid_block_res_sample.shape[2:])
+            loc_down, loc_mid = self._coordnet(
+                loc_emb, time_emb,
+                down_block_sample_shapes=down_shapes, mid_block_sample_shape=mid_shape, return_dict=False
+            )
+            down_block_res_samples = [c + l for c, l in zip(down_block_res_samples, loc_down)]
+            mid_block_res_sample = mid_block_res_sample + loc_mid
+        if return_dict:
+            from diffusers.models.controlnets.controlnet import ControlNetOutput
+            return ControlNetOutput(down_block_res_samples=tuple(down_block_res_samples), mid_block_res_sample=mid_block_res_sample)
+        return down_block_res_samples, mid_block_res_sample

geosynth_pipeline/coordnet.py

@@ -0,0 +1,87 @@
+"""
+GeoSynth CoordNet - diffusers-style model.
+13 cross-attention blocks producing location conditioning for UNet.
+"""
+from typing import List, Optional, Tuple, Union
+
+import torch
+from torch import nn
+from diffusers.configuration_utils import ConfigMixin, register_to_config
+from diffusers.models.modeling_utils import ModelMixin
+
+
+class CoordNetLocationBlock(nn.Module):
+    def __init__(self, embed_dim: int = 256, out_channels: int = 320, num_heads: int = 4, time_embed_dim: int = 1280):
+        super().__init__()
+        self.query_embed = nn.Linear(embed_dim, embed_dim, bias=False)
+        self.key_embed = nn.Linear(time_embed_dim, embed_dim, bias=False)
+        self.value_embed = nn.Linear(time_embed_dim, embed_dim, bias=False)
+        self.cross_attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
+        self.norm1 = nn.LayerNorm(embed_dim)
+        self.norm2 = nn.LayerNorm(embed_dim)
+        self.ff = nn.Linear(embed_dim, embed_dim, bias=False)
+        self.ff_zero = nn.Linear(embed_dim, out_channels, bias=False)
+        nn.init.zeros_(self.ff_zero.weight)
+
+    def forward(self, loc_emb: torch.Tensor, time_emb: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        q = self.query_embed(loc_emb)
+        k = self.key_embed(time_emb).unsqueeze(1)
+        v = self.value_embed(time_emb).unsqueeze(1)
+        x, _ = self.cross_attn(q, k, v)
+        x = self.norm1(x) + loc_emb
+        x = self.norm2(self.ff(x)) + x
+        return x, self.ff_zero(x)
+
+
+class GeoSynthCoordNetModel(ModelMixin, ConfigMixin):
+    # 12 down blocks + 1 middle (separate loc_middle_block)
+    BLOCK_OUT_CHANNELS = [320, 320, 320, 320, 640, 640, 640, 1280, 1280, 1280, 1280, 1280]
+
+    @register_to_config
+    def __init__(self, embed_dim: int = 256, time_embed_dim: int = 1280, num_heads: int = 4,
+                 block_out_channels: Optional[List[int]] = None):
+        super().__init__()
+        chs = block_out_channels or self.BLOCK_OUT_CHANNELS
+        self.loc_blocks = nn.ModuleList([
+            CoordNetLocationBlock(embed_dim=embed_dim, out_channels=ch, num_heads=num_heads, time_embed_dim=time_embed_dim)
+            for ch in chs
+        ])
+        self.loc_middle_block = CoordNetLocationBlock(embed_dim=embed_dim, out_channels=1280, num_heads=num_heads, time_embed_dim=time_embed_dim)
+
+    def forward(self, location_embedding: torch.Tensor, time_emb: torch.Tensor,
+                down_block_sample_shapes: Optional[List[tuple]] = None, mid_block_sample_shape: Optional[tuple] = None,
+                return_dict: bool = True):
+        if down_block_sample_shapes is None:
+            # SD 2.1 ControlNet down block spatial sizes
+            down_block_sample_shapes = [(64, 64), (64, 64), (64, 64), (32, 32), (32, 32), (32, 32),
+                                       (16, 16), (16, 16), (16, 16), (8, 8), (8, 8), (8, 8)]
+        if mid_block_sample_shape is None:
+            mid_block_sample_shape = (8, 8)
+        loc_input = location_embedding.unsqueeze(1)
+        down_residuals = []
+        for block, (h, w) in zip(self.loc_blocks, down_block_sample_shapes):
+            loc_input, loc_zero = block(loc_input, time_emb)
+            loc_spatial = loc_zero.squeeze(1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, h, w)
+            down_residuals.append(loc_spatial)
+        _, mid_zero = self.loc_middle_block(loc_input, time_emb)
+        h, w = mid_block_sample_shape
+        mid_spatial = mid_zero.squeeze(1).unsqueeze(-1).unsqueeze(-1).expand(-1, -1, h, w)
+        if return_dict:
+            return {"down_block_residuals": down_residuals, "mid_block_residual": mid_spatial}
+        return down_residuals, mid_spatial
+
+    @classmethod
+    def from_geosynth_ckpt(cls, ckpt_path: str, device: Optional[Union[str, torch.device]] = None) -> "GeoSynthCoordNetModel":
+        ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
+        state_dict = ckpt.get("state_dict", ckpt)
+        model = cls()
+        sd = {}
+        for k, v in state_dict.items():
+            if "control_model.loc_blocks" in k:
+                sd[k.replace("control_model.loc_blocks", "loc_blocks")] = v
+            elif "control_model.loc_middle_block" in k:
+                sd[k.replace("control_model.loc_middle_block", "loc_middle_block")] = v
+        model.load_state_dict(sd, strict=True)
+        if device is not None:
+            model = model.to(device)
+        return model

geosynth_pipeline/pipeline.py

@@ -0,0 +1,106 @@
+"""
+GeoSynth pipeline with full location conditioning.
+Load from local diffusers-style paths (satclip_location_encoder/, coordnet/) — no external models.
+"""
+from typing import Any, Optional, Union
+
+import os
+import torch
+from diffusers import StableDiffusionControlNetPipeline
+from diffusers.models import ControlNetModel
+
+from .controlnet_location_wrapper import GeoSynthControlNetWithLocation
+from .coordnet import GeoSynthCoordNetModel
+from .satclip_location_encoder import SatCLIPLocationEncoder
+
+
+def _is_diffusers_dir(path: str) -> bool:
+    return os.path.isdir(path) and os.path.isfile(os.path.join(path, "config.json"))
+
+
+def load_geosynth_pipeline_with_location(
+    model_path: str,
+    controlnet_subfolder: str = "controlnet/GeoSynth-Location-OSM",
+    satclip_path: Optional[str] = None,
+    coordnet_path: Optional[str] = None,
+    torch_dtype: Optional[torch.dtype] = None,
+    device: Optional[str] = None,
+    local_files_only: bool = False,
+) -> StableDiffusionControlNetPipeline:
+    """
+    Load GeoSynth pipeline with full location conditioning.
+    satclip_path and coordnet_path: paths relative to this repo (or abs).
+    Use diffusers-style dirs (satclip_location_encoder/, coordnet/) — run convert_ckpts.py first.
+    """
+    repo_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    satclip_path = satclip_path or os.path.join(repo_dir, "satclip_location_encoder")
+    coordnet_path = coordnet_path or os.path.join(repo_dir, "coordnet")
+
+    # Resolve to absolute
+    if not os.path.isabs(satclip_path):
+        satclip_path = os.path.join(repo_dir, satclip_path)
+    if not os.path.isabs(coordnet_path):
+        coordnet_path = os.path.join(repo_dir, coordnet_path)
+
+    device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+    dtype = torch_dtype or (torch.float16 if torch.cuda.is_available() else torch.float32)
+
+    controlnet = ControlNetModel.from_pretrained(
+        model_path, subfolder=controlnet_subfolder, torch_dtype=dtype, local_files_only=local_files_only
+    )
+    pipe = StableDiffusionControlNetPipeline.from_pretrained(
+        model_path, controlnet=controlnet, torch_dtype=dtype, local_files_only=local_files_only
+    )
+
+    # Load SatCLIP: from_pretrained (diffusers) or from_satclip_ckpt (.ckpt)
+    if _is_diffusers_dir(satclip_path):
+        location_encoder = SatCLIPLocationEncoder.from_pretrained(satclip_path)
+    else:
+        location_encoder = SatCLIPLocationEncoder.from_satclip_ckpt(satclip_path)
+    location_encoder = location_encoder.to(device).to(dtype)
+
+    # Load CoordNet: from_pretrained (diffusers) or from_geosynth_ckpt (.ckpt)
+    if _is_diffusers_dir(coordnet_path):
+        coordnet = GeoSynthCoordNetModel.from_pretrained(coordnet_path)
+    else:
+        coordnet = GeoSynthCoordNetModel.from_geosynth_ckpt(coordnet_path)
+    coordnet = coordnet.to(device).to(dtype)
+
+    wrapper = GeoSynthControlNetWithLocation(
+        controlnet=controlnet, coordnet=coordnet, location_encoder=location_encoder, unet=pipe.unet
+    )
+    wrapper = wrapper.to(device).to(dtype)
+    pipe.controlnet = wrapper
+    return pipe
+
+
+def run_with_location(
+    pipe: StableDiffusionControlNetPipeline,
+    prompt: Union[str, list],
+    image: Any,
+    lon: float = -90.2,
+    lat: float = 38.6,
+    num_inference_steps: int = 20,
+    guidance_scale: float = 7.5,
+    generator: Optional[torch.Generator] = None,
+    controlnet_conditioning_scale: float = 1.0,
+    num_images_per_prompt: int = 1,
+    **kwargs,
+) -> Any:
+    """Run inference with location conditioning."""
+    if not hasattr(pipe.controlnet, "set_location_embedding"):
+        return pipe(prompt, image=image, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale,
+            generator=generator, controlnet_conditioning_scale=controlnet_conditioning_scale,
+            num_images_per_prompt=num_images_per_prompt, **kwargs)
+    device = getattr(pipe, "_execution_device", None) or next(pipe.unet.parameters()).device
+    batch_size = 1 if isinstance(prompt, str) else len(prompt)
+    n = batch_size * num_images_per_prompt
+    if guidance_scale > 1:
+        n *= 2
+    lonlat = torch.tensor([[lon, lat]], device=device, dtype=torch.float32).expand(n, 2)
+    with torch.no_grad():
+        loc_emb = pipe.controlnet._location_encoder(lonlat)
+    pipe.controlnet.set_location_embedding(loc_emb)
+    return pipe(prompt, image=image, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale,
+        generator=generator, controlnet_conditioning_scale=controlnet_conditioning_scale,
+        num_images_per_prompt=num_images_per_prompt, **kwargs)

geosynth_pipeline/satclip_location_encoder.py

@@ -0,0 +1,116 @@
+"""
+SatCLIP Location Encoder - diffusers-style model.
+Encodes (lon, lat) -> 256-dim embedding via spherical harmonics + SIREN MLP.
+"""
+import math
+from typing import Optional, Union
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from .spherical_harmonics import SphericalHarmonicsEncoder
+from diffusers.configuration_utils import ConfigMixin, register_to_config
+from diffusers.models.modeling_utils import ModelMixin
+
+
+def _exists(val):
+    return val is not None
+
+
+class Sine(nn.Module):
+    def __init__(self, w0: float = 1.0):
+        super().__init__()
+        self.w0 = w0
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return torch.sin(self.w0 * x)
+
+
+class SirenLayer(nn.Module):
+    def __init__(self, dim_in: int, dim_out: int, w0: float = 1.0, c: float = 6.0, is_first: bool = False,
+                 use_bias: bool = True, activation: Optional[nn.Module] = None, dropout: bool = False):
+        super().__init__()
+        self.dropout = dropout
+        weight = torch.zeros(dim_out, dim_in)
+        bias = torch.zeros(dim_out) if use_bias else None
+        w_std = (1 / dim_in) if is_first else (math.sqrt(c / dim_in) / w0)
+        weight.uniform_(-w_std, w_std)
+        if _exists(bias):
+            bias.uniform_(-w_std, w_std)
+        self.weight = nn.Parameter(weight)
+        self.bias = nn.Parameter(bias) if use_bias else None
+        self.activation = Sine(w0) if activation is None else activation
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        out = F.linear(x, self.weight, self.bias)
+        if self.dropout:
+            out = F.dropout(out, training=self.training)
+        return self.activation(out)
+
+
+class SirenNet(nn.Module):
+    def __init__(self, dim_in: int, dim_hidden: int, dim_out: int, num_layers: int, w0: float = 1.0,
+                 w0_initial: float = 30.0, use_bias: bool = True, dropout: bool = True):
+        super().__init__()
+        self.layers = nn.ModuleList()
+        for i in range(num_layers):
+            is_first = i == 0
+            layer_w0 = w0_initial if is_first else w0
+            layer_dim_in = dim_in if is_first else dim_hidden
+            self.layers.append(SirenLayer(layer_dim_in, dim_hidden, w0=layer_w0, use_bias=use_bias,
+                is_first=is_first, dropout=dropout))
+        self.last_layer = SirenLayer(dim_hidden, dim_out, w0=w0, use_bias=use_bias,
+            activation=nn.Identity(), dropout=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        for layer in self.layers:
+            x = layer(x)
+        return self.last_layer(x)
+
+
+class SatCLIPLocationEncoder(ModelMixin, ConfigMixin):
+    """[lon, lat] -> 256-dim embedding. Load via from_pretrained() or from_satclip_ckpt()."""
+
+    @register_to_config
+    def __init__(self, legendre_polys: int = 10, embed_dim: int = 256, dim_hidden: int = 512, num_hidden_layers: int = 2):
+        super().__init__()
+        self.posenc = SphericalHarmonicsEncoder(legendre_polys=legendre_polys)
+        self.nnet = SirenNet(
+            dim_in=self.posenc.embedding_dim,
+            dim_hidden=dim_hidden,
+            dim_out=embed_dim,
+            num_layers=num_hidden_layers,
+            dropout=True,
+        )
+
+    def forward(self, lonlat: torch.Tensor) -> torch.Tensor:
+        if lonlat.dtype != torch.float64:
+            lonlat = lonlat.double()
+        x = self.posenc(lonlat).float()
+        x = x.to(next(self.nnet.parameters()).dtype)
+        return self.nnet(x)
+
+    @classmethod
+    def from_satclip_ckpt(cls, ckpt_path: str, device: Optional[Union[str, torch.device]] = None,
+                         dtype: Optional[torch.dtype] = None) -> "SatCLIPLocationEncoder":
+        ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
+        hp = ckpt.get("hyper_parameters", {})
+        state_dict = ckpt.get("state_dict", ckpt)
+        model = cls(
+            legendre_polys=hp.get("legendre_polys", 10),
+            embed_dim=hp.get("embed_dim", 256),
+            dim_hidden=hp.get("capacity", 512),
+            num_hidden_layers=hp.get("num_hidden_layers", 2),
+        )
+        nnet_sd = {}
+        for k, v in state_dict.items():
+            if "location" in k and "nnet" in k:
+                new_k = k[k.index("nnet") + 5:]
+                nnet_sd[new_k] = v
+        model.nnet.load_state_dict(nnet_sd, strict=True)
+        if device is not None:
+            model = model.to(device)
+        if dtype is not None:
+            model = model.to(dtype)
+        return model

geosynth_pipeline/spherical_harmonics.py

@@ -0,0 +1,75 @@
+"""
+Spherical harmonics encoding for (lon, lat) -> embedding.
+Adapted from Microsoft SatCLIP: https://github.com/microsoft/satclip
+"""
+import math
+import torch
+from torch import nn
+
+
+def associated_legendre_polynomial(l: int, m: int, x: torch.Tensor) -> torch.Tensor:
+    """Associated Legendre polynomial P_l^m(x)."""
+    pmm = torch.ones_like(x)
+    if m > 0:
+        somx2 = torch.sqrt((1 - x) * (1 + x))
+        fact = 1.0
+        for i in range(1, m + 1):
+            pmm = pmm * (-fact) * somx2
+            fact += 2.0
+    if l == m:
+        return pmm
+    pmmp1 = x * (2.0 * m + 1.0) * pmm
+    if l == m + 1:
+        return pmmp1
+    pll = torch.zeros_like(x)
+    for ll in range(m + 2, l + 1):
+        pll = ((2.0 * ll - 1.0) * x * pmmp1 - (ll + m - 1.0) * pmm) / (ll - m)
+        pmm = pmmp1
+        pmmp1 = pll
+    return pll
+
+
+def sh_renormalization(l: int, m: int) -> float:
+    return math.sqrt((2.0 * l + 1.0) * math.factorial(l - m) / (4 * math.pi * math.factorial(l + m)))
+
+
+def sh_basis(m: int, l: int, phi: torch.Tensor, theta: torch.Tensor) -> torch.Tensor:
+    """Spherical harmonic Y_l^m(theta, phi). phi=lon, theta=lat (in rad)."""
+    if m == 0:
+        return sh_renormalization(l, m) * associated_legendre_polynomial(l, m, torch.cos(theta))
+    elif m > 0:
+        return (
+            math.sqrt(2.0)
+            * sh_renormalization(l, m)
+            * torch.cos(m * phi)
+            * associated_legendre_polynomial(l, m, torch.cos(theta))
+        )
+    else:
+        return (
+            math.sqrt(2.0)
+            * sh_renormalization(l, -m)
+            * torch.sin(-m * phi)
+            * associated_legendre_polynomial(l, -m, torch.cos(theta))
+        )
+
+
+class SphericalHarmonicsEncoder(nn.Module):
+    """Encode (lon, lat) in degrees to spherical harmonics. Output dim = L * M with L=M=legendre_polys."""
+
+    def __init__(self, legendre_polys: int = 10):
+        super().__init__()
+        self.L = self.M = int(legendre_polys)
+        self.embedding_dim = self.L * self.M
+
+    def forward(self, lonlat: torch.Tensor) -> torch.Tensor:
+        lon, lat = lonlat[:, 0], lonlat[:, 1]
+        phi = torch.deg2rad(lon + 180)
+        theta = torch.deg2rad(lat + 90)
+        Y = []
+        for l in range(self.L):
+            for m in range(-l, l + 1):
+                y = sh_basis(m, l, phi, theta)
+                if isinstance(y, float):
+                    y = y * torch.ones_like(phi)
+                Y.append(y)
+        return torch.stack(Y, dim=-1)

inference_demo.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""
+GeoSynth inference demo with full location conditioning.
+Uses only paths relative to this repo — no external code or models.
+
+Requirements:
+  - Run convert_ckpts.py first to populate satclip_location_encoder/ and coordnet/
+  - diffusers, torch, transformers, pillow
+"""
+import argparse
+import os
+import sys
+
+REPO_DIR = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, REPO_DIR)
+
+import torch
+from PIL import Image
+from geosynth_pipeline import load_geosynth_pipeline_with_location, run_with_location
+
+DEFAULT_LON = -90.2
+DEFAULT_LAT = 38.6
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--control", required=True, help="Path to control image (OSM or Canny)")
+    parser.add_argument("--model", default=REPO_DIR, help="Model path (default: this repo)")
+    parser.add_argument("--control_type", choices=["OSM", "Canny"], default="OSM")
+    parser.add_argument("--prompt", default="Satellite image features a city neighborhood")
+    parser.add_argument("--lon", type=float, default=DEFAULT_LON)
+    parser.add_argument("--lat", type=float, default=DEFAULT_LAT)
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--steps", type=int, default=20)
+    parser.add_argument("--out", default=None, help="Output path")
+    args = parser.parse_args()
+
+    # All paths relative to this repo
+    model_path = args.model if os.path.isabs(args.model) else REPO_DIR
+    satclip_path = os.path.join(REPO_DIR, "satclip_location_encoder")
+    coordnet_path = os.path.join(REPO_DIR, "coordnet")
+
+    if not os.path.isdir(satclip_path) or not os.path.isfile(os.path.join(satclip_path, "config.json")):
+        sys.exit("Run convert_ckpts.py first to create satclip_location_encoder/")
+    if not os.path.isdir(coordnet_path) or not os.path.isfile(os.path.join(coordnet_path, "config.json")):
+        sys.exit("Run convert_ckpts.py first to create coordnet/")
+
+    print("Loading pipeline (local paths only)...")
+    pipe = load_geosynth_pipeline_with_location(
+        model_path,
+        controlnet_subfolder=f"controlnet/GeoSynth-Location-{args.control_type}",
+        satclip_path=satclip_path,
+        coordnet_path=coordnet_path,
+        local_files_only=True,
+    )
+    pipe = pipe.to("cuda")
+
+    img = Image.open(args.control).convert("RGB").resize((512, 512))
+    generator = torch.manual_seed(args.seed)
+
+    print(f"Generating (lon={args.lon}, lat={args.lat})...")
+    output = run_with_location(
+        pipe, args.prompt, image=img, lon=args.lon, lat=args.lat,
+        num_inference_steps=args.steps, generator=generator,
+    )
+
+    out_path = args.out
+    if not out_path:
+        os.makedirs(os.path.join(REPO_DIR, "demo_outputs"), exist_ok=True)
+        base = os.path.splitext(os.path.basename(args.control))[0]
+        out_path = os.path.join(REPO_DIR, "demo_outputs", f"output_{base}_lon{args.lon}_lat{args.lat}.jpg")
+    output.images[0].save(out_path)
+    print(f"Saved: {out_path}")
+
+
+if __name__ == "__main__":
+    main()

model_index.json

@@ -0,0 +1,37 @@
+{
+  "_class_name": "StableDiffusionPipeline",
+  "_diffusers_version": "0.37.0",
+  "feature_extractor": [
+    "transformers",
+    "CLIPImageProcessor"
+  ],
+  "image_encoder": [
+    null,
+    null
+  ],
+  "requires_safety_checker": false,
+  "safety_checker": [
+    null,
+    null
+  ],
+  "scheduler": [
+    "diffusers",
+    "DPMSolverMultistepScheduler"
+  ],
+  "text_encoder": [
+    "transformers",
+    "CLIPTextModel"
+  ],
+  "tokenizer": [
+    "transformers",
+    "CLIPTokenizer"
+  ],
+  "unet": [
+    "diffusers",
+    "UNet2DConditionModel"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKL"
+  ]
+}

satclip_location_encoder/config.json

@@ -0,0 +1,8 @@
+{
+  "_class_name": "SatCLIPLocationEncoder",
+  "_diffusers_version": "0.37.0",
+  "dim_hidden": 512,
+  "embed_dim": 256,
+  "legendre_polys": 10,
+  "num_hidden_layers": 2
+}

satclip_location_encoder/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a04fc24fd0edf58754130c1cfcd75a74274476a010c509b5ee23191c3b9bcb5
+size 1783336

scheduler/scheduler_config.json

@@ -0,0 +1,36 @@
+{
+  "_class_name": "DPMSolverMultistepScheduler",
+  "_diffusers_version": "0.37.0",
+  "algorithm_type": "dpmsolver++",
+  "beta_end": 0.085,
+  "beta_schedule": "scaled_linear",
+  "beta_start": 0.02,
+  "clip_sample": false,
+  "dynamic_thresholding_ratio": 0.995,
+  "euler_at_final": false,
+  "final_sigmas_type": "zero",
+  "flow_shift": 1.0,
+  "interpolation_type": "linear",
+  "lambda_min_clipped": -Infinity,
+  "lower_order_final": true,
+  "num_train_timesteps": 1000,
+  "prediction_type": "v_prediction",
+  "rescale_betas_zero_snr": false,
+  "sample_max_value": 1.0,
+  "set_alpha_to_one": false,
+  "skip_prk_steps": true,
+  "solver_order": 2,
+  "solver_type": "midpoint",
+  "steps_offset": 1,
+  "thresholding": false,
+  "time_shift_type": "exponential",
+  "timestep_spacing": "leading",
+  "trained_betas": null,
+  "use_beta_sigmas": false,
+  "use_dynamic_shifting": false,
+  "use_exponential_sigmas": false,
+  "use_flow_sigmas": false,
+  "use_karras_sigmas": false,
+  "use_lu_lambdas": false,
+  "variance_type": null
+}

text_encoder/config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "CLIPTextModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dropout": 0.0,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_size": 1024,
+  "initializer_factor": 1.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 77,
+  "model_type": "clip_text_model",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 23,
+  "pad_token_id": 1,
+  "projection_dim": 512,
+  "transformers_version": "5.3.0",
+  "vocab_size": 49408
+}

text_encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67e013543d4fac905c882e2993d86a2d454ee69dc9e8f37c0c23d33a48959d15
+size 1361596304

tokenizer/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "do_lower_case": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": true,
+  "model_max_length": 77,
+  "pad_token": "!",
+  "tokenizer_class": "CLIPTokenizer",
+  "unk_token": "<|endoftext|>"
+}

unet/config.json

@@ -0,0 +1,72 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.37.0",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "attention_type": "default",
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "class_embed_type": null,
+  "class_embeddings_concat": false,
+  "conv_in_kernel": 3,
+  "conv_out_kernel": 3,
+  "cross_attention_dim": 1024,
+  "cross_attention_norm": null,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "dropout": 0.0,
+  "dual_cross_attention": false,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_only_cross_attention": null,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "out_channels": 4,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_out_scale_factor": 1.0,
+  "resnet_skip_time_act": false,
+  "resnet_time_scale_shift": "default",
+  "reverse_transformer_layers_per_block": null,
+  "sample_size": 96,
+  "time_cond_proj_dim": null,
+  "time_embedding_act_fn": null,
+  "time_embedding_dim": null,
+  "time_embedding_type": "positional",
+  "timestep_post_act": null,
+  "transformer_layers_per_block": 1,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ],
+  "upcast_attention": true,
+  "use_linear_projection": true
+}

unet/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f349f136057c7ddefbf454ba56b1f9a84ae15436b9b2e677a6590aa966f1db64
+size 3463726504

vae/config.json

@@ -0,0 +1,38 @@
+{
+  "_class_name": "AutoencoderKL",
+  "_diffusers_version": "0.37.0",
+  "_name_or_path": "/home/s.sastry/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2-1-base/snapshots/5ede9e4bf3e3fd1cb0ef2f7a3fff13ee514fdf06/vae",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    512,
+    512
+  ],
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "force_upcast": true,
+  "in_channels": 3,
+  "latent_channels": 4,
+  "latents_mean": null,
+  "latents_std": null,
+  "layers_per_block": 2,
+  "mid_block_add_attention": true,
+  "norm_num_groups": 32,
+  "out_channels": 3,
+  "sample_size": 768,
+  "scaling_factor": 0.18215,
+  "shift_factor": null,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ],
+  "use_post_quant_conv": true,
+  "use_quant_conv": true
+}

vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2aa1f43011b553a4cba7f37456465cdbd48aab7b54b9348b890e8058ea7683ec
+size 334643268