Initial release: VectorSynth-GiT10M

README.md

@@ -0,0 +1,100 @@
+---
+license: apache-2.0
+tags:
+  - controlnet
+  - stable-diffusion
+  - satellite-imagery
+  - osm
+  - image-to-image
+  - diffusers
+base_model: stabilityai/stable-diffusion-2-1-base
+pipeline_tag: image-to-image
+library_name: diffusers
+---
+
+# VectorSynth-GiT10M
+
+**VectorSynth-GiT10M** is a ControlNet-based pipeline that generates satellite imagery from OpenStreetMap (OSM) vector data, fine-tuned on the GiT10M dataset of paired OSM + satellite tiles. Like [VectorSynth-COSA](https://huggingface.co/MVRL/VectorSynth-COSA), it conditions [Stable Diffusion 2.1 Base](https://huggingface.co/stabilityai/stable-diffusion-2-1-base) on rendered OSM text using the COSA (Contrastive OSM-Satellite Alignment) embedding space.
+
+## Model Description
+
+VectorSynth-GiT10M uses a two-stage pipeline:
+1. **RenderEncoder**: Projects 768-dim COSA embeddings to 3-channel control images.
+2. **ControlNet + UNet**: Both fine-tuned on the GiT10M dataset to condition Stable Diffusion 2.1 on the rendered control images.
+
+Unlike `VectorSynth-COSA` — which ships only a fine-tuned ControlNet on top of the stock SD 2.1 UNet — this model additionally fine-tunes the UNet on GiT10M, so users should load the full pipeline from this repo rather than from `stable-diffusion-2-1-base`.
+
+## Usage
+
+```python
+import sys
+import torch
+from diffusers import StableDiffusionControlNetPipeline, DDIMScheduler
+from huggingface_hub import snapshot_download
+
+device = "cuda"
+
+# Load pipeline (GiT10M-finetuned UNet + ControlNet, plus base SD 2.1 VAE/text encoder)
+local_dir = snapshot_download("MVRL/VectorSynth-GiT10M")
+pipe = StableDiffusionControlNetPipeline.from_pretrained(
+    local_dir,
+    torch_dtype=torch.float16
+)
+pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
+pipe = pipe.to(device)
+
+# Load RenderEncoder
+sys.path.insert(0, local_dir)
+from render import RenderEncoder
+checkpoint = torch.load(
+    f"{local_dir}/render_encoder/cosa-render_encoder.pth",
+    map_location=device, weights_only=False,
+)
+render_encoder = RenderEncoder(**checkpoint['config']).to(device).eval()
+render_encoder.load_state_dict(checkpoint['state_dict'])
+
+# Your hint tensor should be (H, W, 768) - per-pixel COSA embeddings
+# hint = torch.load("your_hint.pt").to(device)
+# hint = hint.unsqueeze(0).permute(0, 3, 1, 2)  # (1, 768, H, W)
+
+# with torch.no_grad():
+#     control_image = render_encoder(hint)
+
+# Generate
+# output = pipe(
+#     prompt="An aerial image of a residential neighborhood",
+#     image=control_image,
+#     num_inference_steps=40,
+#     guidance_scale=7.5
+# ).images[0]
+```
+
+## Files
+
+- `unet/` — GiT10M-fine-tuned UNet (`diffusion_pytorch_model.safetensors`)
+- `controlnet/` — GiT10M-fine-tuned ControlNet
+- `render_encoder/cosa-render_encoder.pth` — RenderEncoder weights (COSA 768→3)
+- `render.py` — RenderEncoder class definition
+- `vae/`, `text_encoder/`, `tokenizer/`, `scheduler/`, `feature_extractor/` — copied from SD 2.1 Base (unmodified)
+
+## Training Data
+
+Fine-tuned on **GiT10M**, a curated collection of paired OpenStreetMap vector data and Google satellite tiles (zoom 17, ~1m/pix). The dataset is split into a training set and two held-out test splits (random and spatial) for evaluation. See [GeoDiT: Point Conditioned Diffusion Transformer for Satellite Image Synthesis](https://arxiv.org/html/2603.02172v1) for more details on the data.
+
+## Citation
+
+```bibtex
+@inproceedings{cher2025vectorsynth,
+  title={VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics},
+  author={Cher, Daniel and Wei, Brian and Sastry, Srikumar and Jacobs, Nathan},
+  year={2025},
+  eprint={arXiv:2511.07744},
+  note={arXiv preprint}
+}
+```
+
+## Related Models
+
+- [VectorSynth-COSA](https://huggingface.co/MVRL/VectorSynth-COSA) — trained on smaller cities dataset
+- [VectorSynth](https://huggingface.co/MVRL/VectorSynth) — standard CLIP embedding variant
+- [GeoSynth](https://huggingface.co/MVRL/GeoSynth) — text-to-satellite image generation

controlnet/config.json

@@ -0,0 +1,56 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.34.0",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_channels": 3,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "transformer_layers_per_block": 1,
+  "upcast_attention": false,
+  "use_linear_projection": true
+}

controlnet/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:755bf692b7367e416649ee673a9f79a458eaa050454fc0797e11ac3f9f0feb96
+size 1456953560

feature_extractor/preprocessor_config.json

@@ -0,0 +1,20 @@
+{
+  "crop_size": 224,
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_resize": true,
+  "feature_extractor_type": "CLIPFeatureExtractor",
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "size": 224
+}

model_index.json

@@ -0,0 +1,37 @@
+{
+  "_class_name": "StableDiffusionControlNetPipeline",
+  "_diffusers_version": "0.27.2",
+  "controlnet": [
+    "diffusers",
+    "ControlNetModel"
+  ],
+  "feature_extractor": [
+    "transformers",
+    "CLIPImageProcessor"
+  ],
+  "requires_safety_checker": false,
+  "safety_checker": [
+    null,
+    null
+  ],
+  "scheduler": [
+    "diffusers",
+    "DDIMScheduler"
+  ],
+  "text_encoder": [
+    "transformers",
+    "CLIPTextModel"
+  ],
+  "tokenizer": [
+    "transformers",
+    "CLIPTokenizer"
+  ],
+  "unet": [
+    "diffusers",
+    "UNet2DConditionModel"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKL"
+  ]
+}

render.py

@@ -0,0 +1,73 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+class ResidualRenderBlock(nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.block = nn.Sequential(
+            nn.Conv2d(dim, dim, kernel_size=3, padding=1),
+            nn.GroupNorm(8, dim),
+            nn.SiLU(),
+            nn.Conv2d(dim, dim, kernel_size=3, padding=1),
+            nn.GroupNorm(8, dim)
+        )
+
+    def forward(self, x):
+        return x + self.block(x)
+
+class RenderEncoder(nn.Module):
+    def __init__(self, encoder_type="1d", in_channels=768, out_channels=3):
+        super().__init__()
+        self.encoder_type = encoder_type
+
+        if encoder_type == "1d":
+            self.model = nn.Sequential(
+                nn.Conv2d(in_channels, out_channels, kernel_size=1),
+                nn.Sigmoid()
+            )
+
+        elif encoder_type == "residual":
+            self.model = ResidualBlockRender(in_channels, out_channels)
+
+        elif encoder_type == "expressive":
+            mid_channels = 256
+            self.model = nn.Sequential(
+                nn.Conv2d(in_channels, mid_channels, kernel_size=3, padding=1),
+                nn.GroupNorm(8, mid_channels),
+                nn.SiLU(),
+                ResidualRenderBlock(mid_channels),
+                ResidualRenderBlock(mid_channels),
+                ResidualRenderBlock(mid_channels),
+                nn.Conv2d(mid_channels, out_channels, kernel_size=1),
+                nn.Sigmoid()
+            )
+
+        else:
+            raise ValueError(f"Unknown encoder_type '{encoder_type}'. Use '1d', 'residual', or 'expressive'.")
+
+    def forward(self, x):
+        return self.model(x)
+
+class ResidualBlockRender(nn.Module):
+    def __init__(self, in_channels=768, out_channels=3):
+        super().__init__()
+        self.conv1 = nn.Conv2d(in_channels, 256, kernel_size=3, padding=1)
+        self.relu1 = nn.ReLU()
+        self.conv2 = nn.Conv2d(256, 256, kernel_size=3, padding=1)
+        self.relu2 = nn.ReLU()
+        self.conv3 = nn.Conv2d(256, out_channels, kernel_size=1)
+        self.out = nn.Sigmoid()
+
+        if in_channels != out_channels:
+            self.residual_proj = nn.Conv2d(in_channels, out_channels, kernel_size=1)
+        else:
+            self.residual_proj = nn.Identity()
+
+    def forward(self, x):
+        residual = self.residual_proj(x)
+        h = self.relu1(self.conv1(x))
+        h = self.relu2(self.conv2(h))
+        h = self.conv3(h)
+        h = h + residual
+        return self.out(h)

render_encoder/cosa-render_encoder.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abb4d405f0fb319363943275d57870b4a5318b173d16ff8d6a1373929d6ea5ac
+size 10976

scheduler/scheduler_config.json

@@ -0,0 +1,14 @@
+{
+  "_class_name": "PNDMScheduler",
+  "_diffusers_version": "0.10.0.dev0",
+  "beta_end": 0.012,
+  "beta_schedule": "scaled_linear",
+  "beta_start": 0.00085,
+  "clip_sample": false,
+  "num_train_timesteps": 1000,
+  "prediction_type": "epsilon",
+  "set_alpha_to_one": false,
+  "skip_prk_steps": true,
+  "steps_offset": 1,
+  "trained_betas": null
+}

text_encoder/config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "stabilityai/stable-diffusion-2",
+  "architectures": [
+    "CLIPTextModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dropout": 0.0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_size": 1024,
+  "initializer_factor": 1.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 77,
+  "model_type": "clip_text_model",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 23,
+  "pad_token_id": 1,
+  "projection_dim": 512,
+  "torch_dtype": "float32",
+  "transformers_version": "4.25.0.dev0",
+  "vocab_size": 49408
+}

text_encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c
+size 1361597018

tokenizer/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "!",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer/tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "do_lower_case": true,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "model_max_length": 77,
+  "name_or_path": "stabilityai/stable-diffusion-2",
+  "pad_token": "<|endoftext|>",
+  "special_tokens_map_file": "./special_tokens_map.json",
+  "tokenizer_class": "CLIPTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

unet/config.json

@@ -0,0 +1,45 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.10.0.dev0",
+  "act_fn": "silu",
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "cross_attention_dim": 1024,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "dual_cross_attention": false,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "out_channels": 4,
+  "sample_size": 64,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ],
+  "use_linear_projection": true
+}

unet/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dfae3e5f7d459b50f4b0850ead945972c75bb0e1897628933e169eb43974214
+size 3463726498

vae/config.json

@@ -0,0 +1,29 @@
+{
+  "_class_name": "AutoencoderKL",
+  "_diffusers_version": "0.10.0.dev0",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    512,
+    512
+  ],
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "in_channels": 3,
+  "latent_channels": 4,
+  "layers_per_block": 2,
+  "norm_num_groups": 32,
+  "out_channels": 3,
+  "sample_size": 768,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ]
+}

vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815
+size 334643276