Upload RSBuilding-ViT-B

README.md

@@ -0,0 +1,186 @@
+---
+license: mit
+tags:
+- remote-sensing
+- computer-vision
+- vision-transformer
+- sam
+- building-extraction
+- change-detection
+- foundation-model
+datasets:
+- remote-sensing-images
+model-index:
+- name: RSBuilding-ViT-B
+  results: []
+library_name: transformers
+pipeline_tag: feature-extraction
+---
+
+# RSBuilding-ViT-B
+
+HuggingFace Transformers version of RSBuilding ViT-Base model (ViTSAM_Normal), converted from MMCV format to SamVisionModel format.
+
+## Source
+
+- **Source Code**: [https://github.com/Meize0729/RSBuilding](https://github.com/Meize0729/RSBuilding)
+- **Original Checkpoint**: [https://huggingface.co/models/BiliSakura/RSBuilding](https://huggingface.co/models/BiliSakura/RSBuilding)
+
+## Model Information
+
+- **Architecture**: Vision Transformer Base (SAM-style)
+- **Hidden Size**: 768
+- **Number of Layers**: 12
+- **Number of Attention Heads**: 12
+- **MLP Dimension**: 3072
+- **Image Size**: 512×512
+- **Patch Size**: 16×16
+- **Window Size**: 7
+- **Global Attention Indexes**: [2, 5, 8, 11]
+
+## Important Notes
+
+### Missing Neck Module Keys (Expected)
+
+When loading this model, you may see messages about missing neck module keys (typically ~6 keys). **This is expected and normal.**
+
+**What is the neck module?**
+- The neck module is a channel reduction layer that reduces ViT output from 768 channels to 256 channels
+- It consists of: Conv1x1 → LayerNorm → Conv3x3 → LayerNorm
+- Purpose: Improves efficiency and prepares features for downstream tasks (mask decoder, etc.)
+
+**Why they're missing:**
+- The source checkpoint (ViTSAM_Normal) may not include neck/channel_reduction weights
+- The HuggingFace SamVisionModel expects a neck module as part of its architecture
+- Missing neck weights will be initialized using HuggingFace's default initialization
+
+**Action required:** 
+- For inference: The model will work, but you may want to fine-tune the neck module on your downstream task
+- For best results: Consider initializing neck weights from a pretrained SAM checkpoint or fine-tuning them
+
+### Missing Buffer Keys (Expected)
+
+You may also see messages about missing buffer keys. These are buffers computed dynamically:
+- `relative_position_index`: Precomputed index mapping for window attention
+- `relative_coords_table`: Precomputed coordinate table
+
+**Action required:** None. These are computed automatically during initialization.
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install transformers torch pillow
+```
+
+### Inference Example
+
+```python
+from transformers import SamVisionModel, AutoImageProcessor
+from PIL import Image
+import torch
+
+# Load model and processor
+model = SamVisionModel.from_pretrained("BiliSakura/RSBuilding-ViT-B")
+processor = AutoImageProcessor.from_pretrained("BiliSakura/RSBuilding-ViT-B")
+
+# Load and process image
+image = Image.open("your_image.jpg")
+inputs = processor(image, return_tensors="pt")
+
+# Forward pass
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# Get features
+# outputs.last_hidden_state: (batch_size, num_patches, hidden_size)
+# outputs.pooler_output: (batch_size, hidden_size) - pooled representation
+features = outputs.last_hidden_state
+pooled_features = outputs.pooler_output
+
+print(f"Feature shape: {features.shape}")
+print(f"Pooled feature shape: {pooled_features.shape}")
+```
+
+### Feature Extraction for Downstream Tasks
+
+```python
+from transformers import SamVisionModel, AutoImageProcessor
+import torch
+
+model = SamVisionModel.from_pretrained("BiliSakura/RSBuilding-ViT-B")
+processor = AutoImageProcessor.from_pretrained("BiliSakura/RSBuilding-ViT-B")
+
+# Process image
+image = Image.open("your_image.jpg")
+inputs = processor(image, return_tensors="pt")
+
+# Extract features
+with torch.no_grad():
+    outputs = model(**inputs)
+    
+# Use pooled features for classification/regression
+features = outputs.pooler_output  # Shape: (1, 768)
+
+# Or use last hidden state for dense prediction tasks
+spatial_features = outputs.last_hidden_state  # Shape: (1, num_patches, 768)
+
+# Access neck output (after channel reduction to 256)
+# Note: This requires accessing model internals
+neck_output = model.vision_encoder.neck(outputs.last_hidden_state)  # Shape: (1, 256, H, W)
+```
+
+### Fine-tuning the Neck Module
+
+If you need to fine-tune the neck module:
+
+```python
+from transformers import SamVisionModel
+import torch
+
+model = SamVisionModel.from_pretrained("BiliSakura/RSBuilding-ViT-B")
+
+# Option 1: Freeze backbone, train only neck
+for param in model.vision_encoder.encoder.parameters():
+    param.requires_grad = False
+for param in model.vision_encoder.neck.parameters():
+    param.requires_grad = True
+
+# Option 2: Initialize neck from pretrained SAM
+from transformers import SamVisionModel as PretrainedSAM
+pretrained_sam = PretrainedSAM.from_pretrained("facebook/sam-vit-base")
+model.vision_encoder.neck.load_state_dict(pretrained_sam.vision_encoder.neck.state_dict())
+```
+
+## Model Configuration
+
+The model uses the following configuration:
+- `hidden_size`: 768
+- `num_hidden_layers`: 12
+- `num_attention_heads`: 12
+- `mlp_dim`: 3072
+- `image_size`: 512
+- `patch_size`: 16
+- `window_size`: 7
+- `output_channels`: 256 (neck output)
+- `global_attn_indexes`: [2, 5, 8, 11]
+
+## Citation
+
+If you use this model, please cite the original RSBuilding paper:
+
+```bibtex
+@article{wangRSBuildingGeneralRemote2024a,
+  title = {{{RSBuilding}}: {{Toward General Remote Sensing Image Building Extraction}} and {{Change Detection With Foundation Model}}},
+  shorttitle = {{{RSBuilding}}},
+  author = {Wang, Mingze and Su, Lili and Yan, Cilin and Xu, Sheng and Yuan, Pengcheng and Jiang, Xiaolong and Zhang, Baochang},
+  year = {2024},
+  journal = {IEEE Transactions on Geoscience and Remote Sensing},
+  volume = {62},
+  pages = {1--17},
+  issn = {1558-0644},
+  doi = {10.1109/TGRS.2024.3439395},
+  keywords = {Building extraction,Buildings,change detection (CD),Data mining,Feature extraction,federated training,foundation model,Image segmentation,Remote sensing,remote sensing images,Task analysis,Training}
+}
+```

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "SamVisionModel"
+  ],
+  "attention_dropout": 0.0,
+  "dtype": "float32",
+  "global_attn_indexes": [
+    2,
+    5,
+    8,
+    11
+  ],
+  "hidden_act": "gelu",
+  "hidden_size": 768,
+  "image_size": 512,
+  "initializer_range": 1e-10,
+  "layer_norm_eps": 1e-06,
+  "mlp_dim": 3072,
+  "mlp_ratio": 4.0,
+  "model_type": "sam_vision_model",
+  "num_attention_heads": 12,
+  "num_channels": 3,
+  "num_hidden_layers": 12,
+  "num_pos_feats": 128,
+  "output_channels": 256,
+  "patch_size": 16,
+  "qkv_bias": true,
+  "transformers_version": "5.0.0.dev0",
+  "use_abs_pos": true,
+  "use_rel_pos": true,
+  "window_size": 7
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:300505439db2b5847c04fd7dce28e3788f0510ff7de44db5f3c77c2205e6f857
+size 349077576

preprocessor_config.json

@@ -0,0 +1,23 @@
+{
+  "image_processor_type": "SamImageProcessor",
+  "do_resize": true,
+  "size": {
+    "longest_edge": 512
+  },
+  "resample": 2,
+  "do_rescale": true,
+  "rescale_factor": 0.00392156862745098,
+  "do_normalize": true,
+  "image_mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "image_std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "do_convert_rgb": true,
+  "do_pad": false
+}