Upload RMBG folder recursively

.gitattributes

@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 RMBG/BiRefNet/__pycache__/birefnet.cpython-312.pyc filter=lfs diff=lfs merge=lfs -text
 RMBG/RMBG-2.0/__pycache__/birefnet.cpython-311.pyc filter=lfs diff=lfs merge=lfs -text
+SDPose_OOD/SDPose-Wholebody/assets/wholebody_anno.png filter=lfs diff=lfs merge=lfs -text

SDPose_OOD/SDPose-Wholebody/.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/wholebody_anno.png filter=lfs diff=lfs merge=lfs -text

SDPose_OOD/SDPose-Wholebody/README.md

@@ -0,0 +1,137 @@
+---
+language: en
+license: mit
+tags:
+- pose-estimation
+- computer-vision
+- keypoint-detection
+- diffusion-models
+- stable-diffusion
+- out-of-distribution
+- human-pose
+- top-down-pose-estimation
+- coco
+- mmpose
+library_name: pytorch
+---
+
+# SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation (WholeBody - 133 Keypoints)
+
+<div align="center">
+
+[![Paper](https://img.shields.io/badge/arXiv-Paper-b31b1b?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2509.24980)
+[![Project Page](https://img.shields.io/badge/Project-Website-pink?logo=googlechrome&logoColor=white)](https://t-s-liang.github.io/SDPose)
+[![HuggingFace Demo](https://img.shields.io/badge/🤗%20HuggingFace-Demo-yellow)](https://huggingface.co/spaces/teemosliang/SDPose-Body)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+
+</div>
+
+## Model Description
+
+**SDPose** is a state-of-the-art human pose estimation model that leverages the powerful visual priors from **Stable Diffusion** to achieve exceptional performance on out-of-distribution (OOD) scenarios. This model variant estimates **133 wholebody keypoints,** including body, hands, face, feet.
+
+### Model Architecture
+
+SDPose employs a **U-Net backbone** initialized with Stable Diffusion v2 weights, combined with a specialized heatmap head for keypoint prediction. The model operates in a top-down manner:
+
+1. **Person Detection**: Detect human bounding boxes using an object detector (e.g., YOLO11-x)
+2. **Pose Estimation**: Crop and estimate 17 body keypoints for each detected person
+3. **Heatmap Generation**: Produce confidence heatmaps for precise keypoint estimation
+
+**Model Specifications:**
+- **Backbone**: Stable Diffusion v2 U-Net (fine-tuned; minimal architectural changes)
+- **Head**: Custom heatmap prediction head
+- **Input Resolution**: 1024×768 (H×W)
+- **Output**: 133 keypoint heatmaps + coordinates with confidence scores
+- **Framework**: MMPose
+
+## Supported Keypoints (COCO Wholebody Format)
+
+The model predicts 133 body keypoints following the COCO Wholebody keypoint format.
+
+<p align="center">
+  <img src="assets/wholebody_anno.png" width="600"/>
+</p>
+
+## Intended Use
+
+- Human pose estimation in natural images
+- Pose estimation in artistic and stylized domains (paintings, anime, sketches)
+- Animation and video pose tracking
+- Cross-domain pose analysis and research
+- Applications requiring robust pose estimation under distribution shifts
+
+## How to Use
+
+### Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/t-s-liang/SDPose-OOD.git
+cd SDPose-OOD
+
+# Install dependencies
+pip install -r requirements.txt
+# Download YOLO11-x for human detection
+wget https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11x.pt -P models/
+
+# Launch Gradio interface
+cd gradio_app
+bash launch_gradio.sh
+```
+
+## Training Data
+
+### Datasets
+
+Trained exclusively on COCO-2017 train2017 (no extra data).
+
+- **COCO-Wholebody (Common Objects in Context)**: 200K+ images with 133 wholebody keypoints
+
+### Preprocessing
+
+- Images are resized and cropped to 1024×768 resolution
+- Augmentation: random horizontal flip, half-body & bbox transforms, UDP affine; Albumentations (Gaussian/Median blur, coarse dropout).
+- Heatmaps: UDP codec (MMPose style).
+
+### Comparison with Baselines
+
+SDPose significantly outperforms traditional pose estimation models (e.g., Sapiens) on out-of-distribution benchmarks while maintaining competitive performance on in-domain data.
+
+See our [paper](https://arxiv.org/abs/2509.24980) for comprehensive evaluation results.
+
+## Citation
+
+If you use SDPose in your research, please cite our paper:
+
+```bibtex
+@misc{liang2025sdposeexploitingdiffusionpriors,
+      title={SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation}, 
+      author={Shuang Liang and Jing He and Chuanmeizhi Wang and Lejun Liao and Guo Zhang and Yingcong Chen and Yuan Yuan},
+      year={2025},
+      eprint={2509.24980},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2509.24980}, 
+}
+```
+
+## License
+
+This model is released under the [MIT License](https://opensource.org/licenses/MIT).
+
+## Additional Resources
+
+- 🌐 **Project Website**: [https://t-s-liang.github.io/SDPose](https://t-s-liang.github.io/SDPose)
+- 📄 **Paper**: [arXiv:2509.24980](https://arxiv.org/abs/2509.24980)
+- 💻 **Code Repository**: [GitHub](https://github.com/t-s-liang/SDPose-OOD)
+- 🤗 **Demo**: [HuggingFace Space](https://huggingface.co/spaces/teemosliang/SDPose-Body)
+- 📧 **Contact**: tsliang2001@gmail.com
+
+---
+
+<div align="center">
+
+**⭐ Star us on GitHub — it motivates us a lot!**
+
+</div>

SDPose_OOD/SDPose-Wholebody/decoder/decoder.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ee297f62ade11c405ad56ae490aa1bef7be881df794f145ceb8e12e38a853fc
+size 28196828

SDPose_OOD/SDPose-Wholebody/huggingface/.gitignore

@@ -0,0 +1 @@
+*

SDPose_OOD/SDPose-Wholebody/huggingface/download/.gitattributes.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+6ecc7d0c41955e6c984d4ca14753fe548c961e68
+1771726775.832536

SDPose_OOD/SDPose-Wholebody/huggingface/download/README.md.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+e376a63f46becfd4ac7005225a69fbb1b07f2dbd
+1771726775.9407606

SDPose_OOD/SDPose-Wholebody/huggingface/download/assets/wholebody_anno.png.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+5ab1654bde9ccd55d2eb32e5728e6c909a8fc7041f548eb76e1594f3b455d891
+1771726776.1980946

SDPose_OOD/SDPose-Wholebody/huggingface/download/decoder/decoder.safetensors.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+3ee297f62ade11c405ad56ae490aa1bef7be881df794f145ceb8e12e38a853fc
+1771726777.472846

SDPose_OOD/SDPose-Wholebody/huggingface/download/scheduler/scheduler_config.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+ab4573d2098886789be83ac32d48b9737edb2830
+1771726775.8193738

SDPose_OOD/SDPose-Wholebody/huggingface/download/text_encoder/config.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+9c60528fdcb99a7caf834426a94ea13c56cf422b
+1771726775.8028762

SDPose_OOD/SDPose-Wholebody/huggingface/download/text_encoder/model.safetensors.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c
+1771726847.5179892

SDPose_OOD/SDPose-Wholebody/huggingface/download/tokenizer/merges.txt.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+76e821f1b6f0a9709293c3b6b51ed90980b3166b
+1771726776.1691403

SDPose_OOD/SDPose-Wholebody/huggingface/download/tokenizer/special_tokens_map.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+ae0c5be6f35217e51c4c000fd325d8de0294e99c
+1771726776.6460805

SDPose_OOD/SDPose-Wholebody/huggingface/download/tokenizer/tokenizer_config.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+f4fe219b936c0e171504b4bba0c33c7bef6ea211
+1771726776.650374

SDPose_OOD/SDPose-Wholebody/huggingface/download/tokenizer/vocab.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+469be27c5c010538f845f518c4f5e8574c78f7c8
+1771726777.2009902

SDPose_OOD/SDPose-Wholebody/huggingface/download/unet/config.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+466532dc49c3a8bc2fe21ed4e640ca124278b00b
+1771726776.8096793

SDPose_OOD/SDPose-Wholebody/huggingface/download/unet/diffusion_pytorch_model.safetensors.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+e951ac1802f004243bbbea6f0d86abfa08360776424f2d228c62ed63c5b1ab66
+1771726911.1530993

SDPose_OOD/SDPose-Wholebody/huggingface/download/vae/config.json.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+f97af6a6a8235236b1346312f328569ce2d70f81
+1771726777.0636213

SDPose_OOD/SDPose-Wholebody/huggingface/download/vae/diffusion_pytorch_model.safetensors.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815
+1771726805.619076

SDPose_OOD/SDPose-Wholebody/huggingface/download/yolo11x.pt.metadata

@@ -0,0 +1,3 @@
+0c8153e39458bc8ae87fb68790c141bf0f03c11b
+7bc158aa95c0ebfdd87f70f01653c1131b93e92522dbe15c228bcd742e773a24
+1771726788.7594335

SDPose_OOD/SDPose-Wholebody/scheduler/scheduler_config.json

@@ -0,0 +1,14 @@
+{
+  "_class_name": "DDPMScheduler",
+  "_diffusers_version": "0.28.0.dev0",
+  "beta_end": 0.012,
+  "beta_schedule": "scaled_linear",
+  "beta_start": 0.00085,
+  "clip_sample": false,
+  "num_train_timesteps": 1000,
+  "prediction_type": "sample",
+  "set_alpha_to_one": false,
+  "skip_prk_steps": true,
+  "steps_offset": 1,
+  "trained_betas": null
+}

SDPose_OOD/SDPose-Wholebody/text_encoder/config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "hf-models/stable-diffusion-v2-768x768/text_encoder",
+  "architectures": [
+    "CLIPTextModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dropout": 0.0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_size": 1024,
+  "initializer_factor": 1.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 77,
+  "model_type": "clip_text_model",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 23,
+  "pad_token_id": 1,
+  "projection_dim": 512,
+  "torch_dtype": "float32",
+  "transformers_version": "4.25.0.dev0",
+  "vocab_size": 49408
+}

SDPose_OOD/SDPose-Wholebody/text_encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c
+size 1361597018

SDPose_OOD/SDPose-Wholebody/tokenizer/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "!",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

SDPose_OOD/SDPose-Wholebody/tokenizer/tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "do_lower_case": true,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "model_max_length": 77,
+  "name_or_path": "hf-models/stable-diffusion-v2-768x768/tokenizer",
+  "pad_token": "<|endoftext|>",
+  "special_tokens_map_file": "./special_tokens_map.json",
+  "tokenizer_class": "CLIPTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

SDPose_OOD/SDPose-Wholebody/unet/config.json

@@ -0,0 +1,73 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.28.0",
+  "_name_or_path": "/data/coding/model",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": [
+    5,
+    10,
+    20,
+    20
+  ],
+  "attention_type": "default",
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "class_embed_type": "projection",
+  "class_embeddings_concat": false,
+  "conv_in_kernel": 3,
+  "conv_out_kernel": 3,
+  "cross_attention_dim": 1024,
+  "cross_attention_norm": null,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "dropout": 0.0,
+  "dual_cross_attention": false,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_only_cross_attention": null,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "out_channels": 4,
+  "projection_class_embeddings_input_dim": 4,
+  "resnet_out_scale_factor": 1.0,
+  "resnet_skip_time_act": false,
+  "resnet_time_scale_shift": "default",
+  "reverse_transformer_layers_per_block": null,
+  "sample_size": 96,
+  "time_cond_proj_dim": null,
+  "time_embedding_act_fn": null,
+  "time_embedding_dim": null,
+  "time_embedding_type": "positional",
+  "timestep_post_act": null,
+  "transformer_layers_per_block": 1,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ],
+  "upcast_attention": false,
+  "use_linear_projection": true
+}

SDPose_OOD/SDPose-Wholebody/unet/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e951ac1802f004243bbbea6f0d86abfa08360776424f2d228c62ed63c5b1ab66
+size 3470311272

SDPose_OOD/SDPose-Wholebody/vae/config.json

@@ -0,0 +1,30 @@
+{
+  "_class_name": "AutoencoderKL",
+  "_diffusers_version": "0.8.0",
+  "_name_or_path": "hf-models/stable-diffusion-v2-768x768/vae",
+  "act_fn": "silu",
+  "block_out_channels": [
+    128,
+    256,
+    512,
+    512
+  ],
+  "down_block_types": [
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D",
+    "DownEncoderBlock2D"
+  ],
+  "in_channels": 3,
+  "latent_channels": 4,
+  "layers_per_block": 2,
+  "norm_num_groups": 32,
+  "out_channels": 3,
+  "sample_size": 768,
+  "up_block_types": [
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D",
+    "UpDecoderBlock2D"
+  ]
+}

SDPose_OOD/SDPose-Wholebody/vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815
+size 334643276

SDPose_OOD/SDPose-Wholebody/yolo11x.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bc158aa95c0ebfdd87f70f01653c1131b93e92522dbe15c228bcd742e773a24
+size 114636239