Instructions to use Sunmj/IRecon-MCM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use Sunmj/IRecon-MCM with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("Sunmj/IRecon-MCM", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Upload IRecon MCM inference adapter
Browse files- README.md +47 -0
- mcm_fusion_128.pt +3 -0
- mcm_fusion_128.sha256 +1 -0
README.md
CHANGED
|
@@ -1,3 +1,50 @@
|
|
| 1 |
---
|
| 2 |
license: creativeml-openrail-m
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: creativeml-openrail-m
|
| 3 |
+
library_name: diffusers
|
| 4 |
+
pipeline_tag: image-to-image
|
| 5 |
+
tags:
|
| 6 |
+
- infrared
|
| 7 |
+
- image-fusion
|
| 8 |
+
- adapter
|
| 9 |
+
- stable-diffusion
|
| 10 |
+
base_model: runwayml/stable-diffusion-inpainting
|
| 11 |
---
|
| 12 |
+
|
| 13 |
+
# IRecon MCM 128
|
| 14 |
+
|
| 15 |
+
Inference-only modulation and conditioning module for registered SWIR, MWIR, and LWIR fusion.
|
| 16 |
+
|
| 17 |
+
## Architecture
|
| 18 |
+
|
| 19 |
+
- 12 input channels: noisy latent, frozen base noise prediction, three bands, and object mask.
|
| 20 |
+
- 96 hidden channels, six time-conditioned residual blocks.
|
| 21 |
+
- 8 outputs: four noise residual channels and four spatial gates.
|
| 22 |
+
- 1,371,080 trainable parameters.
|
| 23 |
+
- Frozen Stable Diffusion 1.5 inpainting base, 10-step DDIM inference at 128 x 128.
|
| 24 |
+
|
| 25 |
+
## Training provenance
|
| 26 |
+
|
| 27 |
+
- Object-disjoint Objaverse simulation.
|
| 28 |
+
- Direct online-noise training from a randomly initialized MCM; no clean pretraining is contained in this checkpoint.
|
| 29 |
+
- Selected checkpoint: cumulative step 145,000, training seed 3409.
|
| 30 |
+
- Validation masked image MAE at selection: 0.1384654571.
|
| 31 |
+
- Loss: diffusion noise MSE + 0.1 image Smooth-L1 + 0.05 first-gradient loss.
|
| 32 |
+
- Online perturbation: band-dependent global noise and local spherical-region noise.
|
| 33 |
+
- Global noise sigma (SWIR/MWIR/LWIR): 0.08 / 0.025 / 0.01.
|
| 34 |
+
- Local spherical-region sigma (SWIR/MWIR/LWIR): 0.20 / 0.10 / 0.05.
|
| 35 |
+
|
| 36 |
+
The original checkpoint also contained optimizer and scaler state. The released file strips those states and stores only the MCM state dict and non-identifying inference metadata.
|
| 37 |
+
|
| 38 |
+
## Evaluation boundary
|
| 39 |
+
|
| 40 |
+
The thesis reports an MAE of 0.21288, PSNR of 13.9054 dB, and SSIM of 0.54345 for one validation-selected checkpoint averaged over four inference perturbation/sampling seeds. Independent training-seed results are reported separately. The model improves simulated intensity restoration over fixed linear fusion but does not yet improve the downstream GSO pose/mesh initializer over noisy SWIR. It has not been calibrated on a real three-band sensor.
|
| 41 |
+
|
| 42 |
+
## Required base model
|
| 43 |
+
|
| 44 |
+
Supply a legally obtained Stable Diffusion 1.5 inpainting Diffusers directory. This adapter does not include base-model weights.
|
| 45 |
+
|
| 46 |
+
## File integrity
|
| 47 |
+
|
| 48 |
+
SHA256: `573516143fb7a3c59256c449f0d2b34aa56be6945e3c9ba8a0cc307d2cd68739`
|
| 49 |
+
|
| 50 |
+
Repository: [github.com/ss00sxt/IRecon](https://github.com/ss00sxt/IRecon)
|
mcm_fusion_128.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:573516143fb7a3c59256c449f0d2b34aa56be6945e3c9ba8a0cc307d2cd68739
|
| 3 |
+
size 5507658
|
mcm_fusion_128.sha256
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
573516143fb7a3c59256c449f0d2b34aa56be6945e3c9ba8a0cc307d2cd68739 mcm_fusion_128.pt
|