yagmurakarken commited on
Commit
2cb5c6d
·
verified ·
1 Parent(s): 6e5130a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +66 -0
README.md ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ base_model: black-forest-labs/FLUX.1-dev
4
+ tags:
5
+ - diffusion
6
+ - flux
7
+ - dinov3
8
+ - dense-prediction
9
+ - semantic-segmentation
10
+ - salient-object-detection
11
+ - depth-estimation
12
+ ---
13
+
14
+ # MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
15
+
16
+ Trained decoder checkpoints for **MMDiff**, which decodes dense perceptual modalities
17
+ (semantic segmentation, salient object detection, monocular depth) from a **frozen**
18
+ FLUX.1-dev diffusion transformer. Features are fused across multiple denoising
19
+ timesteps (with optional DINOv3 features) and only lightweight decoder heads are
20
+ trained. The same frozen model can also generate images **and** their annotations
21
+ in one pipeline for synthetic data generation.
22
+
23
+ - **Code:** https://github.com/yagmurakarken/mmdiff
24
+ - **Backbone:** [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) (frozen) + [DINOv3 ViT-B/16](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m) (frozen)
25
+
26
+ ## Checkpoints
27
+
28
+ | File | Task | Dataset | Best validation metric |
29
+ |------|------|---------|--------------------------|
30
+ | `duts_saliency.ckpt` | Salient object detection | DUTS | 0.8890 |
31
+ | `pascal_segmentation.ckpt` | Semantic segmentation | Pascal VOC 2012 | mIoU 0.8496 |
32
+ | `nyu_depth.ckpt` | Monocular depth | NYU Depth V2 | RMSE 0.3700 / δ₁ 0.9519 |
33
+
34
+ Each checkpoint is a PyTorch Lightning checkpoint that retains the EMA / training
35
+ state for reproducibility. Inference loads only the model weights (`state_dict`).
36
+
37
+ ## Usage
38
+
39
+ Clone the [code repo](https://github.com/yagmurakarken/mmdiff), then download a
40
+ checkpoint and run inference:
41
+
42
+ ```python
43
+ from huggingface_hub import hf_hub_download
44
+
45
+ ckpt = hf_hub_download("yagmurakarken/mmdiff", "pascal_segmentation.ckpt")
46
+ ```
47
+
48
+ ```bash
49
+ python scripts/inference.py --task pascal \
50
+ --config configs/pascal_voc_config.yaml --checkpoint "$ckpt" \
51
+ --image my_image.jpg --output_dir outputs/
52
+ ```
53
+
54
+ > Architecture flags (`--hidden_dim`, `--num_transformer_layers`, `--num_timesteps`,
55
+ > `--dino_model`) must match the checkpoint. FLUX.1-dev is gated — accept its license
56
+ > and authenticate (`hf auth login`) before first use.
57
+
58
+ ## Citation
59
+
60
+ ```bibtex
61
+ @inproceedings{akarken2026mmdiff,
62
+ title = {{MMDiff}: Extending Diffusion Transformers for Multi-Modal Generation},
63
+ author = {Akarken, Yagmur and Kupyn, Orest and Rupprecht, Christian},
64
+ year = {2026}
65
+ }
66
+ ```