Add model card, pipeline tag and links to paper/code
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,3 +1,73 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
--
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
pipeline_tag: unconditional-image-generation
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising
|
| 7 |
+
|
| 8 |
+
Official implementation of **V-Co**, a systematic study of visual co-denoising within a unified pixel-space diffusion framework.
|
| 9 |
+
|
| 10 |
+
- **Paper:** [V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising](https://huggingface.co/papers/2603.16792)
|
| 11 |
+
- **GitHub Repository:** [HL-hanlin/V-Co](https://github.com/HL-hanlin/V-Co)
|
| 12 |
+
|
| 13 |
+
## Method Description
|
| 14 |
+
|
| 15 |
+
V-Co presents a principled study of visual representation alignment via co-denoising in pixel-space diffusion, systematically isolating the effects of architecture, CFG design, auxiliary losses, and feature calibration. The authors identify four key ingredients for effective visual co-denoising:
|
| 16 |
+
|
| 17 |
+
1. **Fully dual-stream architecture** to enable flexible cross-stream interaction.
|
| 18 |
+
2. **Structural masking** for unconditional classifier-free guidance (CFG) prediction.
|
| 19 |
+
3. **Perceptual-drifting hybrid loss** for stronger semantic supervision.
|
| 20 |
+
4. **RMS-based feature rescaling** for stable cross-stream calibration.
|
| 21 |
+
|
| 22 |
+
Together, these findings yield a simple recipe for visual co-denoising. Experiments on ImageNet-256 show that, at comparable model sizes, V-Co outperforms the underlying pixel-space diffusion baseline (JiT) and strong prior pixel-diffusion methods while using fewer training epochs.
|
| 23 |
+
|
| 24 |
+
## Evaluation
|
| 25 |
+
|
| 26 |
+
To evaluate a pretrained V-Co model (e.g., VCo-B) on ImageNet 256x256, you can use the following command from the official repository:
|
| 27 |
+
|
| 28 |
+
```bash
|
| 29 |
+
torchrun --nproc_per_node=8 --nnodes=1 \
|
| 30 |
+
main_vco.py \
|
| 31 |
+
--P_mean -0.8 --P_std 0.8 \
|
| 32 |
+
--img_size 256 \
|
| 33 |
+
--batch_size 32 \
|
| 34 |
+
--gen_bsz 32 \
|
| 35 |
+
--resume '/path/to/vco_base/checkpoint.pth' \
|
| 36 |
+
--online_eval \
|
| 37 |
+
--model JiT-B/16-co \
|
| 38 |
+
--output_dir '/path/to/output_dir' \
|
| 39 |
+
--data_path '/path/to/imagenet/' \
|
| 40 |
+
--evaluate_gen \
|
| 41 |
+
--num_workers 12 \
|
| 42 |
+
--use_co_embed \
|
| 43 |
+
--use_dinov2 \
|
| 44 |
+
--use_dino_from_rae \
|
| 45 |
+
--use_gated_co_embed \
|
| 46 |
+
--use_mmdit \
|
| 47 |
+
--separate_qkv \
|
| 48 |
+
--use_conv2d_dino_proj \
|
| 49 |
+
--uncond_dino_null \
|
| 50 |
+
--dinov2_null_type 'attn_mask_asymmetric' \
|
| 51 |
+
--dinov2_drop_zero_loss \
|
| 52 |
+
--class_balanced_sampling \
|
| 53 |
+
--drifting_v3_loss \
|
| 54 |
+
--cfg_sweep '1.8'
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
## Citation
|
| 58 |
+
|
| 59 |
+
```bibtex
|
| 60 |
+
@misc{lin2026vcocloserlookvisual,
|
| 61 |
+
title={V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising},
|
| 62 |
+
author={Han Lin and Xichen Pan and Zun Wang and Yue Zhang and Chu Wang and Jaemin Cho and Mohit Bansal},
|
| 63 |
+
year={2026},
|
| 64 |
+
eprint={2603.16792},
|
| 65 |
+
archivePrefix={arXiv},
|
| 66 |
+
primaryClass={cs.CV},
|
| 67 |
+
url={https://arxiv.org/abs/2603.16792},
|
| 68 |
+
}
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
## Acknowledgements
|
| 72 |
+
|
| 73 |
+
The development of V-Co has been inspired by [JiT](https://github.com/LTH14/JiT) and [Latent Forcing](https://github.com/AlanBaade/LatentForcing).
|