Update README.md
Browse files
README.md
CHANGED
|
@@ -1,13 +1,10 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
---
|
| 4 |
# 🌟 Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
|
| 5 |
-
# NeurIPS 2025
|
| 6 |
> [Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation](https://arxiv.org/pdf/2505.14705?).<br>
|
| 7 |
> [Xin Zhang](https://zhangxin-xd.github.io/), Ziruo Zhang, [Jiawei Du](https://scholar.google.com/citations?user=WrJKEzEAAAAJ&hl=zh-CN), [Zuozhu Liu](https://person.zju.edu.cn/en/lzz), [Joey Tianyi Zhou](https://joeyzhouty.github.io/) <br>
|
| 8 |
> Agency for Science, Technology, and Research (ASTAR), Singapore <br>
|
| 9 |
> National University of Singapore, Singapore <br>
|
| 10 |
-
> Zhejiang University, China
|
| 11 |
## 📖 Introduction
|
| 12 |
<p align="center">
|
| 13 |
<img src="imgs/problem.png" alt="problem" title="problem" width="700">
|
|
@@ -43,20 +40,22 @@ To get started, follow these instructions to set up the environment and install
|
|
| 43 |
## 🚀 Usage
|
| 44 |
|
| 45 |
Here’s how to use RepBlend for Multimodal Dataset Distillation:
|
|
|
|
|
|
|
| 46 |
### Pretrained Weights
|
| 47 |
-
The checkpoints for all experimental networks are available from their respective official repositories. For convenience, we have also provided them together [here](https://huggingface.co/xinxin66/RepBlend).
|
| 48 |
Once downloaded, put them in `distill_utils/checkpoints/`.
|
| 49 |
|
| 50 |
### Experimental Datasets
|
| 51 |
The dataset hase been validated on various benchmarks, you can download from their respective links. Once downloaded, put them in `distill_utils/data/`.
|
| 52 |
| datasets | links|
|
| 53 |
|-----|-----|
|
| 54 |
-
| Flickr30K | [images](https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset), [annotations](https://huggingface.co/xinxin66/RepBlend/)|
|
| 55 |
-
| COCO | [images](https://cocodataset.org/#download), [annotations](https://huggingface.co/xinxin66/RepBlend) |
|
| 56 |
-
|LLaVA-cc3m|[images](https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md), [annotations](https://huggingface.co/xinxin66/RepBlend)|
|
| 57 |
|
| 58 |
### Generate Expert Trajectories
|
| 59 |
-
You can generate expert trajectories by running the `scripts/buffer.sh`, or alternatively, download our [pre-generated trajectories](https://huggingface.co/xinxin66/RepBlend) for faster reproduction.
|
| 60 |
```
|
| 61 |
bash scripts/buffer.sh
|
| 62 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# 🌟 Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation
|
| 2 |
+
# NeurIPS 2025
|
| 3 |
> [Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation](https://arxiv.org/pdf/2505.14705?).<br>
|
| 4 |
> [Xin Zhang](https://zhangxin-xd.github.io/), Ziruo Zhang, [Jiawei Du](https://scholar.google.com/citations?user=WrJKEzEAAAAJ&hl=zh-CN), [Zuozhu Liu](https://person.zju.edu.cn/en/lzz), [Joey Tianyi Zhou](https://joeyzhouty.github.io/) <br>
|
| 5 |
> Agency for Science, Technology, and Research (ASTAR), Singapore <br>
|
| 6 |
> National University of Singapore, Singapore <br>
|
| 7 |
+
> Zhejiang University, China <br>
|
| 8 |
## 📖 Introduction
|
| 9 |
<p align="center">
|
| 10 |
<img src="imgs/problem.png" alt="problem" title="problem" width="700">
|
|
|
|
| 40 |
## 🚀 Usage
|
| 41 |
|
| 42 |
Here’s how to use RepBlend for Multimodal Dataset Distillation:
|
| 43 |
+
|
| 44 |
+
First, download the pretrained weights and datasets and place them into their respective folders.
|
| 45 |
### Pretrained Weights
|
| 46 |
+
The checkpoints for all experimental networks are available from their respective official repositories. For convenience, we have also provided them together [🤗 here](https://huggingface.co/xinxin66/RepBlend).
|
| 47 |
Once downloaded, put them in `distill_utils/checkpoints/`.
|
| 48 |
|
| 49 |
### Experimental Datasets
|
| 50 |
The dataset hase been validated on various benchmarks, you can download from their respective links. Once downloaded, put them in `distill_utils/data/`.
|
| 51 |
| datasets | links|
|
| 52 |
|-----|-----|
|
| 53 |
+
| Flickr30K | [images](https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset), [🤗 annotations](https://huggingface.co/xinxin66/RepBlend/)|
|
| 54 |
+
| COCO | [images](https://cocodataset.org/#download), [🤗 annotations](https://huggingface.co/xinxin66/RepBlend) |
|
| 55 |
+
|LLaVA-cc3m|[images](https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md), [🤗 annotations](https://huggingface.co/xinxin66/RepBlend)|
|
| 56 |
|
| 57 |
### Generate Expert Trajectories
|
| 58 |
+
You can generate expert trajectories by running the `scripts/buffer.sh`, or alternatively, download our [pre-generated trajectories](🤗 https://huggingface.co/xinxin66/RepBlend) for faster reproduction.
|
| 59 |
```
|
| 60 |
bash scripts/buffer.sh
|
| 61 |
```
|