Update README.md
Browse files
README.md
CHANGED
|
@@ -162,30 +162,20 @@ bash configs_scripts/large/eval/inbatch/run_eval.sh
|
|
| 162 |
|
| 163 |
## Model Checkpoints
|
| 164 |
|
| 165 |
-
We provide
|
| 166 |
-
|
| 167 |
-
### Stage 1: Residual Quantization Model
|
| 168 |
-
- **Model**: [`rq_clip_large.pth`](https://huggingface.co/Sungyeon/GENIUS/blob/main/checkpoint/rq_clip_large.pth)
|
| 169 |
-
- **Description**: Learns to encode multimodal data into discrete IDs through residual quantization
|
| 170 |
-
- **Size**: ~1.2GB
|
| 171 |
-
|
| 172 |
-
### Stage 2: Generator Model
|
| 173 |
-
- **Model**: [`GENIUS_t5small.pth`](https://huggingface.co/Sungyeon/GENIUS/blob/main/checkpoint/GENIUS_t5small.pth)
|
| 174 |
-
- **Description**: T5-based sequence-to-sequence model that generates target IDs for retrieval
|
| 175 |
-
- **Size**: ~500MB
|
| 176 |
-
|
| 177 |
-
### Stage 0: CLIP-SF Model
|
| 178 |
-
- **Model**: [`clip_sf_large.pth`](https://huggingface.co/TIGER-Lab/UniIR/blob/main/checkpoint/CLIP_SF/clip_sf_large.pth)
|
| 179 |
-
- **Source**: [TIGER-Lab/UniIR](https://huggingface.co/TIGER-Lab/UniIR)
|
| 180 |
-
- **Description**: Score-fusion CLIP model for multimodal feature extraction
|
| 181 |
|
|
|
|
| 182 |
```bash
|
| 183 |
-
#
|
| 184 |
-
git clone https://huggingface.co/Sungyeon/GENIUS
|
| 185 |
-
|
| 186 |
-
# Download CLIP-SF model
|
| 187 |
wget https://huggingface.co/TIGER-Lab/UniIR/resolve/main/checkpoint/CLIP_SF/clip_sf_large.pth -O checkpoint/CLIP_SF/clip_sf_large.pth
|
|
|
|
|
|
|
|
|
|
| 188 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
| 189 |
|
| 190 |
> Note: All three models are required for full functionality. The CLIP-SF model is used for feature extraction, the Residual Quantization model for ID encoding, and the Generator model for retrieval.
|
| 191 |
|
|
|
|
| 162 |
|
| 163 |
## Model Checkpoints
|
| 164 |
|
| 165 |
+
We provide model checkpoints for GENIUS in the 🤗 [Hugging Face](https://huggingface.co/Sungyeon/GENIUS):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
|
| 167 |
+
### How to Download
|
| 168 |
```bash
|
| 169 |
+
# Download the CLIP-SF model (Stage 0)
|
|
|
|
|
|
|
|
|
|
| 170 |
wget https://huggingface.co/TIGER-Lab/UniIR/resolve/main/checkpoint/CLIP_SF/clip_sf_large.pth -O checkpoint/CLIP_SF/clip_sf_large.pth
|
| 171 |
+
|
| 172 |
+
# Clone the GENIUS checkpoints (Stage 1 and 2)
|
| 173 |
+
git clone https://huggingface.co/Sungyeon/GENIUS
|
| 174 |
```
|
| 175 |
+
### Each Checkpoints
|
| 176 |
+
- **CLIP-SF Model** (Stage 0): [`clip_sf_large.pth`](https://huggingface.co/TIGER-Lab/UniIR/blob/main/checkpoint/CLIP_SF/clip_sf_large.pth)
|
| 177 |
+
- **Residual Quantization Model** (Stage 1): [`rq_clip_large.pth`](https://huggingface.co/Sungyeon/GENIUS/blob/main/checkpoint/rq_clip_large.pth)
|
| 178 |
+
- **Generator Model** (Stage 2): [`GENIUS_t5small.pth`](https://huggingface.co/Sungyeon/GENIUS/blob/main/checkpoint/GENIUS_t5small.pth)
|
| 179 |
|
| 180 |
> Note: All three models are required for full functionality. The CLIP-SF model is used for feature extraction, the Residual Quantization model for ID encoding, and the Generator model for retrieval.
|
| 181 |
|