Add library name and link to the code repository
#1
by
nielsr
HF Staff
- opened
README.md
CHANGED
|
@@ -1,10 +1,11 @@
|
|
| 1 |
---
|
| 2 |
-
language:
|
| 3 |
-
- en
|
| 4 |
base_model:
|
| 5 |
- lmsys/vicuna-7b-v1.5
|
| 6 |
- openai/clip-vit-large-patch14
|
| 7 |
- laion/CLIP-convnext_large_d.laion2B-s26B-b102K-augreg
|
|
|
|
|
|
|
|
|
|
| 8 |
pipeline_tag: image-text-to-text
|
| 9 |
tags:
|
| 10 |
- chatrex
|
|
@@ -17,6 +18,8 @@ arxiv.org/abs/2411.18363
|
|
| 17 |
<img src="assets/teaser.jpg" width=600 >
|
| 18 |
</div>
|
| 19 |
|
|
|
|
|
|
|
| 20 |
----
|
| 21 |
|
| 22 |
# 1. Introduction 📚
|
|
@@ -43,7 +46,7 @@ cd chatrex/upn/ops
|
|
| 43 |
pip install -v -e .
|
| 44 |
```
|
| 45 |
|
| 46 |
-
## 2.1 Download Pre-trained
|
| 47 |
We provide model checkpoints for both the ***Universal Proposal Network (UPN)*** and the ***ChatRex model***. You can download the pre-trained models from the following links:
|
| 48 |
- [UPN Checkpoint](https://github.com/IDEA-Research/ChatRex/releases/download/upn-large/upn_large.pth)
|
| 49 |
- [ChatRex-7B Checkpoint](https://huggingface.co/IDEA-Research/ChatRex-7B)
|
|
@@ -173,6 +176,8 @@ Please detect person in the car; cat below the table in this image. Answer the q
|
|
| 173 |
<details close>
|
| 174 |
<summary><strong>Example Code</strong></summary>
|
| 175 |
|
|
|
|
|
|
|
| 176 |
```python
|
| 177 |
import torch
|
| 178 |
from PIL import Image
|
|
@@ -289,6 +294,8 @@ Can you provide me with a one sentence of <obji>? Answer the question with one s
|
|
| 289 |
<details close>
|
| 290 |
<summary><strong>Example Code</strong></summary>
|
| 291 |
|
|
|
|
|
|
|
| 292 |
```python
|
| 293 |
import torch
|
| 294 |
from PIL import Image
|
|
@@ -381,6 +388,8 @@ Please provide a detailed description of the image and detect all the mentioned
|
|
| 381 |
<details close>
|
| 382 |
<summary><strong>Example Code</strong></summary>
|
| 383 |
|
|
|
|
|
|
|
| 384 |
```python
|
| 385 |
import torch
|
| 386 |
from PIL import Image
|
|
@@ -483,6 +492,8 @@ Answer the question in Grounded format. Question
|
|
| 483 |
<details close>
|
| 484 |
<summary><strong>Example Code</strong></summary>
|
| 485 |
|
|
|
|
|
|
|
| 486 |
```python
|
| 487 |
import torch
|
| 488 |
from PIL import Image
|
|
@@ -558,7 +569,6 @@ if __name__ == "__main__":
|
|
| 558 |
)
|
| 559 |
vis_image.save("tests/test_chatrex_grounded_conversation.jpeg")
|
| 560 |
print(f"prediction is saved at tests/test_chatrex_grounded_conversation.jpeg")
|
| 561 |
-
|
| 562 |
```
|
| 563 |
|
| 564 |
The output from LLM is like:
|
|
@@ -576,7 +586,6 @@ The visualization of the output is like:
|
|
| 576 |
|
| 577 |
----
|
| 578 |
|
| 579 |
-
|
| 580 |
# 5. LICENSE
|
| 581 |
|
| 582 |
ChatRex is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights Reserved. Note that this project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses including but not limited to the:
|
|
@@ -595,4 +604,4 @@ ChatRex is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights R
|
|
| 595 |
primaryClass={cs.CV},
|
| 596 |
url={https://arxiv.org/abs/2411.18363},
|
| 597 |
}
|
| 598 |
-
```
|
|
|
|
| 1 |
---
|
|
|
|
|
|
|
| 2 |
base_model:
|
| 3 |
- lmsys/vicuna-7b-v1.5
|
| 4 |
- openai/clip-vit-large-patch14
|
| 5 |
- laion/CLIP-convnext_large_d.laion2B-s26B-b102K-augreg
|
| 6 |
+
language:
|
| 7 |
+
- en
|
| 8 |
+
library_name: transformers
|
| 9 |
pipeline_tag: image-text-to-text
|
| 10 |
tags:
|
| 11 |
- chatrex
|
|
|
|
| 18 |
<img src="assets/teaser.jpg" width=600 >
|
| 19 |
</div>
|
| 20 |
|
| 21 |
+
The code for the model can be found at: https://github.com/IDEA-Research/ChatRex
|
| 22 |
+
|
| 23 |
----
|
| 24 |
|
| 25 |
# 1. Introduction 📚
|
|
|
|
| 46 |
pip install -v -e .
|
| 47 |
```
|
| 48 |
|
| 49 |
+
## 2.1 Download Pre-trained Models
|
| 50 |
We provide model checkpoints for both the ***Universal Proposal Network (UPN)*** and the ***ChatRex model***. You can download the pre-trained models from the following links:
|
| 51 |
- [UPN Checkpoint](https://github.com/IDEA-Research/ChatRex/releases/download/upn-large/upn_large.pth)
|
| 52 |
- [ChatRex-7B Checkpoint](https://huggingface.co/IDEA-Research/ChatRex-7B)
|
|
|
|
| 176 |
<details close>
|
| 177 |
<summary><strong>Example Code</strong></summary>
|
| 178 |
|
| 179 |
+
- [Example Code in python file](tests/test_chatrex_detection.py)
|
| 180 |
+
|
| 181 |
```python
|
| 182 |
import torch
|
| 183 |
from PIL import Image
|
|
|
|
| 294 |
<details close>
|
| 295 |
<summary><strong>Example Code</strong></summary>
|
| 296 |
|
| 297 |
+
- [Example Code in python file](tests/test_chatrex_region_caption.py)
|
| 298 |
+
|
| 299 |
```python
|
| 300 |
import torch
|
| 301 |
from PIL import Image
|
|
|
|
| 388 |
<details close>
|
| 389 |
<summary><strong>Example Code</strong></summary>
|
| 390 |
|
| 391 |
+
- [Example Code in python file](tests/test_chatrex_grounded_image_caption.py)
|
| 392 |
+
|
| 393 |
```python
|
| 394 |
import torch
|
| 395 |
from PIL import Image
|
|
|
|
| 492 |
<details close>
|
| 493 |
<summary><strong>Example Code</strong></summary>
|
| 494 |
|
| 495 |
+
- [Example Code in python file](tests/test_chatrex_grounded_conversation.py)
|
| 496 |
+
|
| 497 |
```python
|
| 498 |
import torch
|
| 499 |
from PIL import Image
|
|
|
|
| 569 |
)
|
| 570 |
vis_image.save("tests/test_chatrex_grounded_conversation.jpeg")
|
| 571 |
print(f"prediction is saved at tests/test_chatrex_grounded_conversation.jpeg")
|
|
|
|
| 572 |
```
|
| 573 |
|
| 574 |
The output from LLM is like:
|
|
|
|
| 586 |
|
| 587 |
----
|
| 588 |
|
|
|
|
| 589 |
# 5. LICENSE
|
| 590 |
|
| 591 |
ChatRex is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights Reserved. Note that this project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses including but not limited to the:
|
|
|
|
| 604 |
primaryClass={cs.CV},
|
| 605 |
url={https://arxiv.org/abs/2411.18363},
|
| 606 |
}
|
| 607 |
+
```
|