Add library name and link to the code repository

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +15 -6
README.md CHANGED
@@ -1,10 +1,11 @@
1
  ---
2
- language:
3
- - en
4
  base_model:
5
  - lmsys/vicuna-7b-v1.5
6
  - openai/clip-vit-large-patch14
7
  - laion/CLIP-convnext_large_d.laion2B-s26B-b102K-augreg
 
 
 
8
  pipeline_tag: image-text-to-text
9
  tags:
10
  - chatrex
@@ -17,6 +18,8 @@ arxiv.org/abs/2411.18363
17
  <img src="assets/teaser.jpg" width=600 >
18
  </div>
19
 
 
 
20
  ----
21
 
22
  # 1. Introduction 📚
@@ -43,7 +46,7 @@ cd chatrex/upn/ops
43
  pip install -v -e .
44
  ```
45
 
46
- ## 2.1 Download Pre-trained UPN Models
47
  We provide model checkpoints for both the ***Universal Proposal Network (UPN)*** and the ***ChatRex model***. You can download the pre-trained models from the following links:
48
  - [UPN Checkpoint](https://github.com/IDEA-Research/ChatRex/releases/download/upn-large/upn_large.pth)
49
  - [ChatRex-7B Checkpoint](https://huggingface.co/IDEA-Research/ChatRex-7B)
@@ -173,6 +176,8 @@ Please detect person in the car; cat below the table in this image. Answer the q
173
  <details close>
174
  <summary><strong>Example Code</strong></summary>
175
 
 
 
176
  ```python
177
  import torch
178
  from PIL import Image
@@ -289,6 +294,8 @@ Can you provide me with a one sentence of <obji>? Answer the question with one s
289
  <details close>
290
  <summary><strong>Example Code</strong></summary>
291
 
 
 
292
  ```python
293
  import torch
294
  from PIL import Image
@@ -381,6 +388,8 @@ Please provide a detailed description of the image and detect all the mentioned
381
  <details close>
382
  <summary><strong>Example Code</strong></summary>
383
 
 
 
384
  ```python
385
  import torch
386
  from PIL import Image
@@ -483,6 +492,8 @@ Answer the question in Grounded format. Question
483
  <details close>
484
  <summary><strong>Example Code</strong></summary>
485
 
 
 
486
  ```python
487
  import torch
488
  from PIL import Image
@@ -558,7 +569,6 @@ if __name__ == "__main__":
558
  )
559
  vis_image.save("tests/test_chatrex_grounded_conversation.jpeg")
560
  print(f"prediction is saved at tests/test_chatrex_grounded_conversation.jpeg")
561
-
562
  ```
563
 
564
  The output from LLM is like:
@@ -576,7 +586,6 @@ The visualization of the output is like:
576
 
577
  ----
578
 
579
-
580
  # 5. LICENSE
581
 
582
  ChatRex is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights Reserved. Note that this project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses including but not limited to the:
@@ -595,4 +604,4 @@ ChatRex is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights R
595
  primaryClass={cs.CV},
596
  url={https://arxiv.org/abs/2411.18363},
597
  }
598
- ```
 
1
  ---
 
 
2
  base_model:
3
  - lmsys/vicuna-7b-v1.5
4
  - openai/clip-vit-large-patch14
5
  - laion/CLIP-convnext_large_d.laion2B-s26B-b102K-augreg
6
+ language:
7
+ - en
8
+ library_name: transformers
9
  pipeline_tag: image-text-to-text
10
  tags:
11
  - chatrex
 
18
  <img src="assets/teaser.jpg" width=600 >
19
  </div>
20
 
21
+ The code for the model can be found at: https://github.com/IDEA-Research/ChatRex
22
+
23
  ----
24
 
25
  # 1. Introduction 📚
 
46
  pip install -v -e .
47
  ```
48
 
49
+ ## 2.1 Download Pre-trained Models
50
  We provide model checkpoints for both the ***Universal Proposal Network (UPN)*** and the ***ChatRex model***. You can download the pre-trained models from the following links:
51
  - [UPN Checkpoint](https://github.com/IDEA-Research/ChatRex/releases/download/upn-large/upn_large.pth)
52
  - [ChatRex-7B Checkpoint](https://huggingface.co/IDEA-Research/ChatRex-7B)
 
176
  <details close>
177
  <summary><strong>Example Code</strong></summary>
178
 
179
+ - [Example Code in python file](tests/test_chatrex_detection.py)
180
+
181
  ```python
182
  import torch
183
  from PIL import Image
 
294
  <details close>
295
  <summary><strong>Example Code</strong></summary>
296
 
297
+ - [Example Code in python file](tests/test_chatrex_region_caption.py)
298
+
299
  ```python
300
  import torch
301
  from PIL import Image
 
388
  <details close>
389
  <summary><strong>Example Code</strong></summary>
390
 
391
+ - [Example Code in python file](tests/test_chatrex_grounded_image_caption.py)
392
+
393
  ```python
394
  import torch
395
  from PIL import Image
 
492
  <details close>
493
  <summary><strong>Example Code</strong></summary>
494
 
495
+ - [Example Code in python file](tests/test_chatrex_grounded_conversation.py)
496
+
497
  ```python
498
  import torch
499
  from PIL import Image
 
569
  )
570
  vis_image.save("tests/test_chatrex_grounded_conversation.jpeg")
571
  print(f"prediction is saved at tests/test_chatrex_grounded_conversation.jpeg")
 
572
  ```
573
 
574
  The output from LLM is like:
 
586
 
587
  ----
588
 
 
589
  # 5. LICENSE
590
 
591
  ChatRex is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights Reserved. Note that this project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses including but not limited to the:
 
604
  primaryClass={cs.CV},
605
  url={https://arxiv.org/abs/2411.18363},
606
  }
607
+ ```