Upload folder using huggingface_hub

README.md

@@ -29,23 +29,23 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
 | :--------------------------: | :-------------: | :------------: | :-----------: | :------------------: |
 |          Model Size          |        -        |       -        |      40B      |         76B          |
 |                              |                 |                |               |                      |
-|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     93.9      |         TODO         |
+|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     93.9      |         94.1         |
 |    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     86.2      |         88.4         |
 |    InfoVQA<sub>test</sub>    |        -        |      72.7      |     78.7      |         82.0         |
 |    TextVQA<sub>val</sub>     |        -        |      73.5      |     83.0      |         84.4         |
-|           OCRBench           |       678       |      754       |      837      |         TODO         |
+|           OCRBench           |       678       |      754       |      837      |         839          |
 |      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2315.0     |        2414.7        |
-|         RealWorldQA          |      68.0       |      67.5      |     71.8      |         TODO         |
+|         RealWorldQA          |      68.0       |      67.5      |     71.8      |         72.2         |
 |     AI2D<sub>test</sub>      |      89.4       |      80.3      |     87.1      |         87.6         |
-|      MMMU<sub>val</sub>      |      63.1       |      58.5      |     53.9      |         55.2         |
+|      MMMU<sub>val</sub>      |   63.1 / 61.7   |  58.5 / 60.6   |  53.9 / 55.2  |     55.2 / 58.2      |
 |  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     86.8      |         86.5         |
 |  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     86.5      |         86.3         |
 |    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     80.6      |         81.0         |
 |  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     68.5      |         69.8         |
-| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     65.5      |         TODO         |
+| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     65.5      |         65.7         |
 |          SEED-Image          |        -        |       -        |     78.2      |         78.2         |
-|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     56.9      |         TODO         |
-| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     63.7      |         TODO         |
+|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     56.9      |         55.2         |
+| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     63.7      |         65.5         |
 
 - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. MMMU, OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
 
@@ -59,7 +59,7 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
 | :------------------: | :----: | :------: | :--------------: | :-----------: | :------------------: |
 |      Model Size      |   -    |   34B    |       34B        |      40B      |         76B          |
 |                      |        |          |                  |               |                      |
-|       MVBench        |   -    |    -     |        -         |     72.5      |         TODO         |
+|       MVBench        |   -    |    -     |        -         |     72.5      |         69.6         |
 | Video-MME<br>wo subs |  59.9  |   59.0   |       52.0       |     TODO      |         TODO         |
 | Video-MME<br>w/ subs |  63.3  |   59.4   |       54.9       |     TODO      |         TODO         |
 
@@ -76,6 +76,7 @@ We also welcome you to experience the InternVL2 series models in our [online dem
 > Please use transformers==4.37.2 to ensure the model works normally.
 
 ```python
+import math
 import numpy as np
 import torch
 import torchvision.transforms as T
@@ -163,17 +164,44 @@ def load_image(image_file, input_size=448, max_num=6):
     return pixel_values
 
 
+def split_model(model_name):
+    device_map = {}
+    world_size = torch.cuda.device_count()
+    num_layers = {'InternVL2-8B': 32, 'InternVL2-26B': 48,
+                  'InternVL2-40B': 60, 'InternVL2-Llama3-76B': 80}[model_name]
+    # Since the first GPU will be used for ViT, treat it as half a GPU.
+    num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5))
+    num_layers_per_gpu = [num_layers_per_gpu] * world_size
+    num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5)
+    layer_cnt = 0
+    for i, num_layer in enumerate(num_layers_per_gpu):
+        for j in range(num_layer):
+            device_map[f'language_model.model.layers.{layer_cnt}'] = i
+            layer_cnt += 1
+    device_map['vision_model'] = 0
+    device_map['mlp1'] = 0
+    device_map['language_model.model.tok_embeddings'] = 0
+    device_map['language_model.model.embed_tokens'] = 0
+    device_map['language_model.output'] = 0
+    device_map['language_model.model.norm'] = 0
+    device_map['language_model.lm_head'] = 0
+    device_map[f'language_model.model.layers.{num_layers - 1}'] = 0
+
+    return device_map
+
+
 path = 'OpenGVLab/InternVL2-Llama3-76B'
-# You need to set device_map='auto' to use multiple GPUs for inference.
-import os
-os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
+device_map = split_model('InternVL2-Llama3-76B')
+print(device_map)
+# If you set `load_in_8bit=True`, you will need two 80GB GPUs.
+# If you set `load_in_8bit=False`, you will need at least three 80GB GPUs.
 model = AutoModel.from_pretrained(
     path,
     torch_dtype=torch.bfloat16,
+    load_in_8bit=True,
     low_cpu_mem_usage=True,
     trust_remote_code=True,
-    device_map='auto').eval()
-
+    device_map=device_map).eval()
 tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
 # set the max number of tiles in `max_num`
 pixel_values = load_image('./examples/image1.jpg', max_num=6).to(torch.bfloat16).cuda()
@@ -317,6 +345,10 @@ print(f'User: {question}')
 print(f'Assistant: {response}')
 ```
 
+## Finetune
+
+SWIFT from ModelScope community has supported the fine-tuning (Image/Video) of InternVL, please check [this link](https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/internvl-best-practice.md) for more details.
+
 ## Deployment
 
 ### LMDeploy
@@ -374,23 +406,23 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 | :--------------------------: | :-------------: | :------------: | :-----------: | :------------------: |
 |           模型大小           |        -        |       -        |      40B      |         76B          |
 |                              |                 |                |               |                      |
-|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     93.9      |                      |
-|    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     86.2      |                      |
-|    InfoVQA<sub>test</sub>    |        -        |      72.7      |     78.7      |                      |
-|    TextVQA<sub>val</sub>     |        -        |      73.5      |     83.0      |                      |
-|           OCRBench           |       678       |      754       |      837      |                      |
-|      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2315.0     |                      |
-|         RealWorldQA          |      68.0       |      67.5      |     71.8      |                      |
-|     AI2D<sub>test</sub>      |      89.4       |      80.3      |     87.1      |                      |
-|      MMMU<sub>val</sub>      |      63.1       |      58.5      |     53.9      |                      |
-|  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     86.8      |                      |
-|  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     86.5      |                      |
-|    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     80.6      |                      |
-|  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     68.5      |                      |
-| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     65.5      |                      |
-|          SEED-Image          |        -        |       -        |     78.2      |                      |
-|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     56.9      |                      |
-| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     63.7      |                      |
+|    DocVQA<sub>test</sub>     |      87.2       |      86.5      |     93.9      |         94.1         |
+|    ChartQA<sub>test</sub>    |      78.1       |      81.3      |     86.2      |         88.4         |
+|    InfoVQA<sub>test</sub>    |        -        |      72.7      |     78.7      |         82.0         |
+|    TextVQA<sub>val</sub>     |        -        |      73.5      |     83.0      |         84.4         |
+|           OCRBench           |       678       |      754       |      837      |         839          |
+|      MME<sub>sum</sub>       |     2070.2      |     2110.6     |    2315.0     |        2414.7        |
+|         RealWorldQA          |      68.0       |      67.5      |     71.8      |         72.2         |
+|     AI2D<sub>test</sub>      |      89.4       |      80.3      |     87.1      |         87.6         |
+|      MMMU<sub>val</sub>      |   63.1 / 61.7   |  58.5 / 60.6   |  53.9 / 55.2  |     55.2 / 58.2      |
+|  MMBench-EN<sub>test</sub>   |      81.0       |      73.9      |     86.8      |         86.5         |
+|  MMBench-CN<sub>test</sub>   |      80.2       |      73.8      |     86.5      |         86.3         |
+|    CCBench<sub>dev</sub>     |      57.3       |      28.4      |     80.6      |         81.0         |
+|  MMVet<sub>GPT-4-0613</sub>  |        -        |       -        |     68.5      |         69.8         |
+| MMVet<sub>GPT-4-Turbo</sub>  |      67.5       |      64.0      |     65.5      |         65.7         |
+|          SEED-Image          |        -        |       -        |     78.2      |         78.2         |
+|   HallBench<sub>avg</sub>    |      43.9       |      45.6      |     56.9      |         55.2         |
+| MathVista<sub>testmini</sub> |      58.1       |      57.7      |     63.7      |         65.5         |
 
 - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
 
@@ -404,7 +436,7 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 | :------------------: | :----: | :------: | :--------------: | :-----------: | :------------------: |
 |       模型大小       |   -    |   34B    |       34B        |      40B      |         76B          |
 |                      |        |          |                  |               |                      |
-|       MVBench        |   -    |    -     |        -         |     72.5      |                      |
+|       MVBench        |   -    |    -     |        -         |     72.5      |         69.6         |
 | Video-MME<br>wo subs |  59.9  |   59.0   |       52.0       |     TODO      |         TODO         |
 | Video-MME<br>w/ subs |  63.3  |   59.4   |       54.9       |     TODO      |         TODO         |
 
@@ -422,6 +454,10 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 
 示例代码请[点击这里](#quick-start)。
 
+## 微调
+
+来自ModelScope社区的SWIFT已经支持对InternVL进行微调（图像/视频），详情请查看[此链接](https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/internvl-best-practice.md)。
+
 ## 部署
 
 ### LMDeploy