Add files using upload-large-folder tool

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/.readthedocs.yaml

@@ -0,0 +1,17 @@
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.8"
+
+formats:
+    - epub
+
+sphinx:
+  configuration: docs/en/conf.py
+
+python:
+  install:
+    - requirements: requirements/docs.txt

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/ConfigSystem.md

@@ -0,0 +1,67 @@
+# Config System
+
+By default, VLMEvalKit launches the evaluation by setting the model name(s) (defined in `/vlmeval/config.py`) and dataset name(s) (defined in `vlmeval/dataset/__init__.py` or `vlmeval/dataset/video_dataset_config.py`) in the `run.py` script with the `--model` and `--data` arguments. Such approach is simple and efficient in most scenarios, however, it may not be flexible enough when the user wants to evaluate multiple models / datasets with different settings.
+
+To address this, VLMEvalKit provides a more flexible config system. The user can specify the model and dataset settings in a json file, and pass the path to the config file to the `run.py` script with the `--config` argument. Here is a sample config json:
+
+```json
+{
+    "model": {
+        "GPT4o_20240806_T00_HIGH": {
+            "class": "GPT4V",
+            "model": "gpt-4o-2024-08-06",
+            "temperature": 0,
+            "img_detail": "high"
+        },
+        "GPT4o_20240806_T10_Low": {
+            "class": "GPT4V",
+            "model": "gpt-4o-2024-08-06",
+            "temperature": 1.0,
+            "img_detail": "low"
+        },
+        "GPT4o_20241120": {}
+    },
+    "data": {
+        "MME-RealWorld-Lite": {
+            "class": "MMERealWorld",
+            "dataset": "MME-RealWorld-Lite"
+        },
+        "MMBench_DEV_EN_V11": {
+            "class": "ImageMCQDataset",
+            "dataset": "MMBench_DEV_EN_V11"
+        },
+        "MMBench_Video_8frame_nopack":{},
+        "Video-MME_16frame_subs": {
+            "class": "VideoMME",
+            "dataset": "Video-MME",
+            "nframe": 16,
+            "use_subtitle": true
+        }
+    }
+}
+```
+
+Explanation of the config json:
+
+1. Now we support two fields: `model` and `data`, each of which is a dictionary. The key of the dictionary is the name of the model / dataset (set by the user), and the value is the setting of the model / dataset.
+2. For items in `model`, the value is a dictionary containing the following keys:
+    - `class`: The class name of the model, which should be a class name defined in `vlmeval/vlm/__init__.py` (open-source models) or `vlmeval/api/__init__.py` (API models).
+    - Other kwargs: Other kwargs are model-specific parameters, please refer to the definition of the model class for detailed usage. For example, `model`, `temperature`, `img_detail` are arguments of the `GPT4V` class. It's noteworthy that the `model` argument is required by most model classes.
+    - Tip: The defined model in the `supported_VLM` of `vlmeval/config.py` can be used as a shortcut, for example, `GPT4o_20241120: {}` is equivalent to `GPT4o_20241120: {'class': 'GPT4V', 'model': 'gpt-4o-2024-11-20', 'temperature': 0, 'img_size': -1, 'img_detail': 'high', 'retry': 10, 'verbose': False}`
+3. For the dictionary `data`, we suggest users to use the official dataset name as the key (or part of the key), since we frequently determine the post-processing / judging settings based on the dataset name. For items in `data`, the value is a dictionary containing the following keys:
+    - `class`: The class name of the dataset, which should be a class name defined in `vlmeval/dataset/__init__.py`.
+    - Other kwargs: Other kwargs are dataset-specific parameters, please refer to the definition of the dataset class for detailed usage. Typically, the `dataset` argument is required by most dataset classes. It's noteworthy that the `nframe` argument or `fps` argument is required by most video dataset classes.
+    - Tip: The defined dataset in the `supported_video_datasets` of `vlmeval/dataset/video_dataset_config.py` can be used as a shortcut, for example, `MMBench_Video_8frame_nopack: {}` is equivalent to `MMBench_Video_8frame_nopack: {'class': 'MMBenchVideo', 'dataset': 'MMBench-Video', 'nframe': 8, 'pack': False}`.
+Saving the example config json to `config.json`, you can launch the evaluation by:
+
+```bash
+python run.py --config config.json
+```
+
+That will generate the following output files under the working directory `$WORK_DIR` (Following the format `{$WORK_DIR}/{$MODEL_NAME}/{$MODEL_NAME}_{$DATASET_NAME}_*`):
+
+- `$WORK_DIR/GPT4o_20240806_T00_HIGH/GPT4o_20240806_T00_HIGH_MME-RealWorld-Lite*`
+- `$WORK_DIR/GPT4o_20240806_T10_Low/GPT4o_20240806_T10_Low_MME-RealWorld-Lite*`
+- `$WORK_DIR/GPT4o_20240806_T00_HIGH/GPT4o_20240806_T00_HIGH_MMBench_DEV_EN_V11*`
+- `$WORK_DIR/GPT4o_20240806_T10_Low/GPT4o_20240806_T10_Low_MMBench_DEV_EN_V11*`
+...

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/Contributors.md

@@ -0,0 +1,21 @@
+# Contributors
+
+## Contributors w. 3+ Major Contributions
+
+> In this section, we list all the contributors who have made significant contributions (3+) to the development of VLMEvalKit.
+
+New Qualified Contributors (2024.09):
+
+1. [amitbcp](https://github.com/amitbcp): The contributor helped support MUIRBench, Phi-3.5, Idefics3, VILA, and xGen-MM
+2. [czczup](https://github.com/czczup): The contributor helped support the InternVL Series (V1.5, Mini-InternVL, V2, etc.)
+3. [DseidLi](https://github.com/DseidLi): The contributor helped support LLaVA-OneVision, GQA, and developed the readthedocs site for VLMEvalKit
+4. [mayubo2333](https://github.com/mayubo2333): The contributor helped support MMLongBench, SlideVQA, and DUDE
+5. [sun-hailong](https://github.com/sun-hailong): The contributor helped support A-OKVQA, Parrot, MMMB, and MTL-MMBench
+6. [PhoenixZ810](https://github.com/PhoenixZ810): The contributor helped support Video-ChatGPT, Chat-UniVI, and Llama-VID
+7. [Cuiunbo](https://github.com/Cuiunbo): The contributor helped support OmniLMM-12B, MiniCPM-V Series (V1, V2, V2.5)
+
+## Full Contributor List
+
+> In this section, we list all the contributors as well as their corresponding contributions to the development of VLMEvalKit.
+
+TBD.

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/Development.md

@@ -0,0 +1,145 @@
+# Develop new Benchmark / MLLM
+
+>  🛠️ How to implement a new Benchmark / VLM in VLMEvalKit?
+
+## Implement a new benchmark
+
+Example PR: **Math-Vision Benchmark** ([#292](https://github.com/open-compass/VLMEvalKit/pull/292/files))
+
+In VLMEvalKit, benchmarks are organized as dataset classes. When you try to implement a new benchmark, you can either reuse existing dataset classes (*e.g.*, You can reuse `ImageMCQDataset` when implementing a new multi-choice benchmark), or support a new dataset class. Each dataset must have the following two member functions (either reuse the one of the parent class or implement your own):
+
+- `build_prompt(self, line)`: The function input `line` is an integer (the sample index) or a `pd.Series` object (the raw record of the sample). The function outputs a `multi-modal message`, serving as the input of an MLLM. The `multi-modal message` is an interleaved list of multi-modal messages adopting the following format (the example includes an image and a text message): `[dict(type='image', value=IMAGE_PTH), dict(type='text', value=prompt)]`.
+- `evaluate(self, eval_file,  **judge_kwargs)`: The function input `eval_file` is the MLLM prediction (typically in `.xlsx` format). If the benchmark requires an external LLM (typically GPT) for evaluation, then `judge_kwargs` can pass the arguments for the LLM. The function outputs the benchmark evaluation results (metrics) in the form of `dict` or `pd.DataFrame`.
+
+We then brief the typical steps to implement a new benchmark under VLMEvalKit:
+
+### 1. Prepare your benchmark tsv file
+
+Currently, we organize a benchmark as one single TSV file. During inference, the data file will be automatically downloaded from the definited `DATASET_URL` link to `$LMUData` file (default path is `$HOME/LMUData`, if not set explicitly). You can upload the prepared TSV file to a downloadable address (e.g., Huggingface) or send it to us at <opencompass@pjlab.org.cn>. We will assist in uploading the dataset to the server. You can also customize `LMUData` path in the environment variable `LMUData=/path/to/your/data`.
+
+The contents of the TSV file consist of:
+
+| Dataset Name \ Fields                   | index | image | image_path | question | hint | multi-choice<br>options | answer | category | l2-category | split |
+| --------------------------------------- | ----- | ----- | ---------- | -------- | ---- | ----------------------- | ------ | -------- | ----------- | ----- |
+| MMBench_DEV_[CN/EN]                     | ✅     | ✅     |            | ✅        | ✅    | ✅                       | ✅      | ✅        | ✅           | ✅     |
+| MMBench_TEST_[CN/EN]                    | ✅     | ✅     |            | ✅        | ✅    | ✅                       |        | ✅        | ✅           | ✅     |
+| CCBench                                 | ✅     | ✅     |            | ✅        |      | ✅                       | ✅      | ✅        |             |       |
+| SEEDBench_IMG                           | ✅     | ✅     |            | ✅        |      | ✅                       | ✅      | ✅        |             |       |
+| MME                                     | ✅     | ✅     |            | ✅        |      |                         | ✅      | ✅        |             |       |
+| MMVet                                   | ✅     | ✅     |            | ✅        |      |                         | ✅      | ✅        |             |       |
+| MMMU_DEV_VAL                            | ✅     | ✅     | ✅          | ✅        |      | ✅                       | ✅      | ✅        | ✅           | ✅     |
+| COCO_VAL                                | ✅     | ✅     |            |          |      |                         | ✅      |          |             |       |
+| OCRVQA_[TEST/TESTCORE]                  | ✅     | ✅     |            | ✅        |      |                         | ✅      |          |             |       |
+| TextVQA_VAL                             | ✅     | ✅     |            | ✅        |      |                         | ✅      |          |             |       |
+| VCR_[EN/ZH]\_[EASY/HARD]\_[ALL/500/100] | ✅     | ✅     |            | ✅        |      |                         | ✅      |          |             |       |
+| MMMB_[en/cn/pt/ar/tr/ru] | ✅     | ✅     |            | ✅        | ✅     | ✅     | ✅      | ✅         |             |✅       |
+| MMBench_dev_[en/cn/pt/ar/tr/ru] | ✅     | ✅     |            | ✅        | ✅     | ✅     | ✅      | ✅         | ✅            |✅       |
+
+<div align="center"><b>Table 1. TSV fields of supported datasets.</b></div>
+
+**Intro to mandatory fields in the `TSV` file:**
+
+- **index:** Integer, Unique for each line in `tsv`
+- **image:** The base64 of the image, you can use APIs implemented in `vlmeval/smp/vlm.py` for encoding and decoding:
+  - Encoding: `encode_image_to_base64 `(for PIL Image) / `encode_image_file_to_base64` (for image file path)
+  - Decoding: `decode_base64_to_image`(for PIL Image) / `decode_base64_to_image_file` (for image file path)
+- **question**: The question corresponding to the image, a string
+- **answer**: The answer to the question, a string. The `test` split does not need this field
+
+### 2. Cutomize your benchmark prompt
+
+`ImageBaseDataset` defines the default prompt format. If you need to add prompts specific to the dataset or input data in the `Interleave` format to the model, you can implement this through the `build_prompt(line)` function. This function takes a line from a TSV file as input, containing fields such as index, image, question, etc. The function returns a dictionary list of multimodal messages `msg` in the format `[dict(type='image', value=IMAGE_PTH), dict(type='text', value=prompt)]`, including the image path and the text prompt to be input into VLMs. For interleave type inputs, you can directly place the dictionary of the image path at the image token position.
+
+### 3. Cutomize your benchmark metrics
+
+To add evaluation for a new benchmark, you need to customize a class object to implement the dataset’s metrics calculation. Multimodal datasets inherit from the `ImageBaseDataset` object in `vlmeval/dataset/image_base.py`. The TYPE defines the type of dataset, `DATASET_URL` is the download address of the dataset, and `DATASET_MD5` is the MD5 checksum for consistency checking of the dataset file.
+
+In this class, **you need to implement** the `evaluate(eval_file, **judge_kwargs)` class function to calculate metrics and output results for the custom dataset. The function input `eval_file` is the path to the model prediction results file `{model_name}_{dataset}.xlsx`. This file can be read as a pandas.DataFrame using the `load(eval_file)` method, containing fields such as index, question, answer, category, prediction, etc. The judge_kwargs will pass a dictionary related to evaluation, such as the name of the `judge model`, the number of API request threads, etc. **The return value** of the function is the calculated accuracy and other metrics, formatted as a dictionary composed of lists, organized into a pandas.DataFrame.
+
+## Implement a new model
+
+Example PR: **Support LLaVA-Next-Interleave** ([#294](https://github.com/open-compass/VLMEvalKit/pull/294))
+
+**1. Support `generate_inner` API (mandatory).**
+
+All existing models are implemented in `vlmeval/vlm`. For a minimal model, your model class **must implement the method** `generate_inner(msgs, dataset=None)`. In this function, you feed a multi-modal message to your VLM and return the VLM prediction (which is a string). The optional argument `dataset` can be used as the flag for the model to switch among various inference strategies.
+
+The multi-modal messages `msgs` is a list of dictionaries, each dictionary has two keys: type and value:
+- `type`: We currently support two types, choices are ["image", "text"].
+- `value`: When type=='text' , the value is the text message (a single string); when type=='image', the value can be the local path of an image file, or the image URL.
+
+Currently a multi-modal message may contain arbitrarily interleaved images and texts. If your model do not support that, a practice can be taking the 1st image and concatenated text messages as the input. You can set the `INTERLEAVE = False` in your model class and use `self.message_to_promptimg(message, dataset=dataset)` to build your prompt and the first image's path.
+
+Here are some examples of multi-modal messages:
+
+```python
+IMAGE_PTH = 'assets/apple.jpg'
+IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
+msg1 = [
+    dict(type='image', value=IMAGE_PTH),
+    dict(type='text', value='What is in this image?')
+]
+msg2 = [
+    dict(type='image', value=IMAGE_URL),
+    dict(type='image', value=IMAGE_URL),
+    dict(type='text', value='How many apples are there in these images?')
+]
+response = model.generate(msg1)
+```
+
+For convenience sake, we also support to take a list of string as inputs. In that case, we will check if a string is an image path or image URL and automatically convert it to the list[dict] format:
+
+```python
+IMAGE_PTH = 'assets/apple.jpg'
+IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
+msg1 = [IMAGE_PTH, 'What is in this image?']
+msg2 = [IMAGE_URL, IMAGE_URL,  'How many apples are there in these images?']
+response = model.generate(msg1)
+```
+
+**Support Custom Prompt (optional).**
+
+Besides, your model can support **custom prompt building** by implementing two optional methods: `use_custom_prompt(dataset)` and `build_prompt(line, dataset=None)`.
+
+Both functions take the dataset name as the input：
+
+-  `use_custom_prompt(dataset)` returns a boolean flag, indicating whether the model should use the custom prompt building strategy.
+- If `use_custom_prompt(dataset)` returns True, `build_prompt(line, dataset)` should return a customly bulit multimodal message for the corresponding `dataset`, given `line`, which is a dictionary that includes the necessary information of a data sample. If `use_custom_prompt(dataset)` returns False, the default prompt building strategy will be used.
+
+**Support multi-turn chatting (optional).**
+
+You can also support the multi-turn chatting and evaluation with your VLM by supporting the `chat_inner(message, dataset)` function. The function outputs a single string response, and the `message` is a list of chat history, following the below format.
+
+```python
+# Assume msg1, msg2, msg3, ... are multi-modal messages following the previously described format
+# `chat_inner` take the following chat history list as input:
+message = [
+    dict(role='user', content=msg1),
+    dict(role='assistant', content=msg2),
+    dict(role='user', content=msg3),
+    dict(role='assistant', content=msg4),
+	......
+    dict(role='user', content=msgn),
+]
+# `message` should contain an odd number of chat utterances, the role of utterances should be interleaved "user" and "assistant", with the role of the last utterance to be "user".
+# The chat function will call `chat_inner`
+response = model.chat(message)
+```
+
+### Example PRs:
+
+- VLM that doesn't support interleaved images and texts, and does not use custom prompts: [[Model] Support glm-4v-9b](https://github.com/open-compass/VLMEvalKit/pull/221)
+- VLM that supports interleaved images and texts and custom prompts: [Add MiniCPM-Llama3-V-2.5](https://github.com/open-compass/VLMEvalKit/pull/205)
+- VLM API: [Feature add glmv](https://github.com/open-compass/VLMEvalKit/pull/201)
+
+## Contribute to VLMEvalKit
+
+If you want to contribute codes to **VLMEvalKit**, please do the pre-commit check before you submit a PR. That helps to keep the code tidy.
+
+```bash
+# Under the directory of VLMEvalKit, install the pre-commit hook:
+pip install pre-commit
+pre-commit install
+pre-commit run --all-files
+# Then you can commit your code.
+```

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/EvalByLMDeploy.md

@@ -0,0 +1,27 @@
+# Using LMDeploy to Accelerate Evaluation and Inference
+
+VLMEvalKit supports testing VLM models deployed by LMDeploy. Below, we use InternVL2-8B as an example to show how to test the model.
+
+## Step 0: Install LMDeploy
+
+```bash
+pip install lmdeploy
+```
+For other installation methods, you can refer to LMDeploy's [documentation](https://github.com/InternLM/lmdeploy).
+
+## Step 1: Start the Inference Service
+
+```bash
+lmdeploy serve api_server OpenGVLab/InternVL2-8B --model-name InternVL2-8B
+```
+> [!IMPORTANT]
+> Since models in VLMEvalKit may have custom behaviors when building prompts for different datasets, such as InternVL2's handling of HallusionBench, it is necessary to specify `--model-name` when starting the server. This allows the VLMEvalKit to select appropriate prompt construction strategy based on the name when using the LMDeploy API.
+>
+> If `--server-port`, is specified, the corresponding environment variable `LMDEPLOY_API_BASE` needs to be set.
+
+
+## Step 2: Evaluation
+
+```bash
+python run.py --data MMStar --model lmdeploy --verbose --api-nproc 64
+```

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/Quickstart.md

@@ -0,0 +1,236 @@
+# Quickstart
+
+Before running the evaluation script, you need to **configure** the VLMs and set the model_paths properly.
+
+After that, you can use a single script `run.py` to inference and evaluate multiple VLMs and benchmarks at a same time.
+
+## Step 0. Installation & Setup essential keys
+
+**Installation.**
+
+```bash
+git clone https://github.com/open-compass/VLMEvalKit.git
+cd VLMEvalKit
+pip install -e .
+```
+
+**Setup Keys.**
+
+To infer with API models (GPT-4v, Gemini-Pro-V, etc.) or use LLM APIs as the **judge or choice extractor**, you need to first setup API keys. VLMEvalKit will use an judge **LLM** to extract answer from the output if you set the key, otherwise it uses the **exact matching** mode (find "Yes", "No", "A", "B", "C"... in the output strings). **The exact matching can only be applied to the Yes-or-No tasks and the Multi-choice tasks.**
+- You can place the required keys in `$VLMEvalKit/.env` or directly set them as the environment variable. If you choose to create a `.env` file, its content will look like:
+
+  ```bash
+  # The .env file, place it under $VLMEvalKit
+  # API Keys of Proprietary VLMs
+  # QwenVL APIs
+  DASHSCOPE_API_KEY=
+  # Gemini w. Google Cloud Backends
+  GOOGLE_API_KEY=
+  # OpenAI API
+  OPENAI_API_KEY=
+  OPENAI_API_BASE=
+  # StepAI API
+  STEPAI_API_KEY=
+  # REKA API
+  REKA_API_KEY=
+  # GLMV API
+  GLMV_API_KEY=
+  # CongRong API
+  CW_API_BASE=
+  CW_API_KEY=
+  # SenseNova API
+  SENSENOVA_API_KEY=
+  # Hunyuan-Vision API
+  HUNYUAN_SECRET_KEY=
+  HUNYUAN_SECRET_ID=
+  # LMDeploy API
+  LMDEPLOY_API_BASE=
+  # MiniMax API
+  MINIMAX_API_KEY=
+  # You can also set a proxy for calling api models during the evaluation stage
+  EVAL_PROXY=
+  ```
+
+- Fill the blanks with your API keys (if necessary). Those API keys will be automatically loaded when doing the inference and evaluation.
+## Step 1. Configuration
+
+**VLM Configuration**: All VLMs are configured in `vlmeval/config.py`. Few legacy VLMs (like MiniGPT-4, LLaVA-v1-7B) requires additional configuration (configuring the code / model_weight root in the config file). During evaluation, you should use the model name specified in `supported_VLM` in `vlmeval/config.py` to select the VLM. Make sure you can successfully infer with the VLM before starting the evaluation with the following command `vlmutil check {MODEL_NAME}`.
+
+Note: For the Qwen-VL series models (Qwen-VL, Qwen2-VL, Qwen2.5-VL), the upper and lower bounds of the number of pixels specified in vlmeval/config.py are as follows:
+
+```
+min_pixels=1280 * 28 * 28,
+max_pixels=16384 * 28 * 28,
+```
+Where 1280 is the maximum value recommended by Qwen for balancing performance, computational resources, and memory, and 16384 is the theoretical maximum value for model input. This setting has a positive effect on some vision tasks that require high resolution (such as document understanding). However, there is no practical basis for considering this setting. If you need to align with the official settings, you can remove these two values, or set them to the following values from the official Qwen demo:
+
+```
+min_pixels=256 * 28 * 28,
+max_pixels=1280 * 28 * 28,
+```
+
+## Step 2. Evaluation
+
+**New!!!**  We integrated a new config system to enable more flexible evaluation settings. Check the [Document](/docs/en/ConfigSystem.md) or run `python run.py --help` for more details 🔥🔥🔥
+
+We use `run.py` for evaluation. To use the script, you can use `$VLMEvalKit/run.py` or create a soft-link of the script (to use the script anywhere):
+
+**Arguments**
+
+- `--data (list[str])`: Set the dataset names that are supported in VLMEvalKit (names can be found in the codebase README).
+- `--model (list[str])`: Set the VLM names that are supported in VLMEvalKit (defined in `supported_VLM` in `vlmeval/config.py`).
+- `--mode (str, default to 'all', choices are ['all', 'infer'])`: When `mode` set to "all", will perform both inference and evaluation; when set to "infer", will only perform the inference.
+- `--api-nproc (int, default to 4)`: The number of threads for OpenAI API calling.
+- `--work-dir (str, default to '.')`: The directory to save evaluation results.
+
+**Command for Evaluating Image Benchmarks **
+
+You can run the script with `python` or `torchrun`:
+
+```bash
+# When running with `python`, only one VLM instance is instantiated, and it might use multiple GPUs (depending on its default behavior).
+# That is recommended for evaluating very large VLMs (like IDEFICS-80B-Instruct).
+
+# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference and Evalution
+python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose
+# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference only
+python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose --mode infer
+
+# When running with `torchrun`, one VLM instance is instantiated on each GPU. It can speed up the inference.
+# However, that is only suitable for VLMs that consume small amounts of GPU memory.
+
+# IDEFICS-9B-Instruct, Qwen-VL-Chat, mPLUG-Owl2 on MMBench_DEV_EN, MME, and SEEDBench_IMG. On a node with 8 GPU. Inference and Evaluation.
+torchrun --nproc-per-node=8 run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct qwen_chat mPLUG-Owl2 --verbose
+# Qwen-VL-Chat on MME. On a node with 2 GPU. Inference and Evaluation.
+torchrun --nproc-per-node=2 run.py --data MME --model qwen_chat --verbose
+```
+
+**Command for Evaluating Video Benchmarks**
+
+```bash
+# When running with `python`, only one VLM instance is instantiated, and it might use multiple GPUs (depending on its default behavior).
+# That is recommended for evaluating very large VLMs (like IDEFICS-80B-Instruct).
+
+# IDEFICS2-8B on MMBench-Video, with 8 frames as inputs and vanilla evaluation. On a node with 8 GPUs. MMBench_Video_8frame_nopack is a defined dataset setting in `vlmeval/dataset/video_dataset_config.py`.
+torchrun --nproc-per-node=8 run.py --data MMBench_Video_8frame_nopack --model idefics2_8
+# GPT-4o (API model) on MMBench-Video, with 1 frame per second as inputs and pack evaluation (all questions of a video in a single query).
+python run.py --data MMBench_Video_1fps_pack --model GPT4o
+```
+
+The evaluation results will be printed as logs, besides. **Result Files** will also be generated in the directory `$YOUR_WORKING_DIRECTORY/{model_name}`. Files ending with `.csv` contain the evaluated metrics.
+
+### Frequently Asked Questions
+
+#### Constructing Input Prompt: The `build_prompt()` Function
+If you find that the model's output does not match the expected results when evaluating a specific benchmark, it could be due to the model not constructing the input prompt correctly.
+
+In VLMEvalKit, each `dataset` class includes a function named `build_prompt()`, which is responsible for formatting input questions. Different benchmarks can either customize their own `build_prompt()` function or use the default implementation.
+
+For instance, when handling the default [Multiple-Choice QA](https://github.com/open-compass/VLMEvalKit/blob/43af13e052de6805a8b08cd04aed5e0d74f82ff5/vlmeval/dataset/image_mcq.py#L164), the `ImageMCQDataset.build_prompt()` method combines elements such as `hint`, `question`, and `options` (if present in the dataset) into a complete question format, as shown below:
+
+```
+HINT
+QUESTION
+Options:
+A. Option A
+B. Option B
+···
+Please select the correct answer from the options above.
+```
+
+Additionally, since different models may have varying evaluation requirements, VLMEvalKit also supports customizing the prompt construction method at the model level through `model.build_prompt()`. For an example, you can refer to [InternVL](https://github.com/open-compass/VLMEvalKit/blob/43af13e052de6805a8b08cd04aed5e0d74f82ff5/vlmeval/vlm/internvl_chat.py#L324).
+
+**Note: If both `model.build_prompt()` and `dataset.build_prompt()` are defined, `model.build_prompt()` will take precedence over `dataset.build_prompt()`, effectively overriding it.**
+
+Some models, such as Qwen2VL and InternVL, define extensive prompt-building methods for various types of benchmarks. To provide more flexibility in adapting to different benchmarks, VLMEvalKit allows users to customize the `model.use_custom_prompt()` function within the model. By adding or modifying the `use_custom_prompt()` function, you can decide which benchmarks should utilize the model's custom prompt logic. Below is an example:
+
+```python
+def use_custom_prompt(self, dataset: str) -> bool:
+    from vlmeval.dataset import DATASET_TYPE, DATASET_MODALITY
+    dataset_type = DATASET_TYPE(dataset, default=None)
+    if not self._use_custom_prompt:
+        return False
+    if listinstr(['MMVet'], dataset):
+        return True
+    if dataset_type == 'MCQ':
+        return True
+    if DATASET_MODALITY(dataset) == 'VIDEO':
+        return False
+    return False
+```
+Only when the `use_custom_prompt()` function returns `True` will VLMEvalKit call the model's `build_prompt()` function for the current benchmark.
+With this approach, you can flexibly control which benchmarks use the model's custom prompt logic based on your specific needs, thereby better adapting to different models and tasks.
+
+#### Model Splitting
+
+Currently, VLMEvalKit automatically supports GPU resource allocation and model splitting between processes on the same machine. This feature is supported when the inference backend is `lmdeploy` or `transformers`, with the following behaviors:
+
+- When launching with `python` command, the model is by default allocated to all available GPUs. If you want to specify which GPUs to use, you can use `CUDA_VISIBLE_DEVICES` environment variable.
+- When starting with `torchrun` command, each model instance will be allocated to `N_GPU // N_PROC` GPUs, where `N_PROC` is the number of processes specified by the `--nproc-per-node` parameter in the torchrun command. The value of `N_GPU` is determined as follows:
+  - If `CUDA_VISIBLE_DEVICES` environment variable is not set, `N_GPU` will be the total number of available GPUs.
+  - If `CUDA_VISIBLE_DEVICES` environment variable is set, `N_GPU` will be the number of GPUs specified by the `CUDA_VISIBLE_DEVICES` environment variable, and only the specified GPUs will be utilized.
+Below are specific examples of running evaluation tasks on a machine equipped with 8 GPUs:
+
+```bash
+<!-- Launch two model instances in data parallel, each instance using 4 GPUs -->
+torchrun --nproc-per-node=2 run.py --data MMBench_DEV_EN --model InternVL3-78B
+<!-- Launch one model instance, using all 8 GPUs -->
+python run.py --data MMBench_DEV_EN --model InternVL3-78B
+<!-- Launch three model instances, each instance using 2 GPUs, GPU 0 and 7 are not used -->
+CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 torchrun --nproc-per-node=3 run.py --data MMBench_DEV_EN --model InternVL3-38B
+```
+
+PS: The feature is not compatible with `vllm` backend. When you evaluate a model with `vllm` backend, please use `python` to launch, and all visible GPU devices will be used.
+
+#### Performance Discrepancies
+
+Model performance may vary across different environments. As a result, you might observe discrepancies between your evaluation results and those listed on the official VLMEvalKit leaderboard. These differences could be attributed to variations in versions of libraries such as `transformers`, `cuda`, and `torch`.
+
+Besides, if you encounter unexpected performance, we recommend first reviewing the local generation records (`{model}_{dataset}.xlsx`) or the evaluation records (`{model}_{dataset}_{judge_model}.xlsx`). This may help you better understand the evaluation outcomes and identify potential issues.
+
+## Deploy a local language model as the judge / choice extractor
+The default setting mentioned above uses OpenAI's GPT as the judge LLM. However, you can also deploy a local judge LLM with [LMDeploy](https://github.com/InternLM/lmdeploy).
+
+First install:
+```
+pip install lmdeploy openai
+```
+
+And then deploy a local judge LLM with the single line of code. LMDeploy will automatically download the model from Huggingface. Assuming we use internlm2-chat-1_8b as the judge, port 23333, and the key sk-123456 (the key must start with "sk-" and follow with any number you like):
+```
+lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
+```
+
+You need to get the model name registered by LMDeploy with the following python code:
+```
+from openai import OpenAI
+client = OpenAI(
+    api_key='sk-123456',
+    base_url="http://0.0.0.0:23333/v1"
+)
+model_name = client.models.list().data[0].id
+```
+
+Now set some environment variables to tell VLMEvalKit how to use the local judge LLM. As mentioned above, you can also set them in `$VLMEvalKit/.env` file:
+```
+OPENAI_API_KEY=sk-123456
+OPENAI_API_BASE=http://0.0.0.0:23333/v1/chat/completions
+LOCAL_LLM=<model_name you get>
+```
+
+Finally, you can run the commands in step 2 to evaluate your VLM with the local judge LLM.
+
+Note that
+
+- If you hope to deploy the judge LLM in a single GPU and evaluate your VLM on other GPUs because of limited GPU memory, try `CUDA_VISIBLE_DEVICES=x` like
+```
+CUDA_VISIBLE_DEVICES=0 lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
+CUDA_VISIBLE_DEVICES=1,2,3 torchrun --nproc-per-node=3 run.py --data HallusionBench  --model qwen_chat --verbose
+```
+- If the local judge LLM is not good enough in following the instructions, the evaluation may fail. Please report such failures (e.g., by issues).
+- It's possible to deploy the judge LLM in different ways, e.g., use a private LLM (not from HuggingFace) or use a quantized LLM. Please refer to the [LMDeploy doc](https://lmdeploy.readthedocs.io/en/latest/serving/api_server.html). You can use any other deployment framework if they support OpenAI API.
+
+
+### Using LMDeploy to Accelerate Evaluation and Inference
+
+You can refer this [doc](/docs/en/EvalByLMDeploy.md)

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/_templates/404.html

@@ -0,0 +1,18 @@
+{% extends "layout.html" %}
+
+{% block body %}
+
+<h1>Page Not Found</h1>
+<p>
+  The page you are looking for cannot be found.
+</p>
+<p>
+  If you just switched documentation versions, it is likely that the page you were on is moved. You can look for it in
+  the content table left, or go to <a href="{{ pathto(root_doc) }}">the homepage</a>.
+</p>
+<!-- <p>
+  If you cannot find documentation you want, please <a
+    href="">open an issue</a> to tell us!
+</p> -->
+
+{% endblock %}

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/conf.py

@@ -0,0 +1,234 @@
+# flake8: noqa
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+
+import ast
+import os
+import subprocess
+import sys
+
+import pytorch_sphinx_theme
+from sphinx.builders.html import StandaloneHTMLBuilder
+
+sys.path.insert(0, os.path.abspath('../../'))
+
+# -- Project information -----------------------------------------------------
+
+project = 'VLMEvalKit'
+copyright = '2023, VLMEvalKit'
+author = 'VLMEvalKit Authors'
+
+# The full version, including alpha/beta/rc tags
+version_file = '../../vlmeval/__init__.py'
+
+
+def get_version():
+    with open(version_file, 'r') as f:
+        file_content = f.read()
+    # Parse the file content into an abstract syntax tree (AST)
+    tree = ast.parse(file_content, filename=version_file)
+
+    # Iterate through the body of the AST, looking for an assignment to __version__
+    for node in tree.body:
+        if isinstance(node, ast.Assign):
+            for target in node.targets:
+                if isinstance(target, ast.Name) and target.id == '__version__':
+                    return node.value.s
+    raise ValueError('__version__ not found')
+
+
+release = get_version()
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.viewcode',
+    'myst_parser',
+    'sphinx_copybutton',
+    'sphinx_tabs.tabs',
+    'notfound.extension',
+    'sphinxcontrib.jquery',
+    'sphinx_design',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+source_suffix = {
+    '.rst': 'restructuredtext',
+    '.md': 'markdown',
+}
+
+language = 'en'
+
+# The master toctree document.
+root_doc = 'index'
+html_context = {
+    'github_version': 'latest',
+}
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'pytorch_sphinx_theme'
+html_theme_path = [pytorch_sphinx_theme.get_html_theme_path()]
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+# yapf: disable
+html_theme_options = {
+    'menu': [
+        {
+            'name': 'GitHub',
+            'url': 'https://github.com/open-compass/VLMEvalKit'
+        },
+    ],
+    # Specify the language of shared menu
+    'menu_lang': 'en',
+    # Disable the default edit on GitHub
+    'default_edit_on_github': False,
+}
+# yapf: enable
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+html_css_files = [
+    'https://cdn.datatables.net/v/bs4/dt-1.12.1/datatables.min.css',
+    'css/readthedocs.css'
+]
+html_js_files = [
+    'https://cdn.datatables.net/v/bs4/dt-1.12.1/datatables.min.js',
+    'js/custom.js'
+]
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'vlmevalkitdoc'
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (root_doc, 'vlmevalkit.tex', 'VLMEvalKit Documentation', author,
+     'manual'),
+]
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [(root_doc, 'vlmevalkit', 'VLMEvalKit Documentation', [author],
+              1)]
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (root_doc, 'vlmevalkit', 'VLMEvalKit Documentation', author,
+     'VLMEvalKit Authors', 'AGI evaluation toolbox and benchmark.',
+     'Miscellaneous'),
+]
+
+# -- Options for Epub output -------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = project
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#
+# epub_identifier = ''
+
+# A unique identification for the text.
+#
+# epub_uid = ''
+
+# A list of files that should not be packed into the epub file.
+epub_exclude_files = ['search.html']
+
+# set priority when building html
+StandaloneHTMLBuilder.supported_image_types = [
+    'image/svg+xml', 'image/gif', 'image/png', 'image/jpeg'
+]
+
+# -- Extension configuration -------------------------------------------------
+# Ignore >>> when copying code
+copybutton_prompt_text = r'>>> |\.\.\. '
+copybutton_prompt_is_regexp = True
+
+# Auto-generated header anchors
+myst_heading_anchors = 3
+# Enable "colon_fence" extension of myst.
+myst_enable_extensions = ['colon_fence', 'dollarmath']
+
+# Configuration for intersphinx
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/3', None),
+    'numpy': ('https://numpy.org/doc/stable', None),
+    'torch': ('https://pytorch.org/docs/stable/', None),
+    'mmengine': ('https://mmengine.readthedocs.io/en/latest/', None),
+    'transformers':
+    ('https://huggingface.co/docs/transformers/main/en/', None),
+}
+napoleon_custom_sections = [
+    # Custom sections for data elements.
+    ('Meta fields', 'params_style'),
+    ('Data fields', 'params_style'),
+]
+
+# Disable docstring inheritance
+autodoc_inherit_docstrings = False
+# Mock some imports during generate API docs.
+autodoc_mock_imports = ['rich', 'attr', 'einops']
+# Disable displaying type annotations, these can be very verbose
+autodoc_typehints = 'none'
+
+# The not found page
+notfound_template = '404.html'

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/docutils.conf

@@ -0,0 +1,2 @@
+[html writers]
+table_style: colwidths-auto

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/en/index.rst

@@ -0,0 +1,41 @@
+Welcome to the VLMEvalKit Tutorial!
+==========================================
+
+VLMEvalKit Getting Started Guide
+-------------------------------
+
+To help users get started quickly, we recommend the following process:
+
+- For users who want to use VLMEvalKit, we recommend reading the "Start Your First Step" section to set up the environment and start a mini-experiment to familiarize yourself with the process.
+
+- If you want to customize more modules, such as adding datasets and models, we provide an "Advanced Tutorial."
+
+We always welcome users' PRs (Pull Requests) and Issues to improve VLMEvalKit!
+
+.. _Start Your First Step:
+.. toctree::
+   :maxdepth: 1
+   :caption: Start Your First Step
+
+   Quickstart.md
+
+.. _Advanced Tutorial:
+.. toctree::
+   :maxdepth: 1
+   :caption: Advanced Tutorial
+
+   Development.md
+   ConfigSystem.md
+
+.. _Other Notes:
+.. toctree::
+   :maxdepth: 1
+   :caption: Other Notes
+
+   Contributors.md
+
+Index and Tables
+==================
+
+* :ref:`genindex`
+* :ref:`search`

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/ja/README_ja.md

@@ -0,0 +1,117 @@
+<div align="center">
+
+![LOGO](http://opencompass.openxlab.space/utils/MMLB.jpg)
+
+<b>VLMEvalKit: 大規模視覚言語モデルの評価ツールキット</b>
+
+[![][github-contributors-shield]][github-contributors-link] • [![][github-forks-shield]][github-forks-link] • [![][github-stars-shield]][github-stars-link] • [![][github-issues-shield]][github-issues-link] • [![][github-license-shield]][github-license-link]
+
+[English](/README.md) | [简体中文](/docs/zh-CN/README_zh-CN.md) | 日本語
+
+<a href="https://rank.opencompass.org.cn/leaderboard-multimodal">🏆 OpenCompass Learderboard </a> •
+<a href="#-datasets-models-and-evaluation-results">📊Datasets & Models </a> •
+<a href="#%EF%B8%8F-quickstart">🏗️Quickstart </a> •
+<a href="#%EF%B8%8F-development-guide">🛠️Development </a> •
+<a href="#-the-goal-of-vlmevalkit">🎯Goal </a> •
+<a href="#%EF%B8%8F-citation">🖊️Citation </a>
+
+<a href="https://huggingface.co/spaces/opencompass/open_vlm_leaderboard">🤗 HF Leaderboard</a> •
+<a href="https://huggingface.co/datasets/VLMEval/OpenVLMRecords">🤗 Evaluation Records</a> •
+<a href="https://discord.gg/evDT4GZmxN">🔊 Discord Channel</a> •
+<a href="https://www.arxiv.org/abs/2407.11691">📝 Technical Report</a>
+</div>
+
+**VLMEvalKit**（pythonパッケージ名は**vlmeval**）は、**大規模視覚言語モデル（LVLMs）**の**オープンソース評価ツールキット**です。このツールキットは、複数のリポジトリでのデータ準備という重労働なしに、さまざまなベンチマークでLVLMsの**ワンコマンド評価**を可能にします。VLMEvalKitでは、すべてのLVLMsに対して**生成ベースの評価**を採用し、**正確なマッチング**と**LLMベースの回答抽出**の両方で得られた評価結果を提供します。
+
+PS: 日本語の README には最新のアップデートがすべて含まれていない場合があります。英語版をご確認ください。
+
+## 📊 データセット、モデル、および評価結果
+
+**公式のマルチモーダルリーダーボードでのパフォーマンス数値は、ここからダウンロードできます！**
+
+[**OpenVLM Leaderboard**](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard): [すべての詳細な結果をダウンロード](http://opencompass.openxlab.space/assets/OpenVLM.json)。
+
+**Supported Benchmarks** in [**VLMEvalKit Features**](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb) を確認して、すべてのサポートされているベンチマーク（70以上）を表示してください。
+
+**Supported LMMs** in [**VLMEvalKit Features**](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb) を確認して、すべてのサポートされている LMMs（200以上）を表示してください。
+
+**Transformersバージョンの推奨事項:**
+
+特定のtransformerバージョンで一部のVLMが実行できない可能性があることに注意してください。各VLMを評価するために、以下の設定を推奨します：
+
+- **`transformers==4.33.0`を使用してください**: `Qwenシリーズ`, `Monkeyシリーズ`, `InternLM-XComposerシリーズ`, `mPLUG-Owl2`, `OpenFlamingo v2`, `IDEFICSシリーズ`, `VisualGLM`, `MMAlaya`, `ShareCaptioner`, `MiniGPT-4シリーズ`, `InstructBLIPシリーズ`, `PandaGPT`, `VXVERSE`, `GLM-4v-9B`.
+- **`transformers==4.37.0`を使用してください**: `LLaVAシリーズ`, `ShareGPT4Vシリーズ`, `TransCore-M`, `LLaVA (XTuner)`, `CogVLMシリーズ`, `EMU2シリーズ`, `Yi-VLシリーズ`, `MiniCPM-[V1/V2]`, `OmniLMM-12B`, `DeepSeek-VLシリーズ`, `InternVLシリーズ`, `Cambrianシリーズ`, `VILA-VLシリーズ`.
+- **`transformers==4.40.0`を使用してください**: `IDEFICS2`, `Bunny-Llama3`, `MiniCPM-Llama3-V2.5`, `360VL-70B`, `Phi-3-Vision`, `WeMM`.
+- **`transformers==4.42.0`を使用してください**: `AKI`.
+- **`transformers==latest`を使用してください**: `LLaVA-Nextシリーズ`, `PaliGemma-3B`, `Chameleon-VLシリーズ`, `Video-LLaVA-7B-HF`, `Ovis1.5シリーズ`, `Mantisシリーズ`, `MiniCPM-V2.6`.
+
+```python
+# デモ
+from vlmeval.config import supported_VLM
+model = supported_VLM['idefics_9b_instruct']()
+# 単一画像のフォワード
+ret = model.generate(['assets/apple.jpg', 'この画像には何がありますか？'])
+print(ret)  # この画像には葉がついた赤いリンゴがあります。
+# 複数画像のフォワード
+ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', '提供された画像にはリンゴが何個ありますか？'])
+print(ret)  # 提供された画像にはリンゴが2個あります。
+```
+
+## 🏗️ クイックスタート
+
+クイックスタートガイドについては、[クイックスタート](/docs/en/Quickstart.md)を参照してください。
+
+## 🛠️ 開発ガイド
+
+カスタムベンチマーク、VLMsを開発するか���単に**VLMEvalKit**に他のコードを貢献する場合は、[開発ガイド](/docs/en/Development.md)を参照してください。
+
+コミュニティからの共有を奨励し、それに応じたクレジットを共有するために、次回のレポート更新では以下のことを実施します：
+
+- 全ての貢献に対して感謝の意を示します
+- 新しいモデル、評価セット、または主要な機能への3つ以上の主要な貢献を持つ貢献者は、テクニカルレポートの著者リストに加わることができます。適格な貢献者は、issueを作成するか、または[VLM評価キット ディスコードチャンネル](https://discord.com/invite/evDT4GZmxN)で kennyutc にDMを送ることができます。私たちはそれに応じてフォローアップします。
+
+## 🎯 VLMEvalKitの目標
+
+**このコードベースは以下を目的として設計されています：**
+
+1. 研究者や開発者が既存のLVLMsを評価し、評価結果を**簡単に再現できるようにする**ための**使いやすい**、**オープンソースの評価ツールキット**を提供します。
+2. VLMの開発者が自分のモデルを簡単に評価できるようにします。複数のサポートされているベンチマークでVLMを評価するには、単一の`generate_inner()`関数を**実装するだけで**、他のすべてのワークロード（データのダウンロード、データの前処理、予測の推論、メトリックの計算）はコードベースによって処理されます。
+
+**このコードベースは以下を目的として設計されていません：**
+
+1. すべての**第三者ベンチマーク**の元の論文で報告された正確な精度数値を再現すること。その理由は2つあります：
+   1. VLMEvalKitは、すべてのVLMに対して**生成ベースの評価**を使用します（オプションで**LLMベースの回答抽出**を使用）。一方、一部のベンチマークは異なるアプローチを使用する場合があります（SEEDBenchはPPLベースの評価を使用します）。これらのベンチマークについては、対応する結果で両方のスコアを比較します。開発者には、コードベースで他の評価パラダイムをサポートすることをお勧めします。
+   2. デフォルトでは、すべてのVLMに対して同じプロンプトテンプレートを使用してベンチマークを評価します。一方、**一部のVLMには特定のプロンプトテンプレートがある**場合があります（現時点ではコードベースでカバーされていない場合があります）。VLMの開発者には、現在カバーされていない場合でも、VLMEvalKitで独自のプロンプトテンプレートを実装することをお勧めします。これにより、再現性が向上します。
+
+## 🖊️ 引用
+
+この作業が役立つ場合は、このリポジトリに**スター🌟**を付けてください。サポートありがとうございます！
+
+[![Stargazers repo roster for @open-compass/VLMEvalKit](https://reporoster.com/stars/open-compass/VLMEvalKit)](https://github.com/open-compass/VLMEvalKit/stargazers)
+
+研究でVLMEvalKitを使用する場合、または公開されたオープンソースの評価結果を参照する場合は、以下のBibTeXエントリと、使用した特定のVLM/ベンチマークに対応するBibTexエントリを使用してください。
+
+```bib
+@misc{duan2024vlmevalkit,
+      title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models},
+      author={Haodong Duan and Junming Yang and Yuxuan Qiao and Xinyu Fang and Lin Chen and Yuan Liu and Xiaoyi Dong and Yuhang Zang and Pan Zhang and Jiaqi Wang and Dahua Lin and Kai Chen},
+      year={2024},
+      eprint={2407.11691},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2407.11691},
+}
+```
+
+<p align="right"><a href="#top">🔝Top に戻る</a></p>
+
+[github-contributors-link]: https://github.com/open-compass/VLMEvalKit/graphs/contributors
+[github-contributors-shield]: https://img.shields.io/github/contributors/open-compass/VLMEvalKit?color=c4f042&labelColor=black&style=flat-square
+[github-forks-link]: https://github.com/open-compass/VLMEvalKit/network/members
+[github-forks-shield]: https://img.shields.io/github/forks/open-compass/VLMEvalKit?color=8ae8ff&labelColor=black&style=flat-square
+[github-issues-link]: https://github.com/open-compass/VLMEvalKit/issues
+[github-issues-shield]: https://img.shields.io/github/issues/open-compass/VLMEvalKit?color=ff80eb&labelColor=black&style=flat-square
+[github-license-link]: https://github.com/open-compass/VLMEvalKit/blob/main/LICENSE
+[github-license-shield]: https://img.shields.io/github/license/open-compass/VLMEvalKit?color=white&labelColor=black&style=flat-square
+[github-stars-link]: https://github.com/open-compass/VLMEvalKit/stargazers
+[github-stars-shield]: https://img.shields.io/github/stars/open-compass/VLMEvalKit?color=ffcb47&labelColor=black&style=flat-square

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/.readthedocs.yaml

@@ -0,0 +1,17 @@
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.8"
+
+formats:
+    - epub
+
+sphinx:
+  configuration: docs/zh-CN/conf.py
+
+python:
+  install:
+    - requirements: requirements/docs.txt

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/ConfigSystem.md

@@ -0,0 +1,69 @@
+
+# 配置系统
+
+默认情况下，VLMEvalKit通过在`run.py`脚本中使用`--model`和`--data`参数设置模型名称（在`/vlmeval/config.py`中定义）和数据集名称（在`vlmeval/dataset/__init__.py` 或 `vlmeval/dataset/video_dataset_config.py` 中定义）来启动评估。这种方法在大多数情况下简单且高效，但当用户希望使用不同设置评估多个模型/数据集时，可能不够灵活。
+
+为了解决这个问题，VLMEvalKit提供了一个更灵活的配置系统。用户可以在json文件中指定模型和数据集设置，并通过`--config`参数将配置文件的路径传递给`run.py`脚本。以下是一个示例配置json：
+
+```json
+{
+    "model": {
+        "GPT4o_20240806_T00_HIGH": {
+            "class": "GPT4V",
+            "model": "gpt-4o-2024-08-06",
+            "temperature": 0,
+            "img_detail": "high"
+        },
+        "GPT4o_20240806_T10_Low": {
+            "class": "GPT4V",
+            "model": "gpt-4o-2024-08-06",
+            "temperature": 1.0,
+            "img_detail": "low"
+        },
+        "GPT4o_20241120": {}
+    },
+    "data": {
+        "MME-RealWorld-Lite": {
+            "class": "MMERealWorld",
+            "dataset": "MME-RealWorld-Lite"
+        },
+        "MMBench_DEV_EN_V11": {
+            "class": "ImageMCQDataset",
+            "dataset": "MMBench_DEV_EN_V11"
+        },
+        "MMBench_Video_8frame_nopack":{},
+        "Video-MME_16frame_subs": {
+            "class": "VideoMME",
+            "dataset": "Video-MME",
+            "nframe": 16,
+            "use_subtitle": true
+        }
+    }
+}
+```
+
+配置json的解释：
+
+1. 现在我们支持两个字段：`model`和`data`，每个字段都是一个字典。字典的键是模型/数据集的名称（由用户设置），值是模型/数据集的设置。
+2. 对于`model`中的项目，值是一个包含以下键的字典：
+    - `class`：模型的类名，应该是`vlmeval/vlm/__init__.py`（开源模型）或`vlmeval/api/__init__.py`（API模型）中定义的类名。
+    - 其他kwargs：其他kwargs是模型特定的参数，请参考模型类的定义以获取详细用法。例如，`model`、`temperature`、`img_detail`是`GPT4V`类的参数。值得注意的是，大多数模型类都需要`model`参数。
+    - Tip：在位于`vlmeval/config.py`的变量`supported_VLM`中的已经被定义的模型可以作为`model`的键，而不需要填对应的值即可启动。例如，`GPT4o_20240806_T00_HIGH: {}`是等价于`GPT4o_20240806_T00_HIGH: {'class': 'GPT4V', 'model': 'gpt-4o-2024-08-06', 'temperature': 0, 'img_size': -1, 'img_detail': 'high', 'retry': 10, 'verbose': False}`。
+3. 对于字典`data`，我们建议用户使用官方数据集名称作为键（或键的一部分），因为我们经常根据数据集名称确定后处理/判断设置。对于`data`中的项目，值是一个包含以下键的字典：
+    - `class`：数据集的类名，应该是`vlmeval/dataset/__init__.py`中定义的类名。
+    - 其他kwargs：其他kwargs是数据集特定的参数，请参考数据集类的定义以获取详细用法。通常，大多数数据集类都需要`dataset`参数。大多数视频数据集类都需要 `nframe` 或 `fps` 参数。
+    - Tip：在位于`vlmeval/dataset/video_dataset_config.py`的变量`supported_video_dataset`中的已经被定义的数据集可以作为`data`的键，而不需要填对应的值即可启动。例如，`MMBench_Video_8frame_nopack: {}`是等价于`MMBench_Video_8frame_nopack: {'class': 'MMBenchVideo', 'dataset': 'MMBench-Video', 'nframe': 8, 'pack': False}`。
+
+将示例配置json保存为`config.json`，您可以通过以下命令启动评估：
+
+```bash
+python run.py --config config.json
+```
+
+这将在工作目录`$WORK_DIR`下生成以下输出文件（格式为`{$WORK_DIR}/{$MODEL_NAME}/{$MODEL_NAME}_{$DATASET_NAME}_*`）：
+
+- `$WORK_DIR/GPT4o_20240806_T00_HIGH/GPT4o_20240806_T00_HIGH_MME-RealWorld-Lite*`
+- `$WORK_DIR/GPT4o_20240806_T10_Low/GPT4o_20240806_T10_Low_MME-RealWorld-Lite*`
+- `$WORK_DIR/GPT4o_20240806_T00_HIGH/GPT4o_20240806_T00_HIGH_MMBench_DEV_EN_V11*`
+- `$WORK_DIR/GPT4o_20240806_T10_Low/GPT4o_20240806_T10_Low_MMBench_DEV_EN_V11*`
+......

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/Development.md

@@ -0,0 +1,139 @@
+# 🛠️ 如何在 VLMEvalKit 中实现一个新的 Benchmark 或多模态模型（VLM）
+
+## 实现一个新的 benchmark
+
+示例 PR: **添加 Math-Vision Benchmark** ([#292](https://github.com/open-compass/VLMEvalKit/pull/292/files))
+
+目前在 VLMEvalKit 中，benchmark 以数据集类的形式呈现，当你新增一个 benchmark 时，你可以选择复用现有的数据集类 (如单选题 benchmark 可复用 `ImageMCQDataset`)，或是实现新的数据集类。你的数据集类必须支持以下两种方法 (复用父类或自行实现):
+
+- `build_prompt(self, line)`: 方法输入 `line` 类型为 int (对应数据 index) 或 `pd.Series` (对应数据原始 record)。方法输出一条 `multi-modal message` 作为多模态模型输入，`multi-modal message` 是一个图文交错的列表，如以下格式 (一图一文): `[dict(type='image', value=IMAGE_PTH), dict(type='text', value=prompt)]`。
+- `evaluate(self, eval_file, **judge_kwargs)`: 方法输入 `eval_file` 为多模态模型的预测结果 (多以 `.xlsx` 格式存在)，如 benchmark evaluation 需要大语言模型 (一般为 GPT) 辅助，则 `judge_kwargs` 传入大语言模型的参数。方法输出 benchmark 的评测结果，以 `dict` 或 `pd.DataFrame` 的形式。
+
+以下，我们简述新增数据集的通常步骤：
+
+### 1. TSV 数据文件准备 (图文评测集)
+
+目前，我们将每一个 benchmark 数据集设置为一个单独的 TSV 文件。在推理过程中，数据文件将从数据集定义的 `DATASET_URL` 链接地址自动下载到 `$LMUData` 中（如果没有明确设置的话，默认路径是 `$HOME/LMUData`）。你可以将准备好的 TSV 文件上传到一个可下载的地址（如：huggingface），或发送给我们 <opencompass@pjlab.org.cn>，我们将帮助上传数据集到服务器中。此外，你也可以在环境变量中自定义设置下载路径 `LMUData=/path/to/your/data`。
+
+TSV 文件中的内容组成为：
+
+| 数据集名称 \ 字段  | index | image | image_path | question | hint | multi-choice<br>options | answer | category | l2-category | split |
+| ---------------------- | ----- | ----- | ---------- | -------- | ---- | ----------------------- | ------ | -------- | ----------- | ----- |
+| MMBench_DEV_[CN/EN]    | ✅     | ✅     |            | ✅        | ✅    | ✅                       | ✅      | ✅        | ✅           | ✅     |
+| MMBench_TEST_[CN/EN]   | ✅     | ✅     |            | ✅        | ✅    | ✅                       |        | ✅        | ✅           | ✅     |
+| CCBench                | ✅     | ✅     |            | ✅        |      | ✅                       | ✅      | ✅        |             |       |
+| SEEDBench_IMG          | ✅     | ✅     |            | ✅        |      | ✅                       | ✅      | ✅        |             |       |
+| MME                    | ✅     | ✅     |            | ✅        |      |                         | ✅      | ✅        |             |       |
+| MMVet                  | ✅     | ✅     |            | ✅        |      |                         | ✅      | ✅        |             |       |
+| MMMU_DEV_VAL           | ✅     | ✅     | ✅          | ✅        |      | ✅                       | ✅      | ✅        | ✅           | ✅     |
+| COCO_VAL               | ✅     | ✅     |            |          |      |                         | ✅      |          |             |       |
+| OCRVQA_[TEST/TESTCORE] | ✅     | ✅     |            | ✅        |      |                         | ✅      |          |             |       |
+| TextVQA_VAL            | ✅     | ✅     |            | ✅        |      |                         | ✅      |          |             |       |
+| VCR_[EN/ZH]\_[EASY/HARD]_[ALL/500/100]            | ✅     | ✅     |            | ✅        |      |                         | ✅      |          |             |       |
+
+<div align="center"><b>表 1. 支持的数据集的 TSV 字段。</b></div>
+
+**TSV 中必须字段的介绍：**
+
+- **index:** 一个整数，`tsv` 中每一行的唯一标识
+- **image:** 图片的 base64 编码，你可以使用 `vlmeval/smp/vlm.py` 中实现的API进行编码和解码：
+    - 编码：`encode_image_to_base64`（对于PIL Image）/ `encode_image_file_to_base64`（对于图片文件路径）
+    - 解码：`decode_base64_to_image`（对于PIL Image）/ `decode_base64_to_image_file`（对于图片文件路径）
+- **question:** 针对图像所提取出的问题，类型为字符串
+- **answer:** 问题的答案，类型为字符串，Test 集可缺失这一字段
+
+### 2. 自定义数据集的 prompt 构建
+
+`ImageBaseDataset` 定义了默认的 prompt 格式。如果需要针对数据集添加 prompt，或给模型输入 `Interleave` 的数据格式，可以通过 `build_prompt(line)` 函数实现。该函数输入为，每次给定 TSV 文件中的一行，包含 index, image, question 等内容作为 line。该函数将返回一个多模态消息 `msg` 的字典列表 `[dict(type='image', value=IMAGE_PTH), dict(type='text', value=prompt)]`，包括图片路径和将被输入到 VLMs 的文本 prompt。对于 interleave 类型输入，可以直接将图片路径的字典放置到 image token 位置。
+
+### 3. 自定义数据集的指标实现
+
+增加对 benchmark 的评测需要自定义一个该数据集的 class 对象，从而实现数据集的指标计算。图文多模态数据集均继承自 `vlmeval/dataset/image_base.py` 中的 `ImageBaseDataset` 对象。其中 `TYPE` 定义了数据集的类型；`DATASET_URL` 为数据集的下载地址；`DATASET_MD5` 为数据集文件的 md5 一致性编码检查。
+
+在 class 中**需要实现** `evaluate(eval_file, **judge_kwargs)` 类函数，对自定义的数据集结果进行指标计算和结果输出。函数输入 `eval_file` 为模型预测结果 `{model_name}_{dataset}.xlsx` 的路径。可以通过 `load(eval_file)` 文件将其读取为 panda.DataFrames 类型，其中包含 index, question, answer, category, prediction 等字段。`judge_kwargs` 参数将传递一个评测相关的字典，如：judge 模型的名称，api 请求线程数等。**函数的返回值**为评估完成的准确度等指标，其格式为由 list 组成的字典，并组织成 panda.DataFrames 类型。
+
+## 实现一个新的模型
+
+示例 PR: **支持 LLaVA-Next-Interleave** ([#294](https://github.com/open-compass/VLMEvalKit/pull/294))
+
+**1. 支持 `generate_inner` API (必须)**
+
+现有所有的模型都在 `vlmeval/vlm` 中实现。对于一个最基本的模型，你的模型类**应该实现方法** `generate_inner(msgs, dataset=None)`。这个函数将向 VLM 输入一个多模态数据，并返回 VLM 的预测（一个字符串）。可选参数 `dataset` 可以用作模型在不同推理策略之间切换的标志。
+
+其中多模态消息 `msgs` 是一个字典列表，每个字典有两个键：类型和值：
+- `type`：我们目前支持两种类型，选项是 ["image", "text"]。
+- `value`：当类型为 `text` 时，值是文本消息（一个字符串）；当类型为 `image` 时，值可以是图像文件的本地路径，或者是图像的URL。
+
+> 目前，一个多模态消息可能包含任意交错的图像和文本。如果你的模型不支持这一点，我们推荐的做法是取第一张图像和连接的文本消息作为模型的输入。你可以在模型的 class 中设置 `INTERLEAVE = False` 并调用 `self.message_to_promptimg(message, dataset=dataset)` 函数来获取你的 prompt 和第一张图片的地址。
+
+一些多模态消息的例子:
+
+```python
+IMAGE_PTH = 'assets/apple.jpg'
+IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
+msg1 = [
+    dict(type='image', value=IMAGE_PTH),
+    dict(type='text', value='What is in this image?')
+]
+msg2 = [
+    dict(type='image', value=IMAGE_URL),
+    dict(type='image', value=IMAGE_URL),
+    dict(type='text', value='How many apples are there in these images?')
+]
+response = model.generate(msg1)
+```
+
+为了方便起见，我们还支持接受字符串列表作为输入。在这种情况下，我们将检查一个字符串是图像路径还是图像 URL，并自动将其转换为 `list[dict]` 格式：
+
+```python
+IMAGE_PTH = 'assets/apple.jpg'
+IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
+msg1 = [IMAGE_PTH, 'What is in this image?']
+msg2 = [IMAGE_URL, IMAGE_URL,  'How many apples are there in these images?']
+response = model.generate(msg1)
+```
+
+**2. 支持自定义提示词构建 (可选)**
+
+此外，你的模型可以通过实现两个可选方法来支持自定义提示构建：`use_custom_prompt(dataset)` 和 `build_prompt(line, dataset=None)`。
+
+- `use_custom_prompt(dataset)` 将返回一个布尔值，指示模型是否应使用自定义提示构建策略。
+- 如果`use_custom_prompt(dataset)`返回 True，`build_prompt(line, dataset)` 应该为相应的数据集返回一个自定义构建的多模态消息，line 数据是一个包含数据样本所需信息的字典。如果`use_custom_prompt(dataset)` 返回False，则将使用默认的 prompt 构建策略。
+
+**3. 支持多轮对话 (可选)**
+
+你可以通过支持 `chat_inner(message, dataset)` API 为你的模型新增多轮对话功能并兼容多轮对话评测。这个 API 输出一个字符串型回复，`message` 包含一个聊天记录的列表，格式如下：
+
+```python
+# Assume msg1, msg2, msg3, ... are multi-modal messages following the previously described format
+# `chat_inner` take the following chat history list as input:
+message = [
+    dict(role='user', content=msg1),
+    dict(role='assistant', content=msg2),
+    dict(role='user', content=msg3),
+    dict(role='assistant', content=msg4),
+	......
+    dict(role='user', content=msgn),
+]
+# `message` should contain an odd number of chat utterances, the role of utterances should be interleaved "user" and "assistant", with the role of the last utterance to be "user".
+# The chat function will call `chat_inner`
+response = model.chat(message)
+```
+
+### 示例 PRs：
+
+- 不支持交错的图像和文本，且不使用自定义提示的VLM：[[模型] 支持 glm-4v-9b](https://github.com/open-compass/VLMEvalKit/pull/221)
+- 支持交错的图像和文本及自定义提示的VLM：[添加 MiniCPM-Llama3-V-2.5](https://github.com/open-compass/VLMEvalKit/pull/205)
+- VLM API：[特征添加 glmv](https://github.com/open-compass/VLMEvalKit/pull/201)
+
+## 为 VLMEvalKit 贡献代码
+
+如果你想为 **VLMEvalKit** 贡献代码，请在提交PR之前进行预提交检查。这有助于保持代码整洁。
+
+```bash
+# 在VLMEvalKit的目录下，安装预提交 hook:
+pip install pre-commit
+pre-commit install
+pre-commit run --all-files
+# 然后提交你的代码。
+```

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/EvalByLMDeploy.md

@@ -0,0 +1,28 @@
+# 使用 LMDeploy 加速评测推理
+
+VLMEvalKit 支持测试由 LMDeploy 部署的 VLM 模型，下面以 InternVL2-8B 为例，展示如何测试模型
+
+## 第0步 安装 LMDeploy
+
+```bash
+pip install lmdeploy
+```
+
+其他安装方式可以参考 LMDeploy 的[文档](https://github.com/InternLM/lmdeploy)
+
+## 第1步 启动推理服务
+
+```bash
+lmdeploy serve api_server OpenGVLab/InternVL2-8B --model-name InternVL2-8B
+```
+> [!IMPORTANT]
+> 因为 VLMEvalKit 中的模型对于不同数据集在构建 prompt 时可能有自定义行为，如 InternVL2 对于 HallusionBench 的处理，所以，server 端在启动的时候需要指定 `--model-name`，这样在使用 LMDEploy api 时可以根据名字选择合适的 prompt 构建策略。
+>
+> 如果指定了 `--server-port`，需要设置对应的环境变量 `LMDEPLOY_API_BASE`
+
+
+## 第2步 评测
+
+```bash
+python run.py --data MMStar --model InternVL2-8B --verbose --api-nproc 64
+```

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/Quickstart.md

@@ -0,0 +1,231 @@
+# 快速开始
+
+在运行评测脚本之前，你需要先**配置** VLMs，并正确设置模型路径。然后你可以使用脚本 `run.py` 进行多个VLMs和基准测试的推理和评估。
+
+## 第0步 安装和设置必要的密钥
+
+**安装**
+
+```bash
+git clone https://github.com/open-compass/VLMEvalKit.git
+cd VLMEvalKit
+pip install -e .
+```
+
+**设置密钥**
+
+要使用 API 模型（如 GPT-4v, Gemini-Pro-V 等）进行推理，或使用 LLM API 作为**评判者或选择提取器**，你需要首先设置 API 密钥。如果你设置了密钥，VLMEvalKit 将使用一个评判 LLM 从输出中提取答案，否则它将使用**精确匹配模式**（在输出字符串中查找 "Yes", "No", "A", "B", "C"...）。**精确匹配模式只能应用于是或否任务和多项选择任务。**
+
+- 你可以将所需的密钥放在 `$VLMEvalKit/.env` 中，或直接将它们设置为环境变量。如果你选择创建 `.env` 文件，其内容将如下所示：
+
+  ```bash
+  # .env 文件，将其放置在 $VLMEvalKit 下
+  # 专有 VLMs 的 API 密钥
+  # QwenVL APIs
+  DASHSCOPE_API_KEY=
+  # Gemini w. Google Cloud Backends
+  GOOGLE_API_KEY=
+  # OpenAI API
+  OPENAI_API_KEY=
+  OPENAI_API_BASE=
+  # StepAI API
+  STEPAI_API_KEY=
+  # REKA API
+  REKA_API_KEY=
+  # GLMV API
+  GLMV_API_KEY=
+  # CongRong API
+  CW_API_BASE=
+  CW_API_KEY=
+  # SenseNova API
+  SENSENOVA_API_KEY=
+  # Hunyuan-Vision API
+  HUNYUAN_SECRET_KEY=
+  HUNYUAN_SECRET_ID=
+  # LMDeploy API
+  LMDEPLOY_API_BASE=
+  # MiniMax API
+  MINIMAX_API_KEY=
+  # 你可以设置一个评估时代理，评估阶段产生的 API 调用将通过这个代理进行
+  EVAL_PROXY=
+  ```
+
+- 如果需要使用 API 在对应键值空白处填写上你的密钥。这些 API 密钥将在进行推理和评估时自动加载。
+## 第1步 配置
+
+**VLM 配置**：所有 VLMs 都在 `vlmeval/config.py` 中配置。对于某些 VLMs（如 MiniGPT-4、LLaVA-v1-7B），需要额外的配置（在配置文件中配置代码 / 模型权重根目录）。在评估时，你应该使用 `vlmeval/config.py` 中 `supported_VLM` 指定的模型名称来选择 VLM。确保在开始评估之前，你可以成功使用 VLM 进行推理，使用以下命令 `vlmutil check {MODEL_NAME}`。
+
+注：对于Qwen-VL系列模型（Qwen-VL, Qwen2-VL, Qwen2.5-VL），vlmeval/config.py 中所指定的像素数量上下界如下：
+
+```
+min_pixels=1280 * 28 * 28,
+max_pixels=16384 * 28 * 28,
+```
+其中，1280为Qwen官方为平衡性能、计算资源与内存的推荐最大值，而16384为模型输入的理论最大值。这种设定对于部分需要高分辨率的视觉任务（如文档理解）有着积极的作用。但考虑这一设定并没有实际的依据，如果需要与官方的设定对齐，可以去掉这两个数值，或是设置为以下来自Qwen官方demo的数值：
+
+```
+min_pixels=256 * 28 * 28,
+max_pixels=1280 * 28 * 28,
+```
+
+## 第2步 评测
+
+**新功能!!!** 我们集成了一个新的配置系统，以实现更灵活的评估设置。查看[文档](/docs/zh-CN/ConfigSystem.md)或运行`python run.py --help`了解更多详情 🔥🔥🔥
+
+我们使用 `run.py` 进行评估。你可以使用 `$VLMEvalKit/run.py` 或创建脚本的软链接运行（以便在任何地方使用该脚本）：
+
+**参数**
+
+- `--data (list[str])`: 设置在 VLMEvalKit 中支持的数据集名称（可以在代码库首页的 README 中找到支持的数据集列表）
+- `--model (list[str])`: 设置在 VLMEvalKit 中支持的 VLM 名称（在 `vlmeval/config.py` 中的 `supported_VLM` 中定义）
+- `--mode (str, 默认值为 'all', 可选值为 ['all', 'infer'])`：当 mode 设置为 "all" 时，将执行推理和评估；当设置为 "infer" 时，只执行推理
+- `--api-nproc (int, 默认值为 4)`: 调用 API 的线程数
+- `--work-dir (str, default to '.')`: 存放测试结果的目录
+
+**用于评测图像多模态评测集的命令**
+
+你可以使用 `python` 或 `torchrun` 来运行脚本:
+
+```bash
+# 使用 `python` 运行时，只实例化一个 VLM，并且它可能使用多个 GPU。
+# 这推荐用于评估参数量非常大的 VLMs（如 IDEFICS-80B-Instruct）。
+
+# 在 MMBench_DEV_EN、MME 和 SEEDBench_IMG 上使用 IDEFICS-80B-Instruct 进行推理和评估
+python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose
+# 在 MMBench_DEV_EN、MME 和 SEEDBench_IMG 上使用 IDEFICS-80B-Instruct 仅进行推理
+python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose --mode infer
+
+# 使用 `torchrun` 运行时，每个 GPU 上实例化一个 VLM 实例。这可以加快推理速度。
+# 但是，这仅适用于消耗少量 GPU 内存的 VLMs。
+
+# 在 MMBench_DEV_EN、MME 和 SEEDBench_IMG 上使用 IDEFICS-9B-Instruct、Qwen-VL-Chat、mPLUG-Owl2。在具有 8 个 GPU 的节点上进行推理和评估。
+torchrun --nproc-per-node=8 run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct qwen_chat mPLUG-Owl2 --verbose
+# 在 MME 上使用 Qwen-VL-Chat。在具有 2 个 GPU 的节点上进行推理和评估。
+torchrun --nproc-per-node=2 run.py --data MME --model qwen_chat --verbose
+```
+
+**用于评测视频多模态评测集的命令**
+
+```bash
+# 使用 `python` 运行时，只实例化一个 VLM，并且它可能使用多个 GPU。
+# 这推荐用于评估参数量非常大的 VLMs（如 IDEFICS-80B-Instruct）。
+
+# 在 MMBench-Video 上评测 IDEFCIS2-8B, 视频采样 8 帧作为输入，不采用 pack 模式评测. MMBench_Video_8frame_nopack 是一个定义在 `vlmeval/dataset/video_dataset_config.py` 的数据集设定.
+torchrun --nproc-per-node=8 run.py --data MMBench_Video_8frame_nopack --model idefics2_8
+# 在 MMBench-Video 上评测 GPT-4o (API 模型), 视频采样每秒一帧作为输入，采用 pack 模式评测
+python run.py --data MMBench_Video_1fps_pack --model GPT4o
+```
+
+评估结果将作为日志打印出来。此外，**结果文件**也会在目录 `$YOUR_WORKING_DIRECTORY/{model_name}` 中生成。以 `.csv` 结尾的文件包含评估的指标。
+### 常见问题
+#### 构建输入prompt：`build_prompt()`函数
+如果您在评测某个benchmark时，发现模型输出的结果与预期不符，可能是因为您使用的模型没有正确构建输入prompt。
+
+在VLMEvalkit中，每个`dataset`类都包含一个名为`build_prompt()`的函数，用于构建输入问题的格式。不同的benchmark可以选择自定义`build_prompt()`函数，也可以使用默认的实现。
+
+例如，在处理默认的[多选题/Multi-Choice QA]([vlmeval/dataset/image_mcq.py](https://github.com/open-compass/VLMEvalKit/blob/43af13e052de6805a8b08cd04aed5e0d74f82ff5/vlmeval/dataset/image_mcq.py#L164))时，`ImageMCQDataset.build_prompt()`类会将`hint`、`question`、`options`等元素（若数据集中包含）组合成一个完整的问题格式，如下所示：
+```
+HINT
+QUESTION
+Options:
+A. Option A
+B. Option B
+···
+Please select the correct answer from the options above.
+```
+
+此外，由于不同模型对评测的需求可能有所不同，VLMEvalkit也支持在模型层面自定义对不同benchmark构建prompt的方法，即`model.build_prompt()`，具体示例可以参考[InternVL](https://github.com/open-compass/VLMEvalKit/blob/43af13e052de6805a8b08cd04aed5e0d74f82ff5/vlmeval/vlm/internvl_chat.py#L324)。
+
+**注意：当同时定义了`model.build_prompt()`以及`dataset.build_prompt()`时，`model.build_prompt()`将优先于`dataset.build_prompt()`，即前者会覆盖后者。**
+
+由于部分模型（如Qwen2VL，InternVL等）对于不同类型的benchmark定义了广泛的prompt构建方法，为了更灵活地适应不同的benchmark，VLMEvalkit支持在模型中自定义`model.use_custom_prompt()`函数。通过添加或者修改`use_custom_prompt()`函数，您可以决定对于哪些benchmark使用模型自定义的`use_custom_prompt()`方法，示例如下：
+```
+def use_custom_prompt(self, dataset: str) -> bool:
+    from vlmeval.dataset import DATASET_TYPE, DATASET_MODALITY
+    dataset_type = DATASET_TYPE(dataset, default=None)
+    if not self._use_custom_prompt:
+        return False
+    if listinstr(['MMVet'], dataset):
+        return True
+    if dataset_type == 'MCQ':
+        return True
+    if DATASET_MODALITY(dataset) == 'VIDEO':
+        return False
+    return False
+```
+仅当`use_custom_prompt()`函数返回`True`时，VLMEvalkit才会对当前benchmark调用模型的`build_prompt()`函数。
+通过这种方式，您可以根据具体需求灵活地控制哪些benchmark使用模型自定义的prompt构建逻辑，从而更好地适配不同模型和任务的需求。
+
+#### 模型切分
+
+目前 VLMEvalKit 的启动方式自动支持同机上进程间 GPU 资源的划分与模型切分。该功能在推理后端为 `lmdeploy` 或 `transformers` 时被支持，具体行为如下：
+
+- 基于 `python` 命令启动时，模型默认分配到所有可用的 GPU 上，如想指定使用哪些 GPU，可以使用 `CUDA_VISIBLE_DEVICES` 环境变量。
+- 基于 `torchrun` 命令启动时，每个模型实例会被分配到 `N_GPU // N_PROC` 个 GPU 上，`N_PROC` 为 torchrun 命令中的 `--nproc-per-node` 参数所指定的进程数。`N_GPU` 的取值为：
+    - 如 `CUDA_VISIBLE_DEVICES` 环境变量未设置，`N_GPU` 为全部可用 GPU 数量。
+    - 如 `CUDA_VISIBLE_DEVICES` 环境变量被设置，`N_GPU` 为 `CUDA_VISIBLE_DEVICES` 环境变量所指定的 GPU 数量，并且，仅有指定的 GPU 会被利用。
+
+下面提供了，在一台配备 8 块 GPU 的机器上运行评测任务的具体示例：
+```bash
+# <!-- 起两个模型实例数据并行，每个实例用 4 GPU -->
+torchrun --nproc-per-node=2 run.py --data MMBench_DEV_EN --model InternVL3-78B
+# <!-- 起一个模型实例，每个实例用 8 GPU -->
+python run.py --data MMBench_DEV_EN --model InternVL3-78B
+# <!-- 起三个模型实例，每个实例用 2 GPU，0 号、7 号 GPU 未被使用 -->
+CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 torchrun --nproc-per-node=3 run.py --data MMBench_DEV_EN --model InternVL3-38B
+```
+
+注：此方式不支持 `vllm` 后端，基于 `vllm` 后端起评测任务时，请用 `python` 命令启动，默认调用所有可见的 GPU。
+
+#### 性能差距
+在不同的运行环境中，模型的性能表现可能会有所差异。因此，在评估过程中，您可能会发现自己的评测结果与VLMEvalKit官方榜单上的结果存在差距。这种差异可能与`transformers`, `cuda`, `torch`等版本的变化有关。
+
+此外，对于异常的表现，我们建议您优先查看运行完成后的本地生成记录`{model}_{dataset}.xlsx`或者评估记录`{model}_{dataset}_{judge_model}.xlsx`，这可能会帮助您更好地理解评估结果并发现问题。
+
+
+
+### 部署本地语言模型作为评判 / 选择提取器
+上述默认设置使用 OpenAI 的 GPT 作为评判 LLM。你也可以使用 [LMDeploy](https://github.com/InternLM/lmdeploy) 部署本地评判 LLM。
+
+首先进行安装:
+```
+pip install lmdeploy openai
+```
+
+然后可以通过一行代码部署本地评判 LLM。LMDeploy 将自动从 Huggingface 下载模型。假设我们使用 internlm2-chat-1_8b 作为评判，端口为 23333，密钥为 sk-123456（密钥必须以 "sk-" 开头，后跟任意数字）：
+```
+lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
+```
+
+使用以下 Python 代码获取由 LMDeploy 注册的模型名称：
+```
+from openai import OpenAI
+client = OpenAI(
+    api_key='sk-123456',
+    base_url="http://0.0.0.0:23333/v1"
+)
+model_name = client.models.list().data[0].id
+```
+
+配置对应环境变量，以告诉 VLMEvalKit 如何使用本地评判 LLM。正如上面提到的，也可以在  `$VLMEvalKit/.env` 文件中设置：
+```
+OPENAI_API_KEY=sk-123456
+OPENAI_API_BASE=http://0.0.0.0:23333/v1/chat/completions
+LOCAL_LLM=<model_name you get>
+```
+
+最后，你可以运行第2步中的命令，使用本地评判 LLM 来评估你的 VLM。
+
+**请注意：**
+
+- 如果你希望将评判 LLM 部署在单独的一个 GPU 上，并且由于 GPU 内存有限而希望在其他 GPU 上评估你的 VLM，可以使用 `CUDA_VISIBLE_DEVICES=x` 这样的方法，例如：
+```
+CUDA_VISIBLE_DEVICES=0 lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
+CUDA_VISIBLE_DEVICES=1,2,3 torchrun --nproc-per-node=3 run.py --data HallusionBench  --model qwen_chat --verbose
+```
+- 如果本地评判 LLM 在遵循指令方面不够好，评估过程可能会失败。请通过 issues 报告此类失败情况。
+- 可以以不同的方式部署评判 LLM，例如使用私有 LLM（而非来自 HuggingFace）或使用量化 LLM。请参考 [LMDeploy doc](https://lmdeploy.readthedocs.io/en/latest/serving/api_server.html) 文档。也可以使用其他支持 OpenAI API 框架的方法。
+
+### 使用 LMDeploy 加速模型推理
+
+可参考[文档](/docs/zh-CN/EvalByLMDeploy.md)

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/README_zh-CN.md

@@ -0,0 +1,131 @@
+<div align="center">
+
+![LOGO](http://opencompass.openxlab.space/utils/MMLB.jpg)
+
+<b>VLMEvalKit: 一种多模态大模型评测工具 </b>
+
+[![][github-contributors-shield]][github-contributors-link] • [![][github-forks-shield]][github-forks-link] • [![][github-stars-shield]][github-stars-link] • [![][github-issues-shield]][github-issues-link] • [![][github-license-shield]][github-license-link]
+
+[English](/README.md) | 简体中文 | [日本語](/docs/ja/README_ja.md)
+
+<a href="https://rank.opencompass.org.cn/leaderboard-multimodal">🏆 OpenCompass 排行榜 </a> •
+<a href="#%EF%B8%8F-quickstart">🏗️ 快速开始 </a> •
+<a href="#-datasets-models-and-evaluation-results">📊 数据集和模型 </a> •
+<a href="#%EF%B8%8F-development-guide">🛠️ 开发指南 </a> •
+<a href="#-the-goal-of-vlmevalkit">🎯 我们的目标 </a> •
+<a href="#%EF%B8%8F-citation">🖊️ 引用 </a>
+
+<a href="https://huggingface.co/spaces/opencompass/open_vlm_leaderboard">🤗 HuggingFace 排行榜 (存档全部性能) </a> •
+<a href="https://huggingface.co/datasets/VLMEval/OpenVLMRecords">🤗 原始评测记录</a> •
+<a href="https://discord.gg/evDT4GZmxN">🔊 Discord</a> •
+<a href="https://www.arxiv.org/abs/2407.11691">📝 技术报告 </a>
+</div>
+
+**VLMEvalKit** (python 包名为 **vlmeval**) 是一款专为大型视觉语言模型 (Large Vision-Language Models， LVLMs) 评测而设计的开源工具包。该工具支持在各种基准测试上对大型视觉语言模型进行**一键评估**，无需进行繁重的数据准备工作，让评估过程更加简便。在 VLMEvalKit 中，我们对所有大型视觉语言模型生成的结果进行评测，并提供基于**精确匹配**与基于 **LLM 的答案提取**两种评测结果。
+
+## 🆕 更新
+
+- **[2025-04-29]** 优化 `torchrun` 启动逻辑：目前 `torchrun` 启动时，若进程数为 M，机器 GPU 卡数为 N，将会自动调整每个进程分配的 GPU 数量为 `N // M`。目前此分配方式适用于 `transformers`, `lmdeploy` 推理后端，`vllm` 推理后端仅支持使用 python 启动 🔥🔥🔥
+- **[2025-02-20]** 支持新模型：**InternVL2.5 series, QwenVL2.5 series, QVQ-72B, Doubao-VL, Janus-Pro-7B, MiniCPM-o-2.6, InternVL2-MPO, LLaVA-CoT, Hunyuan-Standard-Vision, Ovis2, Valley, SAIL-VL, Ross, Long-VITA, EMU3, SmolVLM**。支持新基准：**MMMU-Pro, WeMath, 3DSRBench, LogicVista, VL-RewardBench, CC-OCR, CG-Bench, CMMMU, WorldSense**。请参考[**VLMEvalKit Features**](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb)以获取更多信息。感谢社区的各位贡献者 🔥🔥🔥
+- **[2024-11-21]** 集成了一个新的配置系统，以实现更灵活的评估设置。查看[文档](/docs/zh-CN/ConfigSystem.md)或运行`python run.py --help`了解更多详情 🔥🔥🔥
+- **[2024-11-21]** 支持 **[QSpatial](https://andrewliao11.github.io/spatial_prompt/)**，一个用于定量空间推理的多模态基准（例如，确定大小/距离），感谢 **[andrewliao11](https://github.com/andrewliao11)** 提供官方支持 🔥🔥🔥
+- **[2024-11-21]** 支持 **[MM-Math](https://github.com/kge-sun/mm-math)**，一个包含约6K初中多模态推理数学问题的新多模态数学基准。GPT-4o-20240806在该基准上达到了22.5%的准确率 🔥🔥🔥
+- **[2024-11-16]** 支持 **[OlympiadBench](https://github.com/OpenBMB/OlympiadBench)**，一个多模态基准，包含奥林匹克级别的数学和物理问题 🔥🔥🔥
+- **[2024-11-16]** 支持 **[WildVision](https://huggingface.co/datasets/WildVision/wildvision-bench)**，一个基于多模态竞技场数据的主观多模态基准 🔥🔥🔥
+- **[2024-11-13]** 支持 **[MIA-Bench](https://arxiv.org/abs/2407.01509)**，一个多模态指令跟随基准 🔥🔥🔥
+- **[2024-11-08]** 支持 **[Aria](https://arxiv.org/abs/2410.05993)**，一个多模态原生 MoE 模型，感谢 **[teowu](https://github.com/teowu)** 🔥🔥🔥
+- **[2024-11-04]** 支持 **[WorldMedQA-V](https://www.arxiv.org/abs/2410.12722)**，该基准包含 1000 多个医学 VQA 问题，涵盖巴西、以色列、日本、西班牙等四个国家的语言，以及它们的英文翻译 🔥🔥🔥
+
+## 🏗️ 快速开始 <a id="quickstart"></a>
+
+请参阅[**快速开始**](/docs/zh-CN/Quickstart.md)获取入门指南。
+
+## 📊 评测结果，支持的数据集和模型 <a id="data-model-results"></a>
+
+### 评测结果
+
+**[OpenVLM Leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard)**: **[下载全部细粒度测试结果](http://opencompass.openxlab.space/assets/OpenVLM.json)**.
+
+请查看[**VLMEvalKit Features**](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb)中的 **Supported Benchmarks** 标签，以查看所有支持的图像和视频基准（70+）。
+
+请查看[**VLMEvalKit Features**](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb)中的 **Supported LMMs** 标签，以查看所���支持的 LMMs，包括商业 API、开源模型等（200+）。
+
+### 其他
+
+**Transformers 的版本推荐:**
+
+**请注意**，某些 VLM 可能无法在某些特定的 transformers 版本下运行，我们建议使用以下设置来评估对应的VLM:
+
+- **请用** `transformers==4.33.0` **来运行**: `Qwen series`, `Monkey series`, `InternLM-XComposer Series`, `mPLUG-Owl2`, `OpenFlamingo v2`, `IDEFICS series`, `VisualGLM`, `MMAlaya`, `ShareCaptioner`, `MiniGPT-4 series`, `InstructBLIP series`, `PandaGPT`, `VXVERSE`.
+- **请用** `transformers==4.37.0 ` **来运行**: `LLaVA series`, `ShareGPT4V series`, `TransCore-M`, `LLaVA (XTuner)`, `CogVLM Series`, `EMU2 Series`, `Yi-VL Series`, `MiniCPM-[V1/V2]`, `OmniLMM-12B`, `DeepSeek-VL series`, `InternVL series`, `Cambrian Series`, `VILA Series`, `Llama-3-MixSenseV1_1`, `Parrot-7B`, `PLLaVA Series`.
+- **请用** `transformers==4.40.0 ` **来运行**: `IDEFICS2`, `Bunny-Llama3`, `MiniCPM-Llama3-V2.5`, `360VL-70B`, `Phi-3-Vision`, `WeMM`.
+- **请用** `transformers==4.42.0 ` **来运行**: `AKI`.
+- **请用** `transformers==latest` **来运行**: `LLaVA-Next series`, `PaliGemma-3B`, `Chameleon series`, `Video-LLaVA-7B-HF`, `Ovis series`, `Mantis series`, `MiniCPM-V2.6`, `OmChat-v2.0-13B-sinlge-beta`, `Idefics-3`, `GLM-4v-9B`, `VideoChat2-HD`.
+
+**如何测试一个 VLM 是否可以正常运行:**
+
+```python
+from vlmeval.config import supported_VLM
+model = supported_VLM['idefics_9b_instruct']()
+# 前向单张图片
+ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
+print(ret)  # 这张图片上有一个带叶子的红苹果
+# 前向多张图片
+ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
+print(ret)  # 提供的图片中有两个苹果
+```
+
+## 🛠️ 开发指南 <a id="development"></a>
+
+要开发自定义评测数据集，支持其他 VLMs，或为 VLMEvalKit 贡献代码，请参阅[**开发指南**](/docs/zh-CN/Development_zh-CN.md)。
+
+为激励来自社区的共享并分享相应的 credit，在下一次 report 更新中，我们将：
+
+- 致谢所有的 contribution
+- 具备三个或以上主要贡献 (支持新模型、评测集、或是主要特性) 的贡献者将可以加入技术报告的作者列表 。合条件的贡献者可以创建 issue 或是在 [VLMEvalKit Discord Channel](https://discord.com/invite/evDT4GZmxN) 私信 kennyutc，我们将进行跟进
+
+## 🎯 VLMEvalKit 的目标 <a id="goal-of-vlmevalkit"></a>
+
+**该代码库的设计目标是：**
+
+1. 提供一个**易于使用**的**开源评估工具包**，方便研究人员和开发人员评测现有的多模态大模型，并使评测结果**易于复现**。
+2. 使 VLM 开发人员能够轻松地评测自己的模型。在多个支持的基准测试上评估 VLM，只需实现一个 `generate_inner()` 函数，所有其他工作负载（数据下载、数据预处理、预测推理、度量计算）都由代码库处理。
+
+**该代码库的设计目标不是:**
+
+复现所有**第三方基准测试**原始论文中报告的准确数字。有两个相关的原因:
+1. VLMEvalKit 对所有 VLMs 使用基于生成的评估（可选使用基于 LLM 的答案提取）。同时，一些基准测试可能官方使用不同的方法（*例如，SEEDBench 使用基于 PPL 的评估*）。对于这些基准测试，我们在相应的结果中比较两个得分。我们鼓励开发人员在代码库中支持其他评估范式。
+2. 默认情况下，我们对所有多模态模型使用相同的提示模板来评估基准测试。同时，**一些多模态模型可能有他们特定的提示模板**（目前可能未在代码库中涵盖）。我们鼓励 VLM 的开发人员在 VLMEvalKit 中实现自己的提示模板，如果目前未覆盖。这将有助于提高可复现性。
+
+## 🖊️ 引用 <a id="citation"></a>
+
+如果我们的工作对您有所帮助，请考虑 **star🌟** VLMEvalKit。感谢支持！
+
+[![Stargazers repo roster for @open-compass/VLMEvalKit](https://reporoster.com/stars/open-compass/VLMEvalKit)](https://github.com/open-compass/VLMEvalKit/stargazers)
+
+如果您在研究中使用了 VLMEvalKit，或希望参考已发布的开源评估结果，请使用以下 BibTeX 条目以及与您使用的特定 VLM / 基准测试相对应的 BibTex 条目。
+
+```bib
+@misc{duan2024vlmevalkit,
+      title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models},
+      author={Haodong Duan and Junming Yang and Yuxuan Qiao and Xinyu Fang and Lin Chen and Yuan Liu and Xiaoyi Dong and Yuhang Zang and Pan Zhang and Jiaqi Wang and Dahua Lin and Kai Chen},
+      year={2024},
+      eprint={2407.11691},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2407.11691},
+}
+```
+
+<p align="right"><a href="#top">🔝回到顶部</a></p>
+
+[github-contributors-link]: https://github.com/open-compass/VLMEvalKit/graphs/contributors
+[github-contributors-shield]: https://img.shields.io/github/contributors/open-compass/VLMEvalKit?color=c4f042&labelColor=black&style=flat-square
+[github-forks-link]: https://github.com/open-compass/VLMEvalKit/network/members
+[github-forks-shield]: https://img.shields.io/github/forks/open-compass/VLMEvalKit?color=8ae8ff&labelColor=black&style=flat-square
+[github-issues-link]: https://github.com/open-compass/VLMEvalKit/issues
+[github-issues-shield]: https://img.shields.io/github/issues/open-compass/VLMEvalKit?color=ff80eb&labelColor=black&style=flat-square
+[github-license-link]: https://github.com/open-compass/VLMEvalKit/blob/main/LICENSE
+[github-license-shield]: https://img.shields.io/github/license/open-compass/VLMEvalKit?color=white&labelColor=black&style=flat-square
+[github-stars-link]: https://github.com/open-compass/VLMEvalKit/stargazers
+[github-stars-shield]: https://img.shields.io/github/stars/open-compass/VLMEvalKit?color=ffcb47&labelColor=black&style=flat-square

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/_static/css/readthedocs.css

@@ -0,0 +1,63 @@
+.header-logo {
+  background-image: url("../image/logo.svg");
+  background-size: 275px 80px;
+  height: 80px;
+  width: 275px;
+}
+
+
+@media screen and (min-width: 1100px) {
+  .header-logo {
+    top: -25px;
+  }
+}
+
+pre {
+    white-space: pre;
+}
+
+@media screen and (min-width: 2000px) {
+  .pytorch-content-left {
+    width: 1200px;
+    margin-left: 30px;
+  }
+  article.pytorch-article {
+    max-width: 1200px;
+  }
+  .pytorch-breadcrumbs-wrapper {
+    width: 1200px;
+  }
+  .pytorch-right-menu.scrolling-fixed {
+    position: fixed;
+    top: 45px;
+    left: 1580px;
+  }
+}
+
+
+article.pytorch-article section code {
+  padding: .2em .4em;
+  background-color: #f3f4f7;
+  border-radius: 5px;
+}
+
+/* Disable the change in tables */
+article.pytorch-article section table code {
+  padding: unset;
+  background-color: unset;
+  border-radius: unset;
+}
+
+table.autosummary td {
+  width: 50%
+}
+
+img.align-center {
+  display: block;
+  margin-left: auto;
+  margin-right: auto;
+}
+
+article.pytorch-article p.rubric {
+  font-weight: bold;
+}

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/_static/js/custom.js

@@ -0,0 +1,10 @@
+var collapsedSections = [];
+
+$(document).ready(function () {
+  $('.model-summary').DataTable({
+    "stateSave": false,
+    "lengthChange": false,
+    "pageLength": 20,
+    "order": []
+  });
+});

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/_templates/404.html

@@ -0,0 +1,18 @@
+{% extends "layout.html" %}
+
+{% block body %}
+
+<h1>Page Not Found</h1>
+<p>
+  The page you are looking for cannot be found.
+</p>
+<p>
+  If you just switched documentation versions, it is likely that the page you were on is moved. You can look for it in
+  the content table left, or go to <a href="{{ pathto(root_doc) }}">the homepage</a>.
+</p>
+<!-- <p>
+  If you cannot find documentation you want, please <a
+    href="">open an issue</a> to tell us!
+</p> -->
+
+{% endblock %}

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/_templates/autosummary/class.rst

@@ -0,0 +1,13 @@
+.. role:: hidden
+    :class: hidden-section
+.. currentmodule:: {{ module }}
+
+
+{{ name | underline}}
+
+.. autoclass:: {{ name }}
+    :members:
+
+..
+  autogenerated from _templates/autosummary/class.rst
+  note it does not have :inherited-members:

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/_templates/callable.rst

@@ -0,0 +1,14 @@
+.. role:: hidden
+    :class: hidden-section
+.. currentmodule:: {{ module }}
+
+
+{{ name | underline}}
+
+.. autoclass:: {{ name }}
+    :members:
+    :special-members: __call__
+
+..
+  autogenerated from _templates/callable.rst
+  note it does not have :inherited-members:

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/conf.py

@@ -0,0 +1,242 @@
+# flake8: noqa
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+
+import ast
+import os
+import subprocess
+import sys
+
+import pytorch_sphinx_theme
+from sphinx.builders.html import StandaloneHTMLBuilder
+
+sys.path.insert(0, os.path.abspath('../../'))
+
+# -- Project information -----------------------------------------------------
+
+project = 'VLMEvalKit'
+copyright = '2023, VLMEvalKit'
+author = 'VLMEvalKit Authors'
+
+# The full version, including alpha/beta/rc tags
+version_file = '../../vlmeval/__init__.py'
+
+
+def get_version():
+    with open(version_file, 'r') as f:
+        file_content = f.read()
+    # Parse the file content into an abstract syntax tree (AST)
+    tree = ast.parse(file_content, filename=version_file)
+
+    # Iterate through the body of the AST, looking for an assignment to __version__
+    for node in tree.body:
+        if isinstance(node, ast.Assign):
+            for target in node.targets:
+                if isinstance(target, ast.Name) and target.id == '__version__':
+                    return node.value.s
+    raise ValueError('__version__ not found')
+
+
+release = get_version()
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.viewcode',
+    'myst_parser',
+    'sphinx_copybutton',
+    'sphinx_tabs.tabs',
+    'notfound.extension',
+    'sphinxcontrib.jquery',
+    'sphinx_design',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+source_suffix = {
+    '.rst': 'restructuredtext',
+    '.md': 'markdown',
+}
+
+language = 'cn'
+
+# The master toctree document.
+root_doc = 'index'
+html_context = {
+    'github_version': 'latest',
+}
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'pytorch_sphinx_theme'
+html_theme_path = [pytorch_sphinx_theme.get_html_theme_path()]
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+# yapf: disable
+html_theme_options = {
+    'menu': [
+        {
+            'name': 'GitHub',
+            'url': 'https://github.com/open-compass/VLMEvalKit'
+        },
+    ],
+    # Specify the language of shared menu
+    'menu_lang': 'cn',
+    # Disable the default edit on GitHub
+    'default_edit_on_github': False,
+}
+# yapf: enable
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+html_css_files = [
+    'https://cdn.datatables.net/v/bs4/dt-1.12.1/datatables.min.css',
+    'css/readthedocs.css'
+]
+html_js_files = [
+    'https://cdn.datatables.net/v/bs4/dt-1.12.1/datatables.min.js',
+    'js/custom.js'
+]
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'vlmevalkitdoc'
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (root_doc, 'vlmevalkit.tex', 'VLMEvalKit Documentation', author,
+     'manual'),
+]
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [(root_doc, 'vlmevalkit', 'VLMEvalKit Documentation', [author],
+              1)]
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (root_doc, 'vlmevalkit', 'VLMEvalKit Documentation', author,
+     'VLMEvalKit Authors', 'AGI evaluation toolbox and benchmark.',
+     'Miscellaneous'),
+]
+
+# -- Options for Epub output -------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = project
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#
+# epub_identifier = ''
+
+# A unique identification for the text.
+#
+# epub_uid = ''
+
+# A list of files that should not be packed into the epub file.
+epub_exclude_files = ['search.html']
+
+# set priority when building html
+StandaloneHTMLBuilder.supported_image_types = [
+    'image/svg+xml', 'image/gif', 'image/png', 'image/jpeg'
+]
+
+# -- Extension configuration -------------------------------------------------
+# Ignore >>> when copying code
+copybutton_prompt_text = r'>>> |\.\.\. '
+copybutton_prompt_is_regexp = True
+
+# Auto-generated header anchors
+myst_heading_anchors = 3
+# Enable "colon_fence" extension of myst.
+myst_enable_extensions = ['colon_fence', 'dollarmath']
+
+# Configuration for intersphinx
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/3', None),
+    'numpy': ('https://numpy.org/doc/stable', None),
+    'torch': ('https://pytorch.org/docs/stable/', None),
+    'mmengine': ('https://mmengine.readthedocs.io/en/latest/', None),
+    'transformers':
+    ('https://huggingface.co/docs/transformers/main/en/', None),
+}
+napoleon_custom_sections = [
+    # Custom sections for data elements.
+    ('Meta fields', 'params_style'),
+    ('Data fields', 'params_style'),
+]
+
+# Disable docstring inheritance
+autodoc_inherit_docstrings = False
+# Mock some imports during generate API docs.
+autodoc_mock_imports = ['rich', 'attr', 'einops']
+# Disable displaying type annotations, these can be very verbose
+autodoc_typehints = 'none'
+
+# The not found page
+notfound_template = '404.html'
+
+
+def builder_inited_handler(app):
+    subprocess.run(['./cp_origin_docs.sh'])
+
+
+def setup(app):
+    app.connect('builder-inited', builder_inited_handler)

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/cp_origin_docs.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+# Copy *.md files from docs/ if it doesn't have a Chinese translation
+
+for filename in $(find ../en/ -name '*.md' -printf "%P\n");
+do
+    mkdir -p $(dirname $filename)
+    cp -n ../en/$filename ./$filename
+done

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/docutils.conf

@@ -0,0 +1,2 @@
+[html writers]
+table_style: colwidths-auto

reference/code/Fast-dLLM/third_party/VLMEvalKit/docs/zh-CN/index.rst

@@ -0,0 +1,49 @@
+欢迎来到 VLMEvalKit 中文教程！
+==========================================
+
+VLMEvalKit 上手路线
+-------------------------------
+
+为了用户能够快速上手，我们推荐以下流程：
+
+- 对于想要使用 VLMEvalKit 的用户，我们推荐先阅读 开始你的第一步_ 部分来设置环境，并启动一个迷你实验熟悉流程。
+
+- 若您想进行更多模块的自定义，例如增加数据集和模型，我们提供了 进阶教程_ 。
+
+我们始终非常欢迎用户的 PRs 和 Issues 来完善 VLMEvalKit！
+
+.. _快速开始:
+.. toctree::
+   :maxdepth: 1
+   :caption: 快速开始
+
+   Quickstart.md
+
+
+.. .. _教程:
+.. .. toctree::
+..    :maxdepth: 1
+..    :caption: 教程
+
+..    user_guides/framework_overview.md
+
+.. _进阶教程:
+.. toctree::
+   :maxdepth: 1
+   :caption: 进阶教程
+
+   Development.md
+   ConfigSystem.md
+
+.. .. _其他说明:
+.. .. toctree::
+..    :maxdepth: 1
+..    :caption: 其他说明
+
+..    notes/contribution_guide.md
+
+索引与表格
+==================
+
+* :ref:`genindex`
+* :ref:`search`

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/CGAVCounting/cg_av_counting.py

@@ -0,0 +1,415 @@
+import json
+import os
+import os.path as osp
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import portalocker
+from huggingface_hub import snapshot_download
+from PIL import Image
+
+from vlmeval.smp import (dump, get_cache_path, get_file_extension, get_intermediate_file_path,
+                         load, md5, modelscope_flag_set)
+from ..utils.cgbench import post_process, unzip_hf_zip
+from ..video_base import VideoBaseDataset
+from .utils import get_timestampes, rating_func
+
+
+class CGAVCounting(VideoBaseDataset):
+
+    dataset = "CG-AV-Counting"
+
+    TYPE = "Video-Counting"
+
+    MD5 = "d1cd8486353ab85178098d443264a7d0"
+
+    SYS = ""
+
+    def __init__(
+        self,
+        dataset="CG-AV-Counting",
+        use_frame_time=False,
+        nframe=0,
+        fps=-1,
+    ):
+        super().__init__(dataset=dataset, nframe=nframe, fps=fps)
+        self.use_frame_time = use_frame_time
+        self.dataset_name = dataset
+        self.frame_tmpl_clue = 'frame-{}.jpg'
+
+    @classmethod
+    def supported_datasets(cls):
+        return ["CGAVCounting"]
+
+    def frame_paths_clue(self, video, timestamp_list):
+        frame_root = osp.join(self.frame_root, video)
+        os.makedirs(frame_root, exist_ok=True)
+        return [osp.join(frame_root, self.frame_tmpl_clue.format(i)) for i in timestamp_list]
+
+    def save_video_frames_clue(self, video, uid, timestamp_list):
+        if type(uid) is not str:
+            uid = str(uid)
+        import decord
+        frame_paths = self.frame_paths_clue(uid, timestamp_list)
+        flag = np.all([osp.exists(p) for p in frame_paths])
+        if flag:
+            frame = Image.open(frame_paths[0])
+            return frame_paths, frame.width, frame.height
+        vid_path = osp.join(self.data_root, video)
+        vid = decord.VideoReader(vid_path)
+        frames = []
+        # 获取视频的帧率
+        fps = vid.get_avg_fps()
+        lock_path = osp.splitext(vid_path)[0] + '.lock'
+        with portalocker.Lock(lock_path, 'w', timeout=30):
+            for timestamp_sec in timestamp_list:
+                # 计算视频帧对应的索引
+                frame_idx = int(timestamp_sec * fps)
+
+                # 获取对应帧
+                frame = vid[frame_idx]
+
+                # 将帧转换为PIL图像
+                img = Image.fromarray(frame.asnumpy())
+                frames.append(img)
+            for im, pth in zip(frames, frame_paths):
+                if not osp.exists(pth):
+                    im.save(pth)
+        return frame_paths, frames[0].width, frames[0].height
+
+    def format_time(self, t):
+        return f"{t:.2f}"
+
+    def get_output_filename(self, item):
+        video_id = Path(item["video"]).stem
+        start_str = self.format_time(item["query_interval"][0])
+        end_str = self.format_time(item["query_interval"][1])
+        return f"{video_id}_{start_str}_{end_str}.mp4"
+
+    def prepare_dataset(self, dataset_name="CG-AV-Counting", repo_id="CG-Bench/CG-AV-Counting"):
+
+        def check_integrity(pth):
+            data_file = osp.join(pth, f"{dataset_name}.tsv")
+
+            if not os.path.exists(data_file):
+                return False
+
+            if md5(data_file) != self.MD5:
+                return False
+            data = load(data_file)
+            for video_pth in data["video"]:
+                if not osp.exists(osp.join(pth, video_pth)):
+                    return False
+            return True
+
+        cache_path = get_cache_path(repo_id)
+
+        if cache_path is not None and check_integrity(cache_path):
+            dataset_path = cache_path
+        else:
+
+            def generate_tsv(pth):
+
+                tsv_file = osp.join(pth, f"{dataset_name}.tsv")
+
+                task_modes = ["long_acc", "ref_acc", "clue_acc"]
+                all_data = []
+                for task_mode in task_modes:
+                    with open(osp.join(pth, "cg-av-counting.json"), "r") as f:
+                        data_file = pd.DataFrame(json.load(f))
+
+                    data_file = data_file.assign(index=range(len(data_file)))
+                    data_file["video_uid"] = data_file["video"].replace(".mp4", "")
+                    data_file["video"] = data_file["video"].apply(lambda x: f"cg_videos_720p/{x}")
+
+                    data_file["ref_video_path"] = ""
+                    data_file["ref_video_uid"] = ""
+
+                    if task_mode in ["ref_acc"]:
+                        data_file["ref_video_path"] = data_file.apply(
+                            lambda row: f"ref_videos/{self.get_output_filename(row)}", axis=1
+                        )
+                        data_file["ref_video_uid"] = data_file["ref_video_path"].apply(
+                            lambda x: x.split("/")[-1].replace(".mp4", ""))
+
+                    data_file["task_mode"] = task_mode
+
+                    if task_mode == "clue_acc":
+                        data_file["answer"] = data_file["clue"].apply(json.dumps)
+
+                    data_file = data_file[
+                        [
+                            "index",
+                            "video_uid",
+                            "video",
+                            "ref_video_path",
+                            "ref_video_uid",
+                            "question",
+                            "answer",
+                            "type",
+                            "category",
+                            "task_mode"
+                        ]
+                    ]
+
+                    all_data.append(data_file)
+
+                final_data = pd.concat(all_data, ignore_index=True)
+                final_data["index"] = range(len(final_data))
+                final_data.to_csv(tsv_file, sep="\t", index=False)
+            dataset_path = cache_path
+
+            if modelscope_flag_set():
+                from modelscope import dataset_snapshot_download
+
+                dataset_path = dataset_snapshot_download(dataset_id=repo_id)
+            else:
+                dataset_path = snapshot_download(repo_id=repo_id, repo_type="dataset")
+
+            unzip_hf_zip(dataset_path)
+
+            generate_tsv(dataset_path)
+
+        tsv_file = osp.join(dataset_path, f"{dataset_name}.tsv")
+
+        return dict(data_file=tsv_file, root=dataset_path)
+
+    def build_prompt(self, line, video_llm):
+        if isinstance(line, int):
+            assert line < len(self)
+            line = self.data.iloc[line]
+        task_mode = line["task_mode"]
+        assert task_mode in ["long_acc", "clue_acc", "ref_acc"]
+        if task_mode == "long_acc":
+            user_prompt = ""
+            message = []
+            video_path = line["video"]
+            if video_llm:
+                message.append(dict(type="video", value=osp.join(self.data_root, video_path)))
+            else:
+                image_paths, frame_indices, vid_fps = self.save_video_frames(
+                    video_path, uid=line["video_uid"], num_frames=self.nframe, fps=self.fps
+                )
+                message.extend(dict(type="image", value=im) for im in image_paths)
+
+                if self.use_frame_time:
+                    user_prompt += get_timestampes(frame_indices, vid_fps)
+
+            user_prompt += (
+                f"Please answer the question '{line['question']}' with a number. Just output the number itself, "
+                "don't output anything else."
+            )
+            message.append(dict(type="text", value=user_prompt))
+        elif task_mode == "ref_acc":
+            user_prompt = ""
+            message = []
+            video_path = line["ref_video_path"]
+            if video_llm:
+                message.append(dict(type="video", value=osp.join(self.data_root, video_path)))
+            else:
+                image_paths, frame_indices, vid_fps = self.save_video_frames(
+                    video_path, uid=line["ref_video_uid"], num_frames=self.nframe, fps=self.fps
+                )
+                message.extend(dict(type="image", value=im) for im in image_paths)
+
+                if self.use_frame_time:
+                    user_prompt += get_timestampes(frame_indices, vid_fps)
+            user_prompt += (
+                f"Please answer the question '{line['question']}' with a number. Just output the number itself, "
+                "don't output anything else."
+            )
+            message.append(dict(type="text", value=user_prompt))
+        elif task_mode == "clue_acc":
+            if line["category"] == "event":
+                user_prompt = ""
+                message = []
+                video_path = line["video"]
+                if video_llm:
+                    message.append(dict(type="video", value=osp.join(self.data_root, video_path)))
+                else:
+                    image_paths, frame_indices, vid_fps = self.save_video_frames(
+                        video_path, uid=line["video_uid"], num_frames=self.nframe, fps=self.fps
+                    )
+                    message.extend(dict(type="image", value=im) for im in image_paths)
+                    user_prompt += get_timestampes(frame_indices, vid_fps)
+
+                user_prompt += (
+                    f"Watch the video and provide your answer to the question '{line['question']}', "
+                    "including the start and end timestamps for each event."
+                    "Format your answer in JSON, enclosed in <answer> and </answer> tags. "
+                    "The output should look like this: <answer>[[\"start_time\", \"end_time\"], ...]</answer>. "
+                    "Ensure each timestamp is in seconds (e.g., 'xx.xx')."
+                )
+                message.append(dict(type="text", value=user_prompt))
+            elif line["category"] == "object":
+                user_prompt = ""
+                message = []
+                video_path = line["video"]
+                clue_timestamp_list = []
+                for clue in json.loads(line["answer"]):
+                    if clue["timestamp"] not in clue_timestamp_list:
+                        clue_timestamp_list.append(clue["timestamp"])
+                image_paths, width, height = self.save_video_frames_clue(
+                    video_path, uid=line["video_uid"], timestamp_list=clue_timestamp_list
+                )
+                message.append(
+                    dict(type="text", value=f"There are {len(image_paths)} frames in the size of {width}x{height}"))
+                for idx, im in enumerate(image_paths):
+                    message.append(dict(type="text", value=f"Frame{idx + 1}:"))
+                    message.append(dict(type="image", value=im))
+                user_prompt += (
+                    f"Answer the question '{line['question']}', "
+                    "including the bounding box for the query object in the first frame "
+                    "where it appears. For subsequent frames where the object appears, "
+                    "do not provide the bounding box again. "
+                    "Format your answer in JSON, enclosed within <answer> and </answer> tags. "
+                    "The output should look like this: "
+                    "<answer>{\"Frame1\": [[x_min, y_min, x_max, y_max]], \"Frame2\": [...],...}</answer>. "
+                    "In the output, each frame should either contain the bounding box of the object "
+                    "(if it appears for the first time in that frame) or an empty list `[]` "
+                    "(if the object does not appear or it has already been labeled in a previous frame). "
+                    "Ensure that bounding boxes are listed as [x_min, y_min, x_max, y_max]."
+                )
+                message.append(dict(type="text", value=user_prompt))
+            elif line["category"] == "attribute":
+                user_prompt = ""
+                message = []
+                video_path = line["video"]
+                clue_timestamp_list = []
+                for clue_ in json.loads(line["answer"]):
+                    for clue in clue_:
+                        if clue["timestamp"] not in clue_timestamp_list:
+                            clue_timestamp_list.append(clue["timestamp"])
+                image_paths, width, height = self.save_video_frames_clue(
+                    video_path, uid=line["video_uid"], timestamp_list=clue_timestamp_list
+                )
+                message.append(dict(
+                    type="text",
+                    value=f"There are {len(image_paths)} frames in the size of {width}x{height}"))
+                for idx, im in enumerate(image_paths):
+                    message.append(dict(type="text", value=f"Frame{idx + 1}:"))
+                    message.append(dict(type="image", value=im))
+                user_prompt += (
+                    f"Answer the question '{line['question']}', clustering the objects according to the question. "
+                    "For each unique cluster, assign a unique label and return the bounding box for each object in "
+                    "the first frame where it appears. For subsequent frames where the object appears, "
+                    "do not output anything. "
+                    "Format your answer in JSON, enclosed within <answer> and </answer> tags. "
+                    "The output should look like this: "
+                    "<answer>{\"Frame 1\": [{\"bbox\": [x_min, y_min, x_max, y_max], 'label': \"Label 1\"}], "
+                    "\"Frame 2\": [...], ...}</answer>. "
+                    "In the output, each frame should either contain the bounding box and label for the object "
+                    "(if it appears for the first time in that frame) or an empty list `[]` "
+                    "(if the object has already been labeled or does not appear in that frame). "
+                    "The label should correspond to a unique object cluster according to the question."
+                )
+                message.append(dict(type="text", value=user_prompt))
+        print(message)
+        return message
+
+    def save_video_frames(self, video, uid, num_frames=8, fps=-1):
+
+        if type(uid) is not str:
+            uid = str(uid)
+        import decord
+        vid_path = osp.join(self.data_root, video)
+        vid = decord.VideoReader(vid_path)
+        vid_fps = vid.get_avg_fps()
+        n_frames = len(vid)
+
+        if num_frames > 0 and fps < 0:
+            step_size = len(vid) / (num_frames + 1)
+            indices = [int(i * step_size) for i in range(1, num_frames + 1)]
+
+            frame_paths = self.frame_paths(uid)
+        elif fps > 0:
+            total_duration = n_frames / vid_fps
+            required_frames = int(total_duration * fps)
+            step_size = vid_fps / fps
+            indices = [int(i * step_size) for i in range(required_frames)]
+            frame_paths = self.frame_paths_fps(uid, len(indices))
+
+        # Save and validate frames
+        valid_paths = []
+        valid_indices = []
+        lock_path = osp.splitext(vid_path)[0] + '.lock'
+        with portalocker.Lock(lock_path, 'w', timeout=30):
+            if not np.all([osp.exists(p) for p in frame_paths]):
+                images = [vid[i].asnumpy() for i in indices]
+                for i, (img_array, path) in enumerate(zip(images, frame_paths)):
+                    if osp.exists(path):
+                        try:
+                            with Image.open(path) as img:
+                                img.verify()
+                            valid_paths.append(path)
+                            valid_indices.append(indices[i])
+                        except Exception:
+                            continue
+                    else:
+                        try:
+                            img = Image.fromarray(img_array)
+                            img.save(path)
+                            img.verify()
+                            valid_paths.append(path)
+                            valid_indices.append(indices[i])
+                        except Exception:
+                            continue
+            else:
+                for i, path in enumerate(frame_paths):
+                    try:
+                        with Image.open(path) as img:
+                            img.verify()
+                        valid_paths.append(path)
+                        valid_indices.append(indices[i])
+                    except Exception:
+                        continue
+
+        return valid_paths, valid_indices, vid_fps
+
+    def evaluate(self, eval_file, **judge_kwargs):
+
+        assert get_file_extension(eval_file) in ['xlsx', 'json', 'tsv'], \
+            'data file should be an supported format (xlsx/json/tsv) file'
+
+        tgt_file = get_intermediate_file_path(eval_file, '_rating', 'json')
+        score_file = get_intermediate_file_path(eval_file, '_score', 'csv')
+
+        data = load(eval_file)
+
+        data_un = data[~pd.isna(data["prediction"])]
+        data_pred_na = data[pd.isna(data["prediction"])]
+
+        data_pred_na["score"] = -1
+
+        scores_df = data_un.apply(
+            lambda row: post_process(
+                response=row["prediction"],
+                right_answer=row["answer"],
+                task_mode=row["task_mode"],
+                category=row["category"]
+            ),
+            axis=1,
+            result_type='expand'
+        )
+
+        data_un = pd.concat([data_un, scores_df], axis=1)
+
+        data = pd.concat([data_pred_na, data_un])
+
+        rejected_count = (data["score"] == -1).sum()
+
+        print(
+            f"Among {len(data)} questions, "
+            f"failed to obtain prediction for {len(data_pred_na)} questions, "
+            f"failed to obtain the score for {rejected_count - len(data_pred_na)} questions. "
+            f"Those questions will be counted as -1 score in ALL rating, and will not be counted in VALID rating."
+        )
+
+        dump(data, score_file)
+
+        rating = rating_func(score_file)
+
+        dump(rating, tgt_file)
+
+        return rating

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/CGAVCounting/requirements.txt

@@ -0,0 +1,2 @@
+scipy
+word2number

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/CGAVCounting/utils.py

@@ -0,0 +1,423 @@
+import json
+import math
+import re
+import signal
+import zipfile
+from pathlib import Path
+
+import numpy as np
+from tqdm import tqdm
+
+from vlmeval.smp import load
+
+
+def rating_func(data_path):
+    df = load(data_path)
+
+    task_mode_fields = {
+        "long_acc": ["acc", "oboa", "mae", "rmse"],
+        "ref_acc": ["acc", "oboa", "mae", "rmse"],
+        "clue_acc": ["wcs", "ifa"],
+    }
+
+    rating = {}
+
+    for task_mode, fields in task_mode_fields.items():
+        sub_df = df[df["task_mode"] == task_mode]
+        for field in fields:
+            values = sub_df[field]
+            if field == "rmse":
+                # RMSE: sqrt(mean(x^2))
+                rmse_val = np.sqrt(values.mean())
+                rating[f"{task_mode}/rmse"] = round(rmse_val, 4)
+            else:
+                rating[f"{task_mode}/{field}"] = round(values.mean(), 4)
+
+    return rating
+
+
+def get_timestampes(frame_indices, fps):
+    seconds = list(map(lambda x: str(round(x / fps, 4)), frame_indices))
+    timestamps = ", ".join(seconds)
+    return "A total of {frame_num} frames are sampled. Their corresponding timestamps are:\n\n{timestamps}\n\n".format(
+        frame_num=len(frame_indices), timestamps=timestamps
+    )
+
+
+def time_str_to_seconds(time_str: str) -> float:
+    time_str = time_str.strip()
+    if '.' in time_str:
+        time_main, milliseconds = time_str.split('.')
+        milliseconds = float(f"0.{milliseconds}")
+    else:
+        time_main = time_str
+        milliseconds = 0.0
+
+    parts = list(map(int, time_main.split(":")))
+
+    if len(parts) == 2:
+        minutes, seconds = parts
+        total_seconds = minutes * 60 + seconds
+    elif len(parts) == 3:
+        hours, minutes, seconds = parts
+        total_seconds = hours * 3600 + minutes * 60 + seconds
+    else:
+        raise ValueError(f"Invalid time format: {time_str}")
+
+    return total_seconds + milliseconds
+
+
+def extract_outer_json(text):
+    stack = []
+    start_idx = None
+    opening = {'{': '}', '[': ']'}
+    closing = {'}': '{', ']': '['}
+
+    for i, char in enumerate(text):
+        if char in opening:
+            if not stack:
+                start_idx = i  # 最外层起点
+            stack.append(char)
+        elif char in closing:
+            if stack and stack[-1] == closing[char]:
+                stack.pop()
+                if not stack and start_idx is not None:
+                    candidate = text[start_idx:i + 1]
+                    try:
+                        return json.dumps(json.loads(candidate))
+                    except json.JSONDecodeError:
+                        continue  # 尝试下一个 JSON 块
+    return None
+
+
+def compute_tiou(t1, t2):
+    """Temporal IoU"""
+    inter_start = max(t1[0], t2[0])
+    inter_end = min(t1[1], t2[1])
+    inter = max(0.0, inter_end - inter_start)
+    union = max(t1[1], t2[1]) - min(t1[0], t2[0])
+    return inter / union if union > 0 else 0.0
+
+
+def compute_sIoU(box1, box2):
+    """
+    Complete IoU (sIoU) between two bounding boxes.
+    Args:
+        box1 (list or np.array): [x1, y1, x2, y2] of ground truth box
+        box2 (list or np.array): [x1, y1, x2, y2] of predicted box
+
+    Returns:
+        IoU (float): The IoU score between the two boxes.
+    """
+
+    # Ensure the coordinates are ordered: [min_x, min_y, max_x, max_y]
+    box1 = np.array([min(box1[0], box1[2]), min(box1[1], box1[3]),
+                     max(box1[0], box1[2]), max(box1[1], box1[3])])
+    box2 = np.array([min(box2[0], box2[2]), min(box2[1], box2[3]),
+                     max(box2[0], box2[2]), max(box2[1], box2[3])])
+
+    # Compute the intersection area
+    inter_x1 = max(box1[0], box2[0])
+    inter_y1 = max(box1[1], box2[1])
+    inter_x2 = min(box1[2], box2[2])
+    inter_y2 = min(box1[3], box2[3])
+
+    inter_area = max(0, inter_x2 - inter_x1) * max(0, inter_y2 - inter_y1)
+
+    # Compute areas of the individual boxes
+    area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
+    area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
+
+    # Compute union area
+    union = area1 + area2 - inter_area
+    iou = inter_area / union if union > 0 else 0.0
+
+    return iou
+
+
+def greedy_matching(gt_instances, pred_instances, iou_func):
+    """Greedy matching based on maximum IoU"""
+    unmatched_gt = set(range(len(gt_instances)))
+    unmatched_pred = set(range(len(pred_instances)))
+    matches = []
+
+    while unmatched_gt and unmatched_pred:
+        max_iou = -1
+        best_match = None
+        for gt_idx in unmatched_gt:
+            for pred_idx in unmatched_pred:
+                iou = iou_func(gt_instances[gt_idx], pred_instances[pred_idx])
+                if iou > max_iou:
+                    max_iou = iou
+                    best_match = (gt_idx, pred_idx)
+
+        if best_match:
+            gt_idx, pred_idx = best_match
+            matches.append((gt_idx, pred_idx))
+            unmatched_gt.remove(gt_idx)
+            unmatched_pred.remove(pred_idx)
+
+    return matches
+
+
+def compute_cluster_pair_wcs(gt, pred, iou_type):
+    if iou_type == 'tIoU':
+        loc_sum = 0.0
+        for g in gt:
+            loc_sum += max([compute_tiou(g, p) for p in pred] or [0.0])
+        loc_acc = loc_sum / len(gt) if gt else 0.0
+        count_penalty = 1.0 - abs(len(pred) - len(gt)) / max(len(gt), 1)
+        # count_penalty = 1.0
+        return math.sqrt(loc_acc * max(0, count_penalty))
+
+    elif iou_type == 'sIoU':
+        # group by frame index
+        from collections import defaultdict
+        gt_by_f = defaultdict(list)
+        pred_by_f = defaultdict(list)
+        for f, box in gt:
+            gt_by_f[f].append(box)
+        for f, box in pred:
+            pred_by_f[f].append(box)
+
+        all_f = set(gt_by_f) | set(pred_by_f)
+        wcs = 0.0
+        for f in all_f:
+            gt_f = gt_by_f.get(f, [])
+            pred_f = pred_by_f.get(f, [])
+            matches = greedy_matching(gt_f, pred_f, compute_sIoU)
+            loc_sum = sum([compute_sIoU(gt_f[i], pred_f[j]) for i, j in matches])
+            loc_acc = loc_sum / len(gt_f) if gt_f else 0.0
+            count_penalty = 1.0 - abs(len(pred_f) - len(gt_f)) / max(len(gt_f), 1)
+            # count_penalty = 1.0
+            wcs += math.sqrt(loc_acc * max(0, count_penalty))
+        return wcs / max(len(all_f), 1)
+
+    else:
+        raise ValueError("Unsupported iou_type")
+
+
+class TimeoutException(Exception):
+    pass
+
+
+def timeout_handler(signum, frame):
+    raise TimeoutException("Function execution exceeded the time limit.")
+
+
+def compute_wcs_unlabeled(gt_clusters, pred_clusters, iou_type='tIoU',
+                          timeout=10):  # 主要是给attribute用的，但是object和event视作一个cluster也能用
+    from scipy.optimize import linear_sum_assignment
+
+    # Set the timeout signal handler
+    signal.signal(signal.SIGALRM, timeout_handler)
+    signal.alarm(timeout)  # Set the alarm to go off in 'timeout' seconds
+
+    try:
+        # Original function logic
+        K = len(gt_clusters)
+        M = len(pred_clusters)
+
+        # Build cost matrix (we want max score → min cost)
+        score_matrix = np.zeros((K, M))
+        for i in range(K):
+            for j in range(M):
+                score_matrix[i, j] = compute_cluster_pair_wcs(gt_clusters[i], pred_clusters[j], iou_type)
+
+        cost_matrix = -score_matrix  # maximize score → minimize cost
+
+        row_ind, col_ind = linear_sum_assignment(cost_matrix)
+
+        matched_scores = [score_matrix[i, j] for i, j in zip(row_ind, col_ind)]
+
+        # WCS = average over gt clusters (including unmatched = 0)
+        total_wcs = sum(matched_scores)
+        return total_wcs / K
+
+    except TimeoutException:
+        print(gt_clusters, pred_clusters)
+        print("Function execution exceeded the time limit.")
+        return None  # or you can return some default value to indicate timeout
+
+    finally:
+        signal.alarm(0)  # Cancel the alarm after the function completes or times out
+
+
+def post_process(response, right_answer, task_mode, category):
+    from word2number import w2n
+    if task_mode in ["long_acc", "ref_acc"]:
+        result = {"acc": 0, "oboa": 0, "mae": 0, "rmse": 0}
+        if response:
+            try:
+                pred = w2n.word_to_num(response)
+            except Exception:
+                pred = 0
+            if abs(float(right_answer) - float(pred)) <= 1e-5:
+                result["acc"] = 1
+
+            if abs(float(right_answer) - float(pred)) <= 1:
+                result["oboa"] = 1
+
+            if abs(float(right_answer) - float(pred)) <= max(2 * float(right_answer), 100):
+                result["mae"] = abs(float(right_answer) - float(pred))
+                result["rmse"] = abs(float(right_answer) - float(pred)) ** 2
+            else:
+                result["mae"] = abs(float(right_answer) * 2)
+                result["rmse"] = abs(float(right_answer) * 2) ** 2
+    elif task_mode == "clue_acc":
+        result = {"wcs": 0, "ifa": 0}
+        if response:
+            clues = json.loads(right_answer)
+            content_match = re.search(r"<answer>(.*?)</answer>", response, re.DOTALL)
+            student_answer = content_match.group(1).strip() if content_match else response.strip()
+            j = None
+            try:
+                try:
+                    j = json.loads(student_answer)
+                except Exception:
+                    j = json.loads(extract_outer_json(student_answer))
+            except Exception:
+                pass
+            if j is not None:
+                try:
+                    if category == "event":
+                        pred = []
+                        for e in j:
+
+                            if isinstance(e[0], str) and isinstance(e[1], str) and ":" in e[0] and ":" in e[1]:
+                                pred.append([time_str_to_seconds(e[0]), time_str_to_seconds(e[1])])
+                            else:
+                                pred.append([float(e[0].split(" ")[0]) if isinstance(e[0], str) else e[0],
+                                             float(e[1].split(" ")[0]) if isinstance(e[1], str) else e[1]])
+                        gt = []
+                        for e in clues:
+                            gt.append([float(e['start']), float(e['end'])])
+
+                        result["wcs"] = compute_wcs_unlabeled([gt], [pred], "tIoU")
+                        result["ifa"] = 1
+                    elif category == "object":
+                        gt = []
+                        clue_timestamp_list = []
+                        for clue in clues:
+                            if clue["timestamp"] not in clue_timestamp_list:
+                                clue_timestamp_list.append(clue["timestamp"])
+                        for clue in clues:
+                            gt.append((clue_timestamp_list.index(clue["timestamp"]), clue['bbox']))
+                        pred = []
+                        for key in j.keys():
+                            if "Frame" not in key:
+                                continue
+                            idx = int(key.replace("Frame", "")) - 1
+                            if len(j[key]) == 0:
+                                continue
+                            if isinstance(j[key][0], list) and len(j[key][0]) == 4:
+                                for e in j[key]:
+                                    if isinstance(e, list) and len(e) == 4:
+                                        pred.append((idx, e))
+                            elif isinstance(j[key][0], list) and len(j[key][0]) == 2:
+                                for ii in range(int(len(j[key]) // 2)):
+                                    if isinstance(j[key][ii * 2], list) and len(j[key][ii * 2]) == 2 and isinstance(
+                                            j[key][ii * 2 + 1], list) and len(j[key][ii * 2 + 1]) == 2:
+                                        pred.append((idx, [j[key][ii * 2][0], j[key][ii * 2][1], j[key][ii * 2 + 1][0],
+                                                           j[key][ii * 2 + 1][1]]))
+                        result["wcs"] = compute_wcs_unlabeled([gt], [pred], "sIoU")
+                        result["ifa"] = 1
+                    elif category == "attribute":
+                        gt = []
+                        clue_timestamp_list = []
+                        for clue_ in clues:
+                            for clue in clue_:
+                                if clue["timestamp"] not in clue_timestamp_list:
+                                    clue_timestamp_list.append(clue["timestamp"])
+                        for clue_ in clues:
+                            gt_ = []
+                            for clue in clue_:
+                                gt_.append((clue_timestamp_list.index(clue["timestamp"]), clue['bbox']))
+                            gt.append(gt_)
+                        pred = {}
+                        for key in j.keys():
+                            if "Frame" not in key:
+                                continue
+                            idx = int(key.replace("Frame", "")) - 1
+                            for e in j[key]:
+                                if e['label'] not in pred.keys():
+                                    pred[e['label']] = []
+                                if 'bbox' in e:
+                                    if isinstance(e['bbox'], list) and len(e['bbox']) == 4:
+                                        pred[e['label']].append((idx, e['bbox']))
+                                if 'bbox_2d' in e:
+                                    if isinstance(e['bbox_2d'], list) and len(e['bbox_2d']) == 4:
+                                        pred[e['label']].append((idx, e['bbox_2d']))
+                        pred_list = [pred[key] for key in pred]
+                        result["wcs"] = compute_wcs_unlabeled(gt, pred_list, "sIoU")
+                        result["ifa"] = 1
+                except Exception:
+                    pass
+
+    return result
+
+
+def get_chunk_number(filename):
+    try:
+        num = filename.split("chunk_")[1].split(".zip")[0]
+        return int(num)
+    except Exception:
+        return float('inf')
+
+
+def auto_merge_and_unzip_parts(target_dir, extract_dir, zip_prefix=None):
+    target_dir = Path(target_dir)
+    extract_dir = Path(extract_dir)
+    extract_dir.mkdir(parents=True, exist_ok=True)
+
+    # 匹配 zip 分卷：例如 video_chunk_001.zip.part000
+    part_files = sorted(target_dir.glob("*.zip.part*"))
+    groups = {}
+
+    # 分组：根据前缀提取 group 名（即 zip 文件名）
+    for part_file in part_files:
+        match = re.match(r"(.*\.zip)\.part\d+$", part_file.name)
+        if match:
+            zip_name = match.group(1)
+            if zip_prefix is None or Path(zip_name).stem.startswith(zip_prefix):
+                groups.setdefault(zip_name, []).append(part_file)
+
+    if not groups:
+        print(f"No matching zip parts found with prefix: {zip_prefix}")
+        return
+
+    # 合并每一组分卷 -> 解压
+    for zip_name, parts in tqdm(groups.items(), desc="Merging and unzipping"):
+        parts = sorted(parts, key=lambda p: int(p.name.split("part")[-1]))
+        zip_path = target_dir / zip_name
+
+        # 合并分卷
+        with open(zip_path, 'wb') as outfile:
+            for part in parts:
+                with open(part, 'rb') as infile:
+                    outfile.write(infile.read())
+
+        # 解压合并后的 zip 文件
+        with zipfile.ZipFile(zip_path, 'r') as zip_ref:
+            zip_ref.extractall(extract_dir)
+
+        # 删除合并后的 zip 文件（可注释）
+        zip_path.unlink()
+
+
+def unzip_hf_zip(target_dir):
+    target_dir = Path(target_dir)
+
+    videos_dir = target_dir / "cg_videos_720p"
+    ref_videos_dir = target_dir / "ref_videos"
+
+    if videos_dir.exists() and ref_videos_dir.exists():
+        print("all target dirs exist, skip.")
+        return
+
+    videos_dir.mkdir(parents=True, exist_ok=True)
+
+    auto_merge_and_unzip_parts(target_dir, ref_videos_dir, zip_prefix="ref_videos")
+    auto_merge_and_unzip_parts(target_dir, videos_dir, zip_prefix="videos")
+
+    print("sucessfully unzip all files.")

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/EgoExoBench/tf2023_preprocess.py

@@ -0,0 +1,72 @@
+import json
+import os
+
+import cv2
+import numpy as np
+
+# replace the path with your actual path
+ann_file = 'EgoExoBench/MCQ/Ego-Exo-View-Transition/ego_wearer_identification.json'
+
+
+def add_bbox(bbox_img_path):
+
+    bbox_dir = os.path.dirname(bbox_img_path)
+    os.makedirs(bbox_dir, exist_ok=True)
+    vid, frame_idx, person_id = bbox_img_path.split('/')[-4], bbox_img_path.split('/')[-2], bbox_img_path.split('/')[-1].split('.')[0]  # noqa: E501
+    import os.path as osp
+    json_file = os.path.join(osp.dirname(osp.dirname(osp.dirname(osp.dirname(bbox_img_path)))), vid, 'Segmentation/T', frame_idx + '.json')  # noqa: E501
+    ori_img_path = json_file.replace('.json', '.jpg')
+
+    with open(json_file, mode='r', encoding="utf-8") as f:
+        configs = json.load(f)
+    shapes = configs["shapes"]
+
+    mask = np.zeros((configs["imageHeight"], configs["imageWidth"], 1), np.uint8)
+
+    if not os.path.exists(ori_img_path):
+        ori_img_path = ori_img_path.replace('T/', '')
+
+    if not os.path.exists(ori_img_path):
+        ori_img_path = ori_img_path.replace('Segmentation/', 'frame/T/')
+
+    original_image = cv2.imread(ori_img_path)
+
+    for shape in shapes:
+        if shape['label'] != person_id:
+            continue
+
+        cv2.fillPoly(mask, [np.array(shape["points"], np.int32)], 1)
+
+        retval, labels, stats, centroids = cv2.connectedComponentsWithStats(mask, connectivity=8)
+        stats = stats[stats[:, 4].argsort()]
+        bboxs = stats[:-1]
+
+        for b in bboxs:
+            x0, y0 = b[0], b[1]
+            x1 = b[0] + b[2]
+            y1 = b[1] + b[3]
+
+            start_point, end_point = (x0, y0), (x1, y1)
+            color = (0, 0, 255)
+            thickness = 2
+            mask_bboxs = cv2.rectangle(original_image, start_point, end_point, color, thickness)
+            mask_bboxs = cv2.resize(mask_bboxs, (540, 360))
+            cv2.imwrite(bbox_img_path, mask_bboxs)
+        return
+
+
+def rescale_img(img_path, width, height):
+    img = cv2.imread(img_path)
+    resized_img = cv2.resize(img, (width, height))
+    cv2.imwrite(img_path, resized_img)
+
+
+with open(ann_file, 'r') as f:
+    ann_data = json.load(f)
+    for aitem in ann_data.values():
+        image_paths = []
+        for oitem in aitem['options']:
+            add_bbox(oitem['image_paths'][0])
+
+        for img_path in aitem['query']['image_paths']:
+            rescale_img(img_path, 960, 540)

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/EgoExoBench/utils.py

@@ -0,0 +1,758 @@
+import json
+import math
+import numbers
+import random
+import re
+
+import numpy as np
+import pandas as pd
+import torch
+import torchvision
+from PIL import Image, ImageOps
+
+from vlmeval.smp import load
+from ..utils.multiple_choice import extract_answer_from_item
+
+
+def get_dimension_rating(data_path, category_type='subtask_type'):
+    data = load(data_path)
+    result_board = {}
+    for idx, item in data.iterrows():
+        if item[category_type] not in result_board:
+            result_board[item[category_type]] = [0, 0]
+        result_board[item[category_type]][1] += 1
+        if item['score']:
+            result_board[item[category_type]][0] += 1
+
+    correct = 0
+    total = 0
+    for key, value in result_board.items():
+        correct += value[0]
+        total += value[1]
+        result_board[key].append(f'{value[0] / value[1] * 100:.2f}%')
+
+    result_board['overall'] = [correct, total, f'{correct / total * 100:.2f}%']
+
+    return result_board
+
+
+def extract_characters_regex(s):
+    s = s.strip()
+    answer_prefixes = [
+        'The best answer is',
+        'The correct answer is',
+        'The answer is',
+        'The answer',
+        'The best option is'
+        'The correct option is',
+        'Best answer:'
+        'Best option:',
+        'Answer:',
+        'Option:',
+    ]
+    for answer_prefix in answer_prefixes:
+        s = s.replace(answer_prefix, '')
+
+    if len(s.split()) > 10 and not re.search('[ABCD]', s):
+        return ''
+    matches = re.search(r'[ABCD]', s)
+    if matches is None:
+        return ''
+    return matches[0]
+
+
+def extract_option(model, input_item, dataset_name):
+    options = input_item['question'].split('\n')[1:]
+    for id, option in enumerate(options):
+        option_id = chr(ord('A') + id) + '.'
+        if option.find(option_id) >= 0:
+            input_item[chr(ord('A') + id)] = option[option.find(option_id) + len(option_id):].strip('. \n')
+    return extract_answer_from_item(model, input_item, dataset_name)['opt']
+
+
+def process_results(score_file, model_name):
+    from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score, precision_score,
+                                 recall_score)
+    data = pd.read_excel(score_file)
+
+    # Create the prediction column based on the Score and Answer columns
+    data['prediction'] = data.apply(
+        lambda row: row['answer'] if row['score'] == 1 else ('Yes' if row['answer'] == 'No' else 'No'), axis=1
+    )
+
+    # Recompute metrics for tamper types including 'original' in the calculations but exclude 'original' from the output
+    grouped_metrics_with_original_excluding_original = {}
+
+    original_group = data[data['tamper_type'] == 'original']
+
+    for tamper_type, group in data[data['tamper_type'] != 'original'].groupby('tamper_type'):
+        # Combine the current group with the 'original' group
+        combined_group = pd.concat([group, original_group])
+
+        # Extract ground truth and predictions for the combined group
+        y_true_group = combined_group['answer'].map({'Yes': 1, 'No': 0})
+        y_pred_group = combined_group['prediction'].map({'Yes': 1, 'No': 0})
+
+        # Calculate metrics for the combined group
+        accuracy = accuracy_score(y_true_group, y_pred_group)
+        precision = precision_score(y_true_group, y_pred_group, zero_division=0)
+        recall = recall_score(y_true_group, y_pred_group, zero_division=0)
+        f1 = f1_score(y_true_group, y_pred_group, zero_division=0)
+        conf_matrix = confusion_matrix(y_true_group, y_pred_group)
+
+        # Store metrics for the tamper_type
+        grouped_metrics_with_original_excluding_original[tamper_type] = {
+            "Accuracy": accuracy,
+            "Precision": precision,
+            "Recall": recall,
+            "F1 Score": f1,
+            "Confusion Matrix": conf_matrix.tolist()  # Convert to list for JSON compatibility
+        }
+
+        # Add the Macro Average row to the Dictionary
+        # grouped_metrics_with_original_excluding_original["overall"] = macro_averages
+
+    # Display the metrics in a dataframe for clarity
+    df_grouped_metrics_with_original_excluding_original = pd.DataFrame.from_dict(
+        grouped_metrics_with_original_excluding_original, orient='index'
+    )
+
+    # Compute Macro Averages for Accuracy, Precision, Recall, and F1 Score
+    macro_averages = {
+        "Accuracy": df_grouped_metrics_with_original_excluding_original["Accuracy"].mean(),
+        "Precision": df_grouped_metrics_with_original_excluding_original["Precision"].mean(),
+        "Recall": df_grouped_metrics_with_original_excluding_original["Recall"].mean(),
+        "F1 Score": df_grouped_metrics_with_original_excluding_original["F1 Score"].mean(),
+        "Confusion Matrix": "N/A"  # Macro average doesn't have a meaningful confusion matrix
+    }
+
+    # # Add the Macro Average row to the DataFrame
+    df_grouped_metrics_with_original_excluding_original.loc["overall"] = macro_averages
+
+    # df_grouped_metrics_with_original_excluding_original
+    metrics_dict = json.loads(df_grouped_metrics_with_original_excluding_original.T.to_json())
+    # Process Model Level Metrics
+    formatted_data = []
+    for task, task_metrics in metrics_dict.items():
+        task_metrics['Model'] = model_name
+        task_metrics['Task'] = task
+        formatted_data.append(task_metrics)
+
+    df_metrics = pd.DataFrame(formatted_data)
+
+    # Reorder columns to make 'Model' and 'Task' appear first
+    columns_order = ['Model', 'Task'] + [col for col in df_metrics.columns if col not in ['Model', 'Task']]
+    df_metrics = df_metrics[columns_order]
+
+    return df_metrics
+
+
+def aggregate_metrics_with_macro_average(score_file):
+    from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score, precision_score,
+                                 recall_score)
+
+    # Load data
+    data = pd.read_excel(score_file)
+
+    # Create the prediction column based on the Score and Answer columns
+    data['prediction'] = data.apply(
+        lambda row: row['answer'] if row['score'] == 1 else ('Yes' if row['answer'] == 'No' else 'No'), axis=1
+    )
+
+    # Initialize a dictionary to store metrics
+    task_type_metrics = {}
+
+    # Process each task_type separately
+    for task_type, task_group in data.groupby('task_type'):
+        # Separate the 'original' group for the current task_type
+        original_group = task_group[task_group['tamper_type'] == 'original']
+
+        # Skip if there is no 'original' data for this task_type
+        if original_group.empty:
+            continue
+
+        # Process each tamper type for the current task_type (excluding 'original')
+        tamper_metrics = {}
+        for tamper_type, tamper_group in task_group[task_group['tamper_type'] != 'original'].groupby('tamper_type'):
+
+            # Combine the tamper group with the original group of the current task_type
+            combined_group = pd.concat([tamper_group, original_group])
+
+            # Map answers and predictions to binary values
+            y_true = combined_group['answer'].map({'Yes': 1, 'No': 0})
+            y_pred = combined_group['prediction'].map({'Yes': 1, 'No': 0})
+
+            # Compute metrics
+            accuracy = accuracy_score(y_true, y_pred)
+            precision = precision_score(y_true, y_pred, zero_division=0)
+            recall = recall_score(y_true, y_pred, zero_division=0)
+            f1 = f1_score(y_true, y_pred, zero_division=0)
+            conf_matrix = confusion_matrix(y_true, y_pred)
+
+            # Store metrics for the tamper_type
+            tamper_metrics[tamper_type] = {
+                "Accuracy": accuracy,
+                "Precision": precision,
+                "Recall": recall,
+                "F1 Score": f1,
+                "Confusion Matrix": conf_matrix.tolist()  # Convert to list for JSON compatibility
+            }
+
+        # Compute Macro Averages for the current task_type
+        metrics_df = pd.DataFrame(tamper_metrics).T
+        macro_average = {
+            "Accuracy": metrics_df["Accuracy"].mean(),
+            "Precision": metrics_df["Precision"].mean(),
+            "Recall": metrics_df["Recall"].mean(),
+            "F1 Score": metrics_df["F1 Score"].mean(),
+            "Confusion Matrix": "N/A"  # Macro average doesn't have a meaningful confusion matrix
+        }
+
+        # Add the macro average as "overall" for the task_type
+        tamper_metrics["overall"] = macro_average
+
+        # Add tamper metrics for the current task_type to the main dictionary
+        task_type_metrics[task_type] = tamper_metrics
+
+    # Transform the nested dictionary into a DataFrame
+    dataframes = []
+    for task_type, metrics in task_type_metrics.items():
+        task_df = pd.DataFrame.from_dict(metrics, orient='index')
+        task_df['task_type'] = task_type  # Add the task_type as a column
+        dataframes.append(task_df)
+
+    # Combine all task-specific DataFrames into a single DataFrame
+    result_df = pd.concat(dataframes).reset_index().rename(columns={'index': 'tamper_type'})
+    # Reorder the columns to place task_type first, then tamper_type
+    result_df = result_df[['task_type', 'tamper_type', 'Accuracy', 'Precision', 'Recall',
+                           'F1 Score', 'Confusion Matrix']]
+
+    # Select only numeric columns for aggregation
+    numeric_columns = ['Accuracy', 'Precision', 'Recall', 'F1 Score']
+
+    # Group by task_type and tamper_type, and calculate the mean for numeric columns
+    average_metrics = result_df.groupby(['task_type', 'tamper_type'])[numeric_columns].mean().reset_index()
+
+    return average_metrics
+
+
+def check_ans(pred, gt):
+    """
+    Checks if the predicted answer matches the ground truth.
+
+    Args:
+        pred (str): The predicted answer.
+        gt (str): The ground truth answer.
+
+    Returns:
+        bool: True if the predicted answer matches the ground truth, False otherwise.
+    """
+    # Convert both predictions and ground truths to lowercase and split them into options and contents
+    flag = False
+
+    # Split prediction into option and content
+    pred_list = pred.lower().strip().split(' ')
+    pred_option, _ = pred_list[0], ' '.join(pred_list[1:])
+
+    # Split ground truth into option and content
+    gt_list = gt.lower().strip().split(' ')
+    gt_option, gt_content = gt_list[0], ' '.join(gt_list[1:])
+
+    # Remove trailing period from ground truth content if present
+    if gt_content[-1] == '.':
+        gt_content = gt_content[:-1]
+
+    # Check for matching conditions
+    # Condition 1: If the predicted option is a substring of the ground truth option
+    if pred_option.replace('.', '') in gt_option:
+        flag = True
+    # Condition 2: If the ground truth option is a substring of the predicted option
+    elif gt_option in pred_option:
+        flag = True
+    # Condition 3: If the ground truth is a substring of the predicted answer
+    elif gt in pred:
+        flag = True
+
+    return flag
+
+
+def check_ans_with_model(pred, gt, model, item, dataset_name='MVBench'):
+    """
+    Checks if the predicted answer matches the ground truth using a given model.
+
+    Args:
+        pred (str): The predicted answer.
+        gt (str): The ground truth answer.
+        model: A machine learning model used for additional verification.
+        item (dict): An item containing information about the question or task.
+        dataset_name (str, optional): Name of the dataset being used. Defaults to 'MVBench'.
+
+    Returns:
+        bool: True if the predicted answer matches the ground truth, False otherwise.
+    """
+    # Initialize flag to track match status
+    flag = False
+
+    # Preprocess prediction and ground truth by converting to lowercase and splitting into options and contents
+    pred_list = pred.lower().strip().split(' ')
+    pred_option, _ = pred_list[0], ' '.join(pred_list[1:])
+    gt_list = gt.lower().strip().split(' ')
+    gt_option, gt_content = gt_list[0], ' '.join(gt_list[1:])
+
+    # Remove trailing period from ground truth content if presen
+    if gt_content[-1] == '.':
+        gt_content = gt_content[:-1]
+
+    # Check for matching conditions
+    # Condition 1: If the predicted option is a substring of the ground truth option
+    if pred_option.replace('.', '') in gt_option:
+        flag = True
+    # Condition 2: If the ground truth option is a substring of the predicted option
+    elif gt_option in pred_option:
+        flag = True
+    # Condition 3: Use the provided model to verify the answer
+    elif extract_answer_from_item(model, item, dataset_name)['opt'] == item['answer']:
+        flag = True
+
+    return flag
+
+
+def check_ans_advanced(pred, gt):
+    number_table = {
+        0: 'zero',
+        1: 'one',
+        2: 'two',
+        3: 'three',
+        4: 'four',
+        5: 'five',
+        6: 'six',
+        7: 'seven',
+        8: 'eight',
+        9: 'nine',
+    }
+    flag = False
+
+    pred_list = pred.lower().split(' ')
+    pred_option, _ = pred_list[0], ' '.join(pred_list[1:])
+    gt_list = gt.lower().split(' ')
+    gt_option, gt_content = gt_list[0], ' '.join(gt_list[1:])
+    if gt_content[-1] == '.':
+        gt_content = gt_content[:-1]
+
+    try:
+        gt_content = number_table[int(gt_content.strip('. \n'))]
+        print(gt_content)
+    except Exception:
+        pass
+
+    if pred_option.replace('.', '') in gt_option:
+        flag = True
+    elif gt_option in pred_option:
+        flag = True
+    elif gt_content.lower().strip('. \n') in pred.lower().strip('. \n'):
+        flag = True
+
+    return flag
+
+
+class GroupRandomCrop(object):
+    def __init__(self, size):
+        if isinstance(size, numbers.Number):
+            self.size = (int(size), int(size))
+        else:
+            self.size = size
+
+    def __call__(self, img_group):
+
+        w, h = img_group[0].size
+        th, tw = self.size
+
+        out_images = list()
+
+        x1 = random.randint(0, w - tw)
+        y1 = random.randint(0, h - th)
+
+        for img in img_group:
+            assert (img.size[0] == w and img.size[1] == h)
+            if w == tw and h == th:
+                out_images.append(img)
+            else:
+                out_images.append(img.crop((x1, y1, x1 + tw, y1 + th)))
+
+        return out_images
+
+
+class MultiGroupRandomCrop(object):
+    def __init__(self, size, groups=1):
+        if isinstance(size, numbers.Number):
+            self.size = (int(size), int(size))
+        else:
+            self.size = size
+        self.groups = groups
+
+    def __call__(self, img_group):
+
+        w, h = img_group[0].size
+        th, tw = self.size
+
+        out_images = list()
+
+        for i in range(self.groups):
+            x1 = random.randint(0, w - tw)
+            y1 = random.randint(0, h - th)
+
+            for img in img_group:
+                assert (img.size[0] == w and img.size[1] == h)
+                if w == tw and h == th:
+                    out_images.append(img)
+                else:
+                    out_images.append(img.crop((x1, y1, x1 + tw, y1 + th)))
+
+        return out_images
+
+
+class GroupCenterCrop(object):
+    def __init__(self, size):
+        self.worker = torchvision.transforms.CenterCrop(size)
+
+    def __call__(self, img_group):
+        return [self.worker(img) for img in img_group]
+
+
+class GroupRandomHorizontalFlip(object):
+    """Randomly horizontally flips the given PIL.Image with a probability of 0.5
+    """
+
+    def __init__(self, is_flow=False):
+        self.is_flow = is_flow
+
+    def __call__(self, img_group, is_flow=False):
+        v = random.random()
+        if v < 0.5:
+            ret = [img.transpose(Image.FLIP_LEFT_RIGHT) for img in img_group]
+            if self.is_flow:
+                for i in range(0, len(ret), 2):
+                    # invert flow pixel values when flipping
+                    ret[i] = ImageOps.invert(ret[i])
+            return ret
+        else:
+            return img_group
+
+
+class GroupNormalize(object):
+    def __init__(self, mean, std):
+        self.mean = mean
+        self.std = std
+
+    def __call__(self, tensor):
+        rep_mean = self.mean * (tensor.size()[0] // len(self.mean))
+        rep_std = self.std * (tensor.size()[0] // len(self.std))
+
+        # TODO: make efficient
+        for t, m, s in zip(tensor, rep_mean, rep_std):
+            t.sub_(m).div_(s)
+
+        return tensor
+
+
+class GroupScale(object):
+    """ Rescales the input PIL.Image to the given 'size'.
+    'size' will be the size of the smaller edge.
+    For example, if height > width, then image will be
+    rescaled to (size * height / width, size)
+    size: size of the smaller edge
+    interpolation: Default: PIL.Image.BILINEAR
+    """
+
+    def __init__(self, size, interpolation=Image.BILINEAR):
+        self.worker = torchvision.transforms.Resize(size, interpolation)
+
+    def __call__(self, img_group):
+        return [self.worker(img) for img in img_group]
+
+
+class GroupOverSample(object):
+    def __init__(self, crop_size, scale_size=None, flip=True):
+        self.crop_size = crop_size if not isinstance(
+            crop_size, int) else (crop_size, crop_size)
+
+        if scale_size is not None:
+            self.scale_worker = GroupScale(scale_size)
+        else:
+            self.scale_worker = None
+        self.flip = flip
+
+    def __call__(self, img_group):
+
+        if self.scale_worker is not None:
+            img_group = self.scale_worker(img_group)
+
+        image_w, image_h = img_group[0].size
+        crop_w, crop_h = self.crop_size
+
+        offsets = GroupMultiScaleCrop.fill_fix_offset(
+            False, image_w, image_h, crop_w, crop_h)
+        oversample_group = list()
+        for o_w, o_h in offsets:
+            normal_group = list()
+            flip_group = list()
+            for i, img in enumerate(img_group):
+                crop = img.crop((o_w, o_h, o_w + crop_w, o_h + crop_h))
+                normal_group.append(crop)
+                flip_crop = crop.copy().transpose(Image.FLIP_LEFT_RIGHT)
+
+                if img.mode == 'L' and i % 2 == 0:
+                    flip_group.append(ImageOps.invert(flip_crop))
+                else:
+                    flip_group.append(flip_crop)
+
+            oversample_group.extend(normal_group)
+            if self.flip:
+                oversample_group.extend(flip_group)
+        return oversample_group
+
+
+class GroupFullResSample(object):
+    def __init__(self, crop_size, scale_size=None, flip=True):
+        self.crop_size = crop_size if not isinstance(
+            crop_size, int) else (crop_size, crop_size)
+
+        if scale_size is not None:
+            self.scale_worker = GroupScale(scale_size)
+        else:
+            self.scale_worker = None
+        self.flip = flip
+
+    def __call__(self, img_group):
+
+        if self.scale_worker is not None:
+            img_group = self.scale_worker(img_group)
+
+        image_w, image_h = img_group[0].size
+        crop_w, crop_h = self.crop_size
+
+        w_step = (image_w - crop_w) // 4
+        h_step = (image_h - crop_h) // 4
+
+        offsets = list()
+        offsets.append((0 * w_step, 2 * h_step))  # left
+        offsets.append((4 * w_step, 2 * h_step))  # right
+        offsets.append((2 * w_step, 2 * h_step))  # center
+
+        oversample_group = list()
+        for o_w, o_h in offsets:
+            normal_group = list()
+            flip_group = list()
+            for i, img in enumerate(img_group):
+                crop = img.crop((o_w, o_h, o_w + crop_w, o_h + crop_h))
+                normal_group.append(crop)
+                if self.flip:
+                    flip_crop = crop.copy().transpose(Image.FLIP_LEFT_RIGHT)
+
+                    if img.mode == 'L' and i % 2 == 0:
+                        flip_group.append(ImageOps.invert(flip_crop))
+                    else:
+                        flip_group.append(flip_crop)
+
+            oversample_group.extend(normal_group)
+            oversample_group.extend(flip_group)
+        return oversample_group
+
+
+class GroupMultiScaleCrop(object):
+
+    def __init__(self, input_size, scales=None, max_distort=1,
+                 fix_crop=True, more_fix_crop=True):
+        self.scales = scales if scales is not None else [1, .875, .75, .66]
+        self.max_distort = max_distort
+        self.fix_crop = fix_crop
+        self.more_fix_crop = more_fix_crop
+        self.input_size = input_size if not isinstance(input_size, int) else [
+            input_size, input_size]
+        self.interpolation = Image.BILINEAR
+
+    def __call__(self, img_group):
+
+        im_size = img_group[0].size
+
+        crop_w, crop_h, offset_w, offset_h = self._sample_crop_size(im_size)
+        crop_img_group = [
+            img.crop(
+                (offset_w,
+                 offset_h,
+                 offset_w + crop_w,
+                 offset_h + crop_h)) for img in img_group]
+        ret_img_group = [img.resize((self.input_size[0], self.input_size[1]), self.interpolation)
+                         for img in crop_img_group]
+        return ret_img_group
+
+    def _sample_crop_size(self, im_size):
+        image_w, image_h = im_size[0], im_size[1]
+
+        # find a crop size
+        base_size = min(image_w, image_h)
+        crop_sizes = [int(base_size * x) for x in self.scales]
+        crop_h = [
+            self.input_size[1] if abs(
+                x - self.input_size[1]) < 3 else x for x in crop_sizes]
+        crop_w = [
+            self.input_size[0] if abs(
+                x - self.input_size[0]) < 3 else x for x in crop_sizes]
+
+        pairs = []
+        for i, h in enumerate(crop_h):
+            for j, w in enumerate(crop_w):
+                if abs(i - j) <= self.max_distort:
+                    pairs.append((w, h))
+
+        crop_pair = random.choice(pairs)
+        if not self.fix_crop:
+            w_offset = random.randint(0, image_w - crop_pair[0])
+            h_offset = random.randint(0, image_h - crop_pair[1])
+        else:
+            w_offset, h_offset = self._sample_fix_offset(
+                image_w, image_h, crop_pair[0], crop_pair[1])
+
+        return crop_pair[0], crop_pair[1], w_offset, h_offset
+
+    def _sample_fix_offset(self, image_w, image_h, crop_w, crop_h):
+        offsets = self.fill_fix_offset(
+            self.more_fix_crop, image_w, image_h, crop_w, crop_h)
+        return random.choice(offsets)
+
+    @staticmethod
+    def fill_fix_offset(more_fix_crop, image_w, image_h, crop_w, crop_h):
+        w_step = (image_w - crop_w) // 4
+        h_step = (image_h - crop_h) // 4
+
+        ret = list()
+        ret.append((0, 0))  # upper left
+        ret.append((4 * w_step, 0))  # upper right
+        ret.append((0, 4 * h_step))  # lower left
+        ret.append((4 * w_step, 4 * h_step))  # lower right
+        ret.append((2 * w_step, 2 * h_step))  # center
+
+        if more_fix_crop:
+            ret.append((0, 2 * h_step))  # center left
+            ret.append((4 * w_step, 2 * h_step))  # center right
+            ret.append((2 * w_step, 4 * h_step))  # lower center
+            ret.append((2 * w_step, 0 * h_step))  # upper center
+
+            ret.append((1 * w_step, 1 * h_step))  # upper left quarter
+            ret.append((3 * w_step, 1 * h_step))  # upper right quarter
+            ret.append((1 * w_step, 3 * h_step))  # lower left quarter
+            ret.append((3 * w_step, 3 * h_step))  # lower righ quarter
+
+        return ret
+
+
+class GroupRandomSizedCrop(object):
+    """Random crop the given PIL.Image to a random size of (0.08 to 1.0) of the original size
+    and and a random aspect ratio of 3/4 to 4/3 of the original aspect ratio
+    This is popularly used to train the Inception networks
+    size: size of the smaller edge
+    interpolation: Default: PIL.Image.BILINEAR
+    """
+
+    def __init__(self, size, interpolation=Image.BILINEAR):
+        self.size = size
+        self.interpolation = interpolation
+
+    def __call__(self, img_group):
+        for attempt in range(10):
+            area = img_group[0].size[0] * img_group[0].size[1]
+            target_area = random.uniform(0.08, 1.0) * area
+            aspect_ratio = random.uniform(3. / 4, 4. / 3)
+
+            w = int(round(math.sqrt(target_area * aspect_ratio)))
+            h = int(round(math.sqrt(target_area / aspect_ratio)))
+
+            if random.random() < 0.5:
+                w, h = h, w
+
+            if w <= img_group[0].size[0] and h <= img_group[0].size[1]:
+                x1 = random.randint(0, img_group[0].size[0] - w)
+                y1 = random.randint(0, img_group[0].size[1] - h)
+                found = True
+                break
+        else:
+            found = False
+            x1 = 0
+            y1 = 0
+
+        if found:
+            out_group = list()
+            for img in img_group:
+                img = img.crop((x1, y1, x1 + w, y1 + h))
+                assert (img.size == (w, h))
+                out_group.append(
+                    img.resize(
+                        (self.size, self.size), self.interpolation))
+            return out_group
+        else:
+            # Fallback
+            scale = GroupScale(self.size, interpolation=self.interpolation)
+            crop = GroupRandomCrop(self.size)
+            return crop(scale(img_group))
+
+
+class ConvertDataFormat(object):
+    def __init__(self, model_type):
+        self.model_type = model_type
+
+    def __call__(self, images):
+        if self.model_type == '2D':
+            return images
+        tc, h, w = images.size()
+        t = tc // 3
+        images = images.view(t, 3, h, w)
+        images = images.permute(1, 0, 2, 3)
+        return images
+
+
+class Stack(object):
+
+    def __init__(self, roll=False):
+        self.roll = roll
+
+    def __call__(self, img_group):
+        if img_group[0].mode == 'L':
+            return np.concatenate([np.expand_dims(x, 2)
+                                   for x in img_group], axis=2)
+        elif img_group[0].mode == 'RGB':
+            if self.roll:
+                return np.concatenate([np.array(x)[:, :, ::-1]
+                                       for x in img_group], axis=2)
+            else:
+                # print(np.concatenate(img_group, axis=2).shape)
+                # print(img_group[0].shape)
+                return np.concatenate(img_group, axis=2)
+
+
+class ToTorchFormatTensor(object):
+    """ Converts a PIL.Image (RGB) or numpy.ndarray (H x W x C) in the range [0, 255]
+    to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0] """
+
+    def __init__(self, div=True):
+        self.div = div
+
+    def __call__(self, pic):
+        if isinstance(pic, np.ndarray):
+            # handle numpy array
+            img = torch.from_numpy(pic).permute(2, 0, 1).contiguous()
+        else:
+            # handle PIL Image
+            img = torch.ByteTensor(
+                torch.ByteStorage.from_buffer(
+                    pic.tobytes()))
+            img = img.view(pic.size[1], pic.size[0], len(pic.mode))
+            # put it from HWC to CHW format
+            # yikes, this transpose takes 80% of the loading time/CPU
+            img = img.transpose(0, 1).transpose(0, 2).contiguous()
+        return img.float().div(255) if self.div else img.float()
+
+
+class IdentityTransform(object):
+
+    def __call__(self, data):
+        return data

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/GUI/osworld_g.py

@@ -0,0 +1,441 @@
+import ast
+import itertools
+import json
+import os
+import os.path as osp
+import re
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+from PIL import Image
+from tqdm import tqdm
+
+from vlmeval.dataset.image_base import ImageBaseDataset
+from vlmeval.smp import LMUDataRoot, dump, get_intermediate_file_path, get_logger, load, toliststr
+
+logger = get_logger(__name__)
+
+SYSTEM_PROMPT = """You are a GUI agent. You are given a task and a screenshot of the screen. You need to perform pyautogui click/moveTo action to complete the task. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+
+USER_INSTRUCTION = """Please complete the following tasks by clicking using `pyautogui.click`:\n{instruction}"""  # noqa: E501
+
+SYSTEM_PROMPT_V2 = """You are a GUI agent. You are given a screenshot of the screen and the description of a target element. You need to click the target element using `pyautogui.click`. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+
+USER_INSTRUCTION_V2 = """Please click the following target element using `pyautogui.click`:\n{description}"""
+
+
+def parse_bbox_aguvis(response):
+    match = re.search(r"x=([\d.]+), y=([\d.]+)", response)
+    if match:
+        click_point = [float(match.group(1)), float(match.group(2))]
+    else:
+        click_point = [0.0, 0.0]
+    return click_point
+
+
+def compute_iou(box1, box2):
+    """
+    Compute the Intersection over Union (IoU) of two bounding boxes.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - float: IoU of box1 and box2.
+    """
+    # Determine the coordinates of the intersection rectangle
+    x_left = max(box1[0], box2[0])
+    y_top = max(box1[1], box2[1])
+    x_right = min(box1[2], box2[2])
+    y_bottom = min(box1[3], box2[3])
+
+    # Compute the area of intersection
+    intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top)
+
+    # Compute the area of both bounding boxes
+    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
+    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
+
+    # Compute the area of the union
+    union_area = box1_area + box2_area - intersection_area
+
+    # Compute the Intersection over Union
+    iou = intersection_area / union_area
+
+    return iou
+
+
+def compute_accuracy(box1, box2, threshold=0.5):
+    """
+    Compute the accuracy of two bounding boxes based on a specified threshold.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - threshold (float): Threshold for the IoU to consider the prediction correct.
+
+    Returns:
+    - float: Accuracy of the prediction based on the IoU threshold.
+    """
+    iou = compute_iou(box1, box2)
+    return iou >= threshold
+
+
+def compute_center_accuracy(box1, box2):
+    """
+    Compute if the center point of box 2 is within box 1.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - bool: True if the center point of box 2 is within box 1, False otherwise.
+    """
+    # Compute the center point of box 2
+    center_x = (box2[0] + box2[2]) / 2
+    center_y = (box2[1] + box2[3]) / 2
+
+    # Check if the center point is within box 1
+    return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3]
+
+
+def convert_bbox(bbox, image_path, convert_xywh_to_x1y1x2y2=True):
+    new_bbox = bbox if isinstance(bbox, list) else ast.literal_eval(bbox)
+    if convert_xywh_to_x1y1x2y2:
+        new_bbox = [
+            new_bbox[0],
+            new_bbox[1],
+            new_bbox[0] + new_bbox[2],
+            new_bbox[1] + new_bbox[3],
+        ]
+    image = Image.open(image_path)
+    img_size = image.size
+    new_bbox = [
+        new_bbox[0] / img_size[0],
+        new_bbox[1] / img_size[1],
+        new_bbox[2] / img_size[0],
+        new_bbox[3] / img_size[1],
+    ]
+    return new_bbox
+
+
+class OSWorld_G(ImageBaseDataset):
+    MODALITY = "IMAGE"
+    TYPE = "GUI"
+    DATASET_URL = {
+        "OSWorld_G": "https://opencompass.openxlab.space/utils/VLMEval/OSWorld_G.tsv",  # Optional, dummy URL
+    }  # path
+    DATASET_MD5 = {
+        'OSWorld_G': 'eee81b61210f580cbc98b11c6bced928'
+    }
+    EVAL_TYPE = "point"  # point or rectangle
+    RE_TYPE = "functional"  # type of referring expressions: functional or composite
+
+    def __init__(
+        self,
+        dataset="OSWorld_G",
+        skip_noimg=True,
+        skeleton=False,
+        re_type="functional",
+    ):
+        # st()
+        ROOT = LMUDataRoot()
+        # You can override this variable to save image files to a different directory
+        self.dataset_name = dataset
+        self.img_root = osp.join(ROOT, "images", self.dataset_name)
+        self.RE_TYPE = re_type
+        if skeleton:
+            return
+
+        data = self.load_data(dataset)
+        self.skip_noimg = skip_noimg
+        if skip_noimg and "image" in data:
+            data = data[~pd.isna(data["image"])]
+
+        data["index"] = [str(idx + 1) for idx, x in enumerate(data["bbox"])]
+
+        self.meta_only = True
+        self.parse_response_func = parse_bbox_aguvis  # TODO: parse function can be specified through kwargs when initializing the dataset # noqa: E501
+
+        # The image field can store the base64 encoded image or another question index (for saving space) # noqa: E501
+        if "image" in data:
+            data["image"] = [str(x) for x in data["image"]]
+            image_map = {x: y for x, y in zip(data["index"], data["image"])}
+            for k in image_map:
+                if len(image_map[k]) <= 64:
+                    idx = image_map[k]
+                    assert idx in image_map and len(image_map[idx]) > 64
+                    image_map[k] = image_map[idx]
+
+            images = [toliststr(image_map[k]) for k in data["index"]]
+            data["image"] = [x[0] if len(x) == 1 else x for x in images]
+            self.meta_only = False
+
+        self.data = data
+
+    @classmethod
+    def get_action_space(self):
+        return ""
+
+    @classmethod
+    def get_trajectory(self, line):
+        traj_dict = {}
+        if self.RE_TYPE == "functional":
+            traj_dict["task"] = line["question"]
+        else:
+            traj_dict["task"] = line["description"]
+        return traj_dict
+
+    def build_prompt(self, line):
+        if isinstance(line, int):
+            line = self.data.iloc[line]
+        tgt_path = self.dump_image(line)
+
+        if self.RE_TYPE == "functional":
+            user_instruction = USER_INSTRUCTION.format(instruction=line["question"])
+        else:
+            user_instruction = USER_INSTRUCTION_V2.format(
+                description=line["description"]
+            )
+
+        msgs = []
+        # add system prompt
+        if self.RE_TYPE == "functional":
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT))
+        else:
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT_V2))
+        if isinstance(tgt_path, list):
+            msgs.extend([dict(type="image", value=p) for p in tgt_path])
+        else:
+            msgs = [dict(type="image", value=tgt_path)]
+        msgs.append(dict(type="text", value=user_instruction))
+        return msgs
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        # st()
+        if self.EVAL_TYPE == "point":
+            return self.evaluate_point(eval_file, **judge_kwargs)
+
+        elif self.EVAL_TYPE == "rectangle":
+            return self.evaluate_rectangle(eval_file, **judge_kwargs)
+
+    def evaluate_rectangle(self, eval_file, **judge_kwargs):
+        scorers = {
+            "IoU": compute_iou,
+            "ACC@0.1": lambda x, y: compute_accuracy(x, y, 0.1),
+            "ACC@0.3": lambda x, y: compute_accuracy(x, y, 0.3),
+            "ACC@0.5": lambda x, y: compute_accuracy(x, y, 0.5),
+            "ACC@0.7": lambda x, y: compute_accuracy(x, y, 0.7),
+            "ACC@0.9": lambda x, y: compute_accuracy(x, y, 0.9),
+            "Center_ACC": compute_center_accuracy,
+        }
+        results_dict = {}
+        for key in scorers.keys():
+            results_dict.update(
+                {
+                    key: [],
+                    key + "_text": [],
+                    key + "_icon": [],
+                }
+            )
+
+        result = []
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = convert_bbox(
+                line["bbox"], os.path.join(self.img_root, line["image_path"]), convert_xywh_to_x1y1x2y2=False
+            )
+            prediction = str(line["prediction"])
+            try:
+                click_point = parse_bbox_aguvis(prediction)
+
+                match = {}
+                for score_key, score_value in scorers.items():
+                    score = score_value(bbox, click_point)
+                    if score_key != "IoU":
+                        match[score_key.replace("ACC", "match")] = score
+                    results_dict[score_key].append(score)
+                    if line["ui_type"] == "text":
+                        results_dict[score_key + "_text"].append(score)
+                    else:
+                        results_dict[score_key + "_icon"].append(score)
+            except Exception:
+                click_point = None
+                match = {score_key: False for score_key in scorers.keys() if score_key != "IoU"}
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "source": line["application"],
+                    "pred": click_point,
+                    "num_matched": sum(match.values()),
+                    **match,
+                }
+            )
+        for key in results_dict:
+            if len(results_dict[key]) == 0:
+                results_dict[key] = str(0)
+            else:
+                results_dict[key] = str(sum(results_dict[key]) / len(results_dict[key]))
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(results_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            failure_cases = [res for res in result if not res["match"] and res["is_wrong_format"]]
+            failure_cases.sort(key=lambda r: r["num_matched"], reverse=True)
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+        return results_dict
+
+    def evaluate_point(self, eval_file, **judge_kwargs):
+        # -1: format_err, 0: wrong, 1: correct
+        stats = defaultdict(list)
+        # Will include instance-level results
+        result = []
+
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = (
+                line["bbox"]
+                if isinstance(line["bbox"], list)
+                else ast.literal_eval(line["bbox"])
+            )
+            # The format of bbox is (x1, y1, x2, y2)
+
+            image = Image.open(os.path.join(self.img_root, line["image_path"]))
+            img_size = image.size
+
+            def make_safe(value):
+                if value == -1:
+                    # we can tolerate -1 as a special value and nomalize it to 0
+                    return 0
+                else:
+                    return value
+
+            bbox = [
+                make_safe(bbox[0]) / img_size[0],
+                make_safe(bbox[1]) / img_size[1],
+                make_safe(bbox[0] + bbox[2]) / img_size[0],
+                make_safe(bbox[1] + bbox[3]) / img_size[1],
+            ]
+
+            key = line["category"] + ":" + line['ui_type']
+            prediction = str(line["prediction"])
+            try:
+                click_point = self.parse_response_func(prediction)
+                # Do Normalization By Default
+                # if click_point[0] > 1 or click_point[1] > 1:
+                click_point = (click_point[0] / 1000, click_point[1] / 1000)
+
+                match = (bbox[0] <= click_point[0] <= bbox[2]) and \
+                    (bbox[1] <= click_point[1] <= bbox[3])
+                # draw click point and box on image
+                # from PIL import ImageDraw
+                # draw = ImageDraw.Draw(image)
+                # draw.rectangle([bbox[0] * img_size[0], bbox[1] * img_size[1],
+                #                 bbox[2] * img_size[0], bbox[3] * img_size[1]], outline="red", width=2)
+                # draw.ellipse([click_point[0] * img_size[0] - 5, click_point[1] * img_size[1] - 5,
+                #               click_point[0] * img_size[0] + 5, click_point[1] * img_size[1] + 5],
+                #              outline="red", width=2)
+                # image.save(f"debug_{i}.png")
+
+                if match:
+                    stats[key].append(1)
+                else:
+                    stats[key].append(0)
+                is_wrong_format = False
+
+            except Exception as e:
+                logger.warning(f"exception in screenspot eval:{e}")
+                stats[key].append(-1)
+                match, is_wrong_format, click_point = False, True, None
+
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "source": line["application"],
+                    "match": match,
+                    "is_wrong_format": is_wrong_format,
+                    "pred": click_point,
+                }
+            )
+
+        final_score_dict = {}
+        # Record the number of each category
+        final_score_dict.update({k + ':cnt': len(stats[k]) for k in stats})
+        # Calculate the Overall stats
+        full_stats = []
+        for v in stats.values():
+            full_stats.extend(v)
+        final_score_dict['Overall_Accuracy'] = np.mean([x > 0 for x in full_stats]) * 100
+        final_score_dict['Format_Err_Rate'] = np.mean([x < 0 for x in full_stats]) * 100
+        # Calculate the Accuracy of Text / Icon
+        text_stats = [v for k, v in stats.items() if k.split(":")[1] == "text" for x in v]
+        text_stats = itertools.chain(*text_stats)
+        final_score_dict['Text_Accuracy'] = np.mean([x > 0 for x in text_stats]) * 100
+        icon_stats = [v for k, v in stats.items() if k.split(":")[1] == "icon" for x in v]
+        icon_stats = itertools.chain(*icon_stats)
+        final_score_dict['Icon_Accuracy'] = np.mean([x > 0 for x in icon_stats]) * 100
+        # Calculate the Accuracy of Each Category
+        cates = list(set(data['category']))
+        for c in cates:
+            sub_stats = [v for k, v in stats.items() if k.split(":")[0] == c for x in v]
+            sub_stats = itertools.chain(*sub_stats)
+            final_score_dict[c + '_Accuracy'] = np.mean([x > 0 for x in sub_stats]) * 100
+
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(final_score_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            def click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                w, h = x2 - x1, y2 - y1
+                abs_shift_to_center = [abs(x - xc), abs(y - yc)]  # noqa: E501
+                width_outside, height_outside = [max(0, abs_shift_to_center[0] - w / 2), max(0, abs_shift_to_center[1] - h / 2)]  # noqa: E501
+                return (width_outside ** 2 + height_outside ** 2) ** 0.5  # noqa: E501
+
+            wrong_format_result = [res for res in result if res["is_wrong_format"]]
+            missed_result = [res for res in result if not res["match"] and not res["is_wrong_format"]]
+            missed_result.sort(key=lambda r: click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            failure_cases = wrong_format_result + missed_result
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+
+        successful_cases_path = os.environ.get("SUCCESSFUL_CASES_PATH", None)
+        if successful_cases_path is not None:
+            def _click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                x_shift, y_shift = x - xc, y - yc
+                return (x_shift ** 2 + y_shift ** 2) ** 0.5
+
+            successful_cases = [res for res in result if res["match"]]
+            successful_cases.sort(key=lambda r: _click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            with open(successful_cases_path, "w") as f:
+                json.dump(successful_cases, f, indent=4, ensure_ascii=False)
+        return final_score_dict

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/GUI/screenspot.py

@@ -0,0 +1,462 @@
+import ast
+import itertools
+import json
+import os
+import os.path as osp
+import re
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+from PIL import Image
+from tqdm import tqdm
+
+from vlmeval.dataset.image_base import ImageBaseDataset
+from vlmeval.smp import LMUDataRoot, dump, get_intermediate_file_path, get_logger, load, toliststr
+
+logger = get_logger(__name__)
+
+"""
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [42, 1102, 197, 70],
+    "question": "view the details of the item",
+    "data_type": "text",
+    "data_source": "shop"
+},
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [93, 74, 86, 132],
+    "question": "view the previous photo",
+    "data_type": "icon",
+    "data_source": "shop"
+}
+"""
+
+SYSTEM_PROMPT = """You are a GUI agent. You are given a task and a screenshot of the screen. You need to perform pyautogui click/moveTo action to complete the task. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+
+USER_INSTRUCTION = """Please complete the following tasks by clicking using `pyautogui.click`:\n{instruction}"""
+
+SYSTEM_PROMPT_V2 = """You are a GUI agent. You are given a screenshot of the screen and the description of a target element. You need to click the target element using `pyautogui.click`. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+USER_INSTRUCTION_V2 = """Please click the following target element using `pyautogui.click`:\n{description}"""
+
+
+def parse_bbox_aguvis(response):
+    match = re.search(r"x=([\d.]+), y=([\d.]+)", response)
+    if match:
+        click_point = [float(match.group(1)), float(match.group(2))]
+    else:
+        click_point = [0.0, 0.0]
+    return click_point
+
+
+def compute_iou(box1, box2):
+    """
+    Compute the Intersection over Union (IoU) of two bounding boxes.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - float: IoU of box1 and box2.
+    """
+    # Determine the coordinates of the intersection rectangle
+    x_left = max(box1[0], box2[0])
+    y_top = max(box1[1], box2[1])
+    x_right = min(box1[2], box2[2])
+    y_bottom = min(box1[3], box2[3])
+
+    # Compute the area of intersection
+    intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top)
+
+    # Compute the area of both bounding boxes
+    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
+    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
+
+    # Compute the area of the union
+    union_area = box1_area + box2_area - intersection_area
+
+    # Compute the Intersection over Union
+    iou = intersection_area / union_area
+
+    return iou
+
+
+def compute_accuracy(box1, box2, threshold=0.5):
+    """
+    Compute the accuracy of two bounding boxes based on a specified threshold.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - threshold (float): Threshold for the IoU to consider the prediction correct.
+
+    Returns:
+    - float: Accuracy of the prediction based on the IoU threshold.
+    """
+    iou = compute_iou(box1, box2)
+    return iou >= threshold
+
+
+def compute_center_accuracy(box1, box2):
+    """
+    Compute if the center point of box 2 is within box 1.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - bool: True if the center point of box 2 is within box 1, False otherwise.
+    """
+    # Compute the center point of box 2
+    center_x = (box2[0] + box2[2]) / 2
+    center_y = (box2[1] + box2[3]) / 2
+
+    # Check if the center point is within box 1
+    return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3]
+
+
+def convert_bbox(bbox, image_path):
+    new_bbox = bbox if isinstance(bbox, list) else ast.literal_eval(bbox)
+    new_bbox = [
+        new_bbox[0],
+        new_bbox[1],
+        new_bbox[0] + new_bbox[2],
+        new_bbox[1] + new_bbox[3],
+    ]
+    image = Image.open(image_path)
+    img_size = image.size
+    new_bbox = [
+        new_bbox[0] / img_size[0],
+        new_bbox[1] / img_size[1],
+        new_bbox[2] / img_size[0],
+        new_bbox[3] / img_size[1],
+    ]
+    return new_bbox
+
+
+class ScreenSpot(ImageBaseDataset):
+    MODALITY = "IMAGE"
+    TYPE = "GUI"
+    DATASET_URL = {
+        "ScreenSpot_Mobile": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_Mobile.tsv",  # noqa
+        "ScreenSpot_Desktop": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_Desktop.tsv",  # noqa
+        "ScreenSpot_Web": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot/ScreenSpot_Web.tsv",  # noqa
+        "ScreenSpot_v2_Mobile": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_v2/ScreenSpot_v2_Mobile.tsv",  # noqa
+        "ScreenSpot_v2_Desktop": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_v2/ScreenSpot_v2_Desktop.tsv",  # noqa
+        "ScreenSpot_v2_Web": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_v2/ScreenSpot_v2_Web.tsv",  # noqa
+    }  # path
+    DATASET_URL_V2 = {
+        "ScreenSpot_Mobile": "$WORK_DIR/screenspot_mobile_ug.json",
+        "ScreenSpot_Desktop": "$WORK_DIR/screenspot_desktop_ug.json",
+        "ScreenSpot_Web": "$WORK_DIR/screenspot_web_ug.json",
+    }  # path
+    DATASET_MD5 = {
+        "ScreenSpot_Mobile": "a5b5299843a75c9b9574c47bc13b2c53",
+        "ScreenSpot_Desktop": "e6e7bac21b6b2475276404fce2458132",
+        "ScreenSpot_Web": "e51d168c14b8582427cf3107d236cfc5",
+        "ScreenSpot_v2_Mobile": "234c858ab4f0e787e8388a73df65a4b7",
+        "ScreenSpot_v2_Desktop": "5f2aa2a497327bd33b2512a0c75cf994",
+        "ScreenSpot_v2_Web": "01cd0877ee1b735a6d5190b053ba9482",
+    }
+    EVAL_TYPE = "point"  # point or rectangle
+    RE_TYPE = "functional"  # type of referring expressions: functional or composite
+
+    def __init__(
+        self,
+        dataset="ScreenSpot_Mobile",
+        skip_noimg=True,
+        skeleton=False,
+        re_type="functional",
+    ):
+        # st()
+        ROOT = LMUDataRoot()
+        # You can override this variable to save image files to a different directory
+        self.dataset_name = dataset
+        self.img_root = osp.join(ROOT, "images", self.dataset_name)
+        self.RE_TYPE = re_type
+        if skeleton:
+            return
+
+        data = self.load_data(dataset)
+        self.skip_noimg = skip_noimg
+        if skip_noimg and "image" in data:
+            data = data[~pd.isna(data["image"])]
+
+        self.meta_only = True
+        self.parse_response_func = parse_bbox_aguvis  # TODO: parse function can be specified through kwargs when initializing the dataset # noqa: E501
+
+        # The image field can store the base64 encoded image or another question index (for saving space)
+        if "image" in data:
+            data["image"] = [str(x) for x in data["image"]]
+            image_map = {x: y for x, y in zip(data["index"], data["image"])}
+            for k in image_map:
+                if len(image_map[k]) <= 64:
+                    idx = image_map[k]
+                    assert idx in image_map and len(image_map[idx]) > 64
+                    image_map[k] = image_map[idx]
+
+            images = [toliststr(image_map[k]) for k in data["index"]]
+            data["image"] = [x[0] if len(x) == 1 else x for x in images]
+            self.meta_only = False
+
+        self.data = data
+
+    def prepare_tsv(self, url, file_md5=None):
+        # st()
+        if self.RE_TYPE == "functional":
+            return super().prepare_tsv(url=url, file_md5=file_md5)
+        else:
+            data_path = self.DATASET_URL_V2[self.dataset_name]
+        return pd.DataFrame(load(data_path))
+
+    @classmethod
+    def get_action_space(self):
+        return ""
+
+    @classmethod
+    def get_trajectory(self, line):
+        traj_dict = {}
+        if self.RE_TYPE == "functional":
+            traj_dict["task"] = line["question"]
+        else:
+            traj_dict["task"] = line["description"]
+        return traj_dict
+
+    def build_prompt(self, line):
+        # st()
+        if isinstance(line, int):
+            line = self.data.iloc[line]
+        tgt_path = self.dump_image(line)
+
+        if self.RE_TYPE == "functional":
+            user_instruction = USER_INSTRUCTION.format(instruction=line["question"])
+        else:
+            user_instruction = USER_INSTRUCTION_V2.format(
+                description=line["description"]
+            )
+
+        msgs = []
+        # add system prompt
+        if self.RE_TYPE == "functional":
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT))
+        else:
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT_V2))
+        if isinstance(tgt_path, list):
+            msgs.extend([dict(type="image", value=p) for p in tgt_path])
+        else:
+            msgs = [dict(type="image", value=tgt_path)]
+        msgs.append(dict(type="text", value=user_instruction))
+        return msgs
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        # st()
+        if self.EVAL_TYPE == "point":
+            return self.evaluate_point(eval_file, **judge_kwargs)
+
+        elif self.EVAL_TYPE == "rectangle":
+            return self.evaluate_rectangle(eval_file, **judge_kwargs)
+
+    def evaluate_rectangle(self, eval_file, **judge_kwargs):
+        scorers = {
+            "IoU": compute_iou,
+            "ACC@0.1": lambda x, y: compute_accuracy(x, y, 0.1),
+            "ACC@0.3": lambda x, y: compute_accuracy(x, y, 0.3),
+            "ACC@0.5": lambda x, y: compute_accuracy(x, y, 0.5),
+            "ACC@0.7": lambda x, y: compute_accuracy(x, y, 0.7),
+            "ACC@0.9": lambda x, y: compute_accuracy(x, y, 0.9),
+            "Center_ACC": compute_center_accuracy,
+        }
+        results_dict = {}
+        for key in scorers.keys():
+            results_dict.update(
+                {
+                    key: [],
+                    key + "_text": [],
+                    key + "_icon": [],
+                }
+            )
+
+        result = []
+        data = load(eval_file)
+
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = convert_bbox(
+                line["bbox"], os.path.join(self.img_root, line["image_path"])
+            )
+            prediction = str(line["prediction"])
+            try:
+                click_point = parse_bbox_aguvis(prediction)
+
+                match = {}
+                for score_key, score_value in scorers.items():
+                    score = score_value(bbox, click_point)
+                    if score_key != "IoU":
+                        match[score_key.replace("ACC", "match")] = score
+                    results_dict[score_key].append(score)
+                    if line["data_type"] == "text":
+                        results_dict[score_key + "_text"].append(score)
+                    else:
+                        results_dict[score_key + "_icon"].append(score)
+            except Exception:
+                click_point = None
+                match = {score_key: False for score_key in scorers.keys() if score_key != "IoU"}
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["data_type"],
+                    "source": line["data_source"],
+                    "pred": click_point,
+                    "num_matched": sum(match.values()),
+                    **match,
+                }
+            )
+        for key in results_dict:
+            if len(results_dict[key]) == 0:
+                results_dict[key] = str(0)
+            else:
+                results_dict[key] = str(sum(results_dict[key]) / len(results_dict[key]))
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(results_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            failure_cases = [res for res in result if not res["match"] and res["is_wrong_format"]]
+            failure_cases.sort(key=lambda r: r["num_matched"], reverse=True)
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+        return results_dict
+
+    def evaluate_point(self, eval_file, **judge_kwargs):
+        # -1: format_err, 0: wrong, 1: correct
+        stats = defaultdict(list)
+        # Will include instance-level results
+        result = []
+
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = (
+                line["bbox"]
+                if isinstance(line["bbox"], list)
+                else ast.literal_eval(line["bbox"])
+            )
+            # The format of bbox is (x1, y1, w, h)
+            x1, y1, w, h = bbox
+            bbox = (x1, y1, x1 + w - 1, y1 + h - 1)
+
+            image = Image.open(os.path.join(self.img_root, line["image_path"]))
+            img_size = image.size
+
+            def make_safe(value):
+                if value == -1:
+                    # we can tolerate -1 as a special value and nomalize it to 0
+                    return 0
+                else:
+                    return value
+
+            bbox = [
+                make_safe(bbox[0]) / img_size[0],
+                make_safe(bbox[1]) / img_size[1],
+                make_safe(bbox[2]) / img_size[0],
+                make_safe(bbox[3]) / img_size[1],
+            ]
+
+            if any([x < 0 or x > 1 for x in bbox]):
+                raise ValueError(f"bbox out of range: {bbox} | {line['bbox']} | {img_size}")
+
+            key = line['data_type'] if 'category' not in line else line['category'] + ":" + line['data_type']
+            prediction = str(line["prediction"])
+            try:
+                click_point = parse_bbox_aguvis(prediction)
+                # Do Normalization By Default
+                if click_point[0] > 1 or click_point[1] > 1:
+                    click_point = (click_point[0] / img_size[0], click_point[1] / img_size[1])
+
+                match = (bbox[0] <= click_point[0] <= bbox[2]) and \
+                    (bbox[1] <= click_point[1] <= bbox[3])
+
+                if match:
+                    stats[key].append(1)
+                else:
+                    stats[key].append(0)
+                is_wrong_format = False
+
+            except Exception as e:
+                logger.warning(f"exception in screenspot eval:{e}")
+                stats[key].append(-1)
+                match, is_wrong_format, click_point = False, True, None
+
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["data_type"],
+                    "source": line["data_source"],
+                    "match": match,
+                    "is_wrong_format": is_wrong_format,
+                    "pred": click_point,
+                }
+            )
+
+        final_score_dict = {}
+        # Record the number of each category
+        final_score_dict.update({k + ':cnt': len(stats[k]) for k in stats})
+        # Calculate the Overall stats
+        full_stats = []
+        for v in stats.values():
+            full_stats.extend(v)
+        final_score_dict['Overall_Accuracy'] = np.mean([x > 0 for x in full_stats]) * 100
+        final_score_dict['Format_Err_Rate'] = np.mean([x < 0 for x in full_stats]) * 100
+        # Calculate the Accuracy of Text / Icon
+        text_stats = [v for k, v in stats.items() if k.endswith('text') for x in v]
+        text_stats = itertools.chain(*text_stats)
+        final_score_dict['Text_Accuracy'] = np.mean([x > 0 for x in text_stats]) * 100
+        icon_stats = [v for k, v in stats.items() if k.endswith('icon') for x in v]
+        icon_stats = itertools.chain(*icon_stats)
+        final_score_dict['Icon_Accuracy'] = np.mean([x > 0 for x in icon_stats]) * 100
+        # Calculate the Accuracy of Each Category
+        if 'category' in data:
+            cates = list(set(data['category']))
+            for c in cates:
+                sub_stats = [v for k, v in stats.items() if k.split(":")[0] == c for x in v]
+                sub_stats = itertools.chain(*sub_stats)
+                final_score_dict[c + '_Accuracy'] = np.mean([x > 0 for x in sub_stats]) * 100
+
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(final_score_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            def click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                w, h = x2 - x1, y2 - y1
+                abs_shift_to_center = [abs(x - xc), abs(y - yc)]  # noqa: E501
+                width_outside, height_outside = [max(0, abs_shift_to_center[0] - w / 2), max(0, abs_shift_to_center[1] - h / 2)]  # noqa: E501
+                return (width_outside ** 2 + height_outside ** 2) ** 0.5  # noqa: E501
+
+            wrong_format_result = [res for res in result if res["is_wrong_format"]]
+            missed_result = [res for res in result if not res["match"] and not res["is_wrong_format"]]
+            missed_result.sort(key=lambda r: click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            failure_cases = wrong_format_result + missed_result
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+        return final_score_dict

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/GUI/screenspot_pro.py

@@ -0,0 +1,461 @@
+import ast
+import itertools
+import json
+import os
+import os.path as osp
+import re
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+from PIL import Image
+from tqdm import tqdm
+
+from vlmeval.dataset.image_base import ImageBaseDataset
+from vlmeval.smp import LMUDataRoot, dump, get_intermediate_file_path, get_logger, load, toliststr
+
+logger = get_logger(__name__)
+
+"""
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [42, 1102, 197, 70],
+    "question": "view the details of the item",
+    "data_type": "text",
+    "data_source": "shop"
+},
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [93, 74, 86, 132],
+    "question": "view the previous photo",
+    "data_type": "icon",
+    "data_source": "shop"
+}
+"""
+
+SYSTEM_PROMPT = """You are a GUI agent. You are given a task and a screenshot of the screen. You need to perform pyautogui click/moveTo action to complete the task. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+
+USER_INSTRUCTION = """Please complete the following tasks by clicking using `pyautogui.click`:\n{instruction}"""  # noqa: E501
+
+SYSTEM_PROMPT_V2 = """You are a GUI agent. You are given a screenshot of the screen and the description of a target element. You need to click the target element using `pyautogui.click`. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+USER_INSTRUCTION_V2 = """Please click the following target element using `pyautogui.click`:\n{description}"""
+
+
+def parse_bbox_aguvis(response):
+    match = re.search(r"x=([\d.]+), y=([\d.]+)", response)
+    if match:
+        click_point = [float(match.group(1)), float(match.group(2))]
+    else:
+        click_point = [0.0, 0.0]
+    return click_point
+
+
+def compute_iou(box1, box2):
+    """
+    Compute the Intersection over Union (IoU) of two bounding boxes.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - float: IoU of box1 and box2.
+    """
+    # Determine the coordinates of the intersection rectangle
+    x_left = max(box1[0], box2[0])
+    y_top = max(box1[1], box2[1])
+    x_right = min(box1[2], box2[2])
+    y_bottom = min(box1[3], box2[3])
+
+    # Compute the area of intersection
+    intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top)
+
+    # Compute the area of both bounding boxes
+    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
+    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
+
+    # Compute the area of the union
+    union_area = box1_area + box2_area - intersection_area
+
+    # Compute the Intersection over Union
+    iou = intersection_area / union_area
+
+    return iou
+
+
+def compute_accuracy(box1, box2, threshold=0.5):
+    """
+    Compute the accuracy of two bounding boxes based on a specified threshold.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - threshold (float): Threshold for the IoU to consider the prediction correct.
+
+    Returns:
+    - float: Accuracy of the prediction based on the IoU threshold.
+    """
+    iou = compute_iou(box1, box2)
+    return iou >= threshold
+
+
+def compute_center_accuracy(box1, box2):
+    """
+    Compute if the center point of box 2 is within box 1.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - bool: True if the center point of box 2 is within box 1, False otherwise.
+    """
+    # Compute the center point of box 2
+    center_x = (box2[0] + box2[2]) / 2
+    center_y = (box2[1] + box2[3]) / 2
+
+    # Check if the center point is within box 1
+    return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3]
+
+
+def convert_bbox(bbox, image_path, convert_xywh_to_x1y1x2y2=True):
+    new_bbox = bbox if isinstance(bbox, list) else ast.literal_eval(bbox)
+    if convert_xywh_to_x1y1x2y2:
+        new_bbox = [
+            new_bbox[0],
+            new_bbox[1],
+            new_bbox[0] + new_bbox[2],
+            new_bbox[1] + new_bbox[3],
+        ]
+    image = Image.open(image_path)
+    img_size = image.size
+    new_bbox = [
+        new_bbox[0] / img_size[0],
+        new_bbox[1] / img_size[1],
+        new_bbox[2] / img_size[0],
+        new_bbox[3] / img_size[1],
+    ]
+    return new_bbox
+
+
+class ScreenSpot_Pro(ImageBaseDataset):
+    MODALITY = "IMAGE"
+    TYPE = "GUI"
+    DATASET_URL = {
+        "ScreenSpot_Pro_Development": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_Pro/ScreenSpot_Pro_Development.tsv",  # noqa
+        "ScreenSpot_Pro_Creative": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_Pro/ScreenSpot_Pro_Creative.tsv",  # noqa
+        "ScreenSpot_Pro_CAD": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_Pro/ScreenSpot_Pro_CAD.tsv",  # noqa
+        "ScreenSpot_Pro_Scientific": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_Pro/ScreenSpot_Pro_Scientific.tsv",  # noqa
+        "ScreenSpot_Pro_Office": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_Pro/ScreenSpot_Pro_Office.tsv",  # noqa
+        "ScreenSpot_Pro_OS": "https://opencompass.openxlab.space/utils/benchmarks/GUI/ScreenSpot_Pro/ScreenSpot_Pro_OS.tsv",  # noqa
+    }  # path
+    DATASET_MD5 = {
+        'ScreenSpot_Pro_Development': '45b93df1d5814885011d682fe1b0f959',
+        'ScreenSpot_Pro_Creative': 'a15867fee82ba8cd95581895c55f03cd',
+        'ScreenSpot_Pro_CAD': '0faa3bc29eba359766c3a7ca2c4d8917',
+        'ScreenSpot_Pro_Scientific': 'edc2e1f2b53af5fff6480b77c4986b81',
+        'ScreenSpot_Pro_Office': '8756c128cf567274c2647423ccc4eaf0',
+        'ScreenSpot_Pro_OS': '49c3eaaa7df6d22475c39120fe8f1c06'
+    }
+    EVAL_TYPE = "point"  # point or rectangle
+    RE_TYPE = "functional"  # type of referring expressions: functional or composite
+
+    def __init__(
+        self,
+        dataset="ScreenSpot_Pro_Development",
+        skip_noimg=True,
+        skeleton=False,
+        re_type="functional",
+    ):
+        # st()
+        ROOT = LMUDataRoot()
+        # You can override this variable to save image files to a different directory
+        self.dataset_name = dataset
+        self.img_root = osp.join(ROOT, "images", self.dataset_name)
+        self.RE_TYPE = re_type
+        if skeleton:
+            return
+
+        data = self.load_data(dataset)
+        self.skip_noimg = skip_noimg
+        if skip_noimg and "image" in data:
+            data = data[~pd.isna(data["image"])]
+
+        data["index"] = [str(idx + 1) for idx, x in enumerate(data["bbox"])]
+
+        self.meta_only = True
+        self.parse_response_func = parse_bbox_aguvis  # TODO: parse function can be specified through kwargs when initializing the dataset # noqa: E501
+
+        # The image field can store the base64 encoded image or another question index (for saving space) # noqa: E501
+        if "image" in data:
+            data["image"] = [str(x) for x in data["image"]]
+            image_map = {x: y for x, y in zip(data["index"], data["image"])}
+            for k in image_map:
+                if len(image_map[k]) <= 64:
+                    idx = image_map[k]
+                    assert idx in image_map and len(image_map[idx]) > 64
+                    image_map[k] = image_map[idx]
+
+            images = [toliststr(image_map[k]) for k in data["index"]]
+            data["image"] = [x[0] if len(x) == 1 else x for x in images]
+            self.meta_only = False
+
+        self.data = data
+
+    @classmethod
+    def get_action_space(self):
+        return ""
+
+    @classmethod
+    def get_trajectory(self, line):
+        traj_dict = {}
+        if self.RE_TYPE == "functional":
+            traj_dict["task"] = line["question"]
+        else:
+            traj_dict["task"] = line["description"]
+        return traj_dict
+
+    def build_prompt(self, line):
+        if isinstance(line, int):
+            line = self.data.iloc[line]
+        tgt_path = self.dump_image(line)
+
+        if self.RE_TYPE == "functional":
+            user_instruction = USER_INSTRUCTION.format(instruction=line["question"])
+        else:
+            user_instruction = USER_INSTRUCTION_V2.format(
+                description=line["description"]
+            )
+
+        msgs = []
+        # add system prompt
+        if self.RE_TYPE == "functional":
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT))
+        else:
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT_V2))
+        if isinstance(tgt_path, list):
+            msgs.extend([dict(type="image", value=p) for p in tgt_path])
+        else:
+            msgs = [dict(type="image", value=tgt_path)]
+        msgs.append(dict(type="text", value=user_instruction))
+        return msgs
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        # st()
+        if self.EVAL_TYPE == "point":
+            return self.evaluate_point(eval_file, **judge_kwargs)
+
+        elif self.EVAL_TYPE == "rectangle":
+            return self.evaluate_rectangle(eval_file, **judge_kwargs)
+
+    def evaluate_rectangle(self, eval_file, **judge_kwargs):
+        scorers = {
+            "IoU": compute_iou,
+            "ACC@0.1": lambda x, y: compute_accuracy(x, y, 0.1),
+            "ACC@0.3": lambda x, y: compute_accuracy(x, y, 0.3),
+            "ACC@0.5": lambda x, y: compute_accuracy(x, y, 0.5),
+            "ACC@0.7": lambda x, y: compute_accuracy(x, y, 0.7),
+            "ACC@0.9": lambda x, y: compute_accuracy(x, y, 0.9),
+            "Center_ACC": compute_center_accuracy,
+        }
+        results_dict = {}
+        for key in scorers.keys():
+            results_dict.update(
+                {
+                    key: [],
+                    key + "_text": [],
+                    key + "_icon": [],
+                }
+            )
+
+        result = []
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = convert_bbox(
+                line["bbox"], os.path.join(self.img_root, line["image_path"]), convert_xywh_to_x1y1x2y2=False
+            )
+            prediction = str(line["prediction"])
+            try:
+                click_point = parse_bbox_aguvis(prediction)
+
+                match = {}
+                for score_key, score_value in scorers.items():
+                    score = score_value(bbox, click_point)
+                    if score_key != "IoU":
+                        match[score_key.replace("ACC", "match")] = score
+                    results_dict[score_key].append(score)
+                    if line["ui_type"] == "text":
+                        results_dict[score_key + "_text"].append(score)
+                    else:
+                        results_dict[score_key + "_icon"].append(score)
+            except Exception:
+                click_point = None
+                match = {score_key: False for score_key in scorers.keys() if score_key != "IoU"}
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "source": line["application"],
+                    "pred": click_point,
+                    "num_matched": sum(match.values()),
+                    **match,
+                }
+            )
+        for key in results_dict:
+            if len(results_dict[key]) == 0:
+                results_dict[key] = str(0)
+            else:
+                results_dict[key] = str(sum(results_dict[key]) / len(results_dict[key]))
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(results_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            failure_cases = [res for res in result if not res["match"] and res["is_wrong_format"]]
+            failure_cases.sort(key=lambda r: r["num_matched"], reverse=True)
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+        return results_dict
+
+    def evaluate_point(self, eval_file, **judge_kwargs):
+        # -1: format_err, 0: wrong, 1: correct
+        stats = defaultdict(list)
+        # Will include instance-level results
+        result = []
+
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = (
+                line["bbox"]
+                if isinstance(line["bbox"], list)
+                else ast.literal_eval(line["bbox"])
+            )
+            # The format of bbox is (x1, y1, x2, y2)
+
+            image = Image.open(os.path.join(self.img_root, line["image_path"]))
+            img_size = image.size
+
+            def make_safe(value):
+                if value == -1:
+                    # we can tolerate -1 as a special value and nomalize it to 0
+                    return 0
+                else:
+                    return value
+
+            bbox = [
+                make_safe(bbox[0]) / img_size[0],
+                make_safe(bbox[1]) / img_size[1],
+                make_safe(bbox[2]) / img_size[0],
+                make_safe(bbox[3]) / img_size[1],
+            ]
+
+            if any([x < 0 or x > 1 for x in bbox]):
+                raise ValueError(f"bbox out of range: {bbox} | {line['bbox']} | {img_size}")
+
+            key = line["category"] + ":" + line['ui_type']
+            prediction = str(line["prediction"])
+            try:
+                click_point = self.parse_response_func(prediction)
+                # Do Normalization By Default
+                if click_point[0] > 1 or click_point[1] > 1:
+                    click_point = (click_point[0] / img_size[0], click_point[1] / img_size[1])
+
+                match = (bbox[0] <= click_point[0] <= bbox[2]) and \
+                    (bbox[1] <= click_point[1] <= bbox[3])
+
+                if match:
+                    stats[key].append(1)
+                else:
+                    stats[key].append(0)
+                is_wrong_format = False
+
+            except Exception as e:
+                logger.warning(f"exception in screenspot eval:{e}")
+                stats[key].append(-1)
+                match, is_wrong_format, click_point = False, True, None
+
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "source": line["application"],
+                    "match": match,
+                    "is_wrong_format": is_wrong_format,
+                    "pred": click_point,
+                }
+            )
+
+        final_score_dict = {}
+        # Record the number of each category
+        final_score_dict.update({k + ':cnt': len(stats[k]) for k in stats})
+        # Calculate the Overall stats
+        full_stats = []
+        for v in stats.values():
+            full_stats.extend(v)
+        final_score_dict['Overall_Accuracy'] = np.mean([x > 0 for x in full_stats]) * 100
+        final_score_dict['Format_Err_Rate'] = np.mean([x < 0 for x in full_stats]) * 100
+        # Calculate the Accuracy of Text / Icon
+        text_stats = [v for k, v in stats.items() if k.split(":")[1] == "text" for x in v]
+        text_stats = itertools.chain(*text_stats)
+        final_score_dict['Text_Accuracy'] = np.mean([x > 0 for x in text_stats]) * 100
+        icon_stats = [v for k, v in stats.items() if k.split(":")[1] == "icon" for x in v]
+        icon_stats = itertools.chain(*icon_stats)
+        final_score_dict['Icon_Accuracy'] = np.mean([x > 0 for x in icon_stats]) * 100
+        # Calculate the Accuracy of Each Category
+        cates = list(set(data['category']))
+        for c in cates:
+            sub_stats = [v for k, v in stats.items() if k.split(":")[0] == c for x in v]
+            sub_stats = itertools.chain(*sub_stats)
+            final_score_dict[c + '_Accuracy'] = np.mean([x > 0 for x in sub_stats]) * 100
+
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(final_score_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            def click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                w, h = x2 - x1, y2 - y1
+                abs_shift_to_center = [abs(x - xc), abs(y - yc)]  # noqa: E501
+                width_outside, height_outside = [max(0, abs_shift_to_center[0] - w / 2), max(0, abs_shift_to_center[1] - h / 2)]  # noqa: E501
+                return (width_outside ** 2 + height_outside ** 2) ** 0.5  # noqa: E501
+
+            wrong_format_result = [res for res in result if res["is_wrong_format"]]
+            missed_result = [res for res in result if not res["match"] and not res["is_wrong_format"]]
+            missed_result.sort(key=lambda r: click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            failure_cases = wrong_format_result + missed_result
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+
+        successful_cases_path = os.environ.get("SUCCESSFUL_CASES_PATH", None)
+        if successful_cases_path is not None:
+            def _click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                x_shift, y_shift = x - xc, y - yc
+                return (x_shift ** 2 + y_shift ** 2) ** 0.5
+
+            successful_cases = [res for res in result if res["match"]]
+            successful_cases.sort(key=lambda r: _click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            with open(successful_cases_path, "w") as f:
+                json.dump(successful_cases, f, indent=4, ensure_ascii=False)
+        return final_score_dict

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/GUI/screenspot_v2.py

@@ -0,0 +1,203 @@
+import ast
+import os.path as osp
+import re
+
+import pandas as pd
+from PIL import Image
+
+from vlmeval.smp import LMUDataRoot, get_logger, load, toliststr
+from .screenspot import ScreenSpot
+
+logger = get_logger(__name__)
+
+"""
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [42, 1102, 197, 70],
+    "instruction": "view the details of the item",
+    "data_type": "text",
+    "data_source": "shop"
+},
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [93, 74, 86, 132],
+    "instruction": "view the previous photo",
+    "data_type": "icon",
+    "data_source": "shop"
+}
+"""
+
+SYSTEM_PROMPT = """You are a GUI agent. You are given a task and a screenshot of the screen. You need to perform pyautogui click/moveTo action to complete the task. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+
+USER_INSTRUCTION = """Please complete the following tasks by clicking using `pyautogui.click`:\n{instruction}"""  # noqa: E501
+
+SYSTEM_PROMPT_V2 = """You are a GUI agent. You are given a screenshot of the screen and the description of a target element. You need to click the target element using `pyautogui.click`. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+USER_INSTRUCTION_V2 = """Please click the following target element using `pyautogui.click`:\n{description}"""
+
+
+def parse_bbox_aguvis(response):
+    match = re.search(r"x=([\d.]+), y=([\d.]+)", response)
+    if match:
+        click_point = [float(match.group(1)), float(match.group(2))]
+    else:
+        click_point = [0.0, 0.0]
+    return click_point
+
+
+def compute_iou(box1, box2):
+    """
+    Compute the Intersection over Union (IoU) of two bounding boxes.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - float: IoU of box1 and box2.
+    """
+    # Determine the coordinates of the intersection rectangle
+    x_left = max(box1[0], box2[0])
+    y_top = max(box1[1], box2[1])
+    x_right = min(box1[2], box2[2])
+    y_bottom = min(box1[3], box2[3])
+
+    # Compute the area of intersection
+    intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top)
+
+    # Compute the area of both bounding boxes
+    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
+    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
+
+    # Compute the area of the union
+    union_area = box1_area + box2_area - intersection_area
+
+    # Compute the Intersection over Union
+    iou = intersection_area / union_area
+
+    return iou
+
+
+def compute_accuracy(box1, box2, threshold=0.5):
+    """
+    Compute the accuracy of two bounding boxes based on a specified threshold.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - threshold (float): Threshold for the IoU to consider the prediction correct.
+
+    Returns:
+    - float: Accuracy of the prediction based on the IoU threshold.
+    """
+    iou = compute_iou(box1, box2)
+    return iou >= threshold
+
+
+def compute_center_accuracy(box1, box2):
+    """
+    Compute if the center point of box 2 is within box 1.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - bool: True if the center point of box 2 is within box 1, False otherwise.
+    """
+    # Compute the center point of box 2
+    center_x = (box2[0] + box2[2]) / 2
+    center_y = (box2[1] + box2[3]) / 2
+
+    # Check if the center point is within box 1
+    return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3]
+
+
+def convert_bbox(bbox, image_path):
+    new_bbox = bbox if isinstance(bbox, list) else ast.literal_eval(bbox)
+    new_bbox = [
+        new_bbox[0],
+        new_bbox[1],
+        new_bbox[0] + new_bbox[2],
+        new_bbox[1] + new_bbox[3],
+    ]
+    image = Image.open(image_path)
+    img_size = image.size
+    new_bbox = [
+        new_bbox[0] / img_size[0],
+        new_bbox[1] / img_size[1],
+        new_bbox[2] / img_size[0],
+        new_bbox[3] / img_size[1],
+    ]
+    return new_bbox
+
+
+class ScreenSpotV2(ScreenSpot):
+    MODALITY = "IMAGE"
+    TYPE = "GUI"
+    DATASET_URL = {
+        "ScreenSpot_v2_Mobile": "ScreenSpot_v2_Mobile.tsv",
+        "ScreenSpot_v2_Desktop": "ScreenSpot_v2_Desktop.tsv",
+        "ScreenSpot_v2_Web": "ScreenSpot_v2_Web.tsv",
+    }  # path
+    DATASET_MD5 = {}
+    EVAL_TYPE = "point"  # point or rectangle
+    RE_TYPE = "functional"  # type of referring expressions: functional or composite
+
+    def __init__(
+        self,
+        dataset="ScreenSpot_Mobile",
+        skip_noimg=True,
+        skeleton=False,
+        re_type="functional",
+    ):
+        # st()
+        ROOT = LMUDataRoot()
+        # You can override this variable to save image files to a different directory
+        self.dataset_name = dataset
+        self.img_root = osp.join(ROOT, "ScreenSpot_v2", "screenspotv2_image")
+        self.RE_TYPE = re_type
+        if skeleton:
+            return
+
+        data = self.load_data(dataset)
+        self.skip_noimg = skip_noimg
+        if skip_noimg and "image" in data:
+            data = data[~pd.isna(data["image"])]
+
+        data["index"] = [str(idx + 1) for idx, x in enumerate(data["bbox"])]
+
+        self.meta_only = True
+        self.parse_response_func = parse_bbox_aguvis  # TODO: parse function can be specified through kwargs when initializing the dataset # noqa: E501
+
+        # The image field can store the base64 encoded image or another question index (for saving space)
+        if "image" in data:
+            data["image"] = [str(x) for x in data["image"]]
+            image_map = {x: y for x, y in zip(data["index"], data["image"])}
+            for k in image_map:
+                if len(image_map[k]) <= 64:
+                    idx = image_map[k]
+                    assert idx in image_map and len(image_map[idx]) > 64
+                    image_map[k] = image_map[idx]
+
+            images = [toliststr(image_map[k]) for k in data["index"]]
+            data["image"] = [x[0] if len(x) == 1 else x for x in images]
+            self.meta_only = False
+
+        if "img_filename" in data:
+            paths = [toliststr(x) for x in data["img_filename"]]
+            data["image_path"] = [x[0] if len(x) == 1 else x for x in paths]
+
+        # if np.all([istype(x, int) for x in data["index"]]):
+        #     data["index"] = [int(x) for x in data["index"]]
+
+        self.data = data
+        self.post_build(dataset)
+
+    def prepare_tsv(self, url, file_md5=None):
+        # st()
+        if self.RE_TYPE == "functional":
+            data_root = LMUDataRoot()
+            data_path = osp.join(data_root, "ScreenSpot_v2", url)
+        else:
+            data_path = self.DATASET_URL_V2[self.dataset_name]
+        return pd.DataFrame(load(data_path))

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/GUI/vbgd.py

@@ -0,0 +1,447 @@
+import ast
+import itertools
+import json
+import os
+import os.path as osp
+import re
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+from PIL import Image
+from tqdm import tqdm
+
+from vlmeval.dataset.image_base import ImageBaseDataset
+from vlmeval.smp import LMUDataRoot, dump, get_intermediate_file_path, get_logger, load, toliststr
+
+logger = get_logger(__name__)
+
+"""
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [42, 1102, 197, 70],
+    "question": "view the details of the item",
+    "data_type": "text",
+    "data_source": "shop"
+},
+{
+    "img_filename": "web_3b0ad239-da6b-4f6f-8f12-f674dc90ff33.png",
+    "bbox": [93, 74, 86, 132],
+    "question": "view the previous photo",
+    "data_type": "icon",
+    "data_source": "shop"
+}
+"""
+
+SYSTEM_PROMPT = """You are a GUI agent. You are given a task and a screenshot of the screen. You need to perform pyautogui click/moveTo action to complete the task. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+
+USER_INSTRUCTION = """Please complete the following tasks by clicking using `pyautogui.click`:\n{instruction}"""  # noqa: E501
+
+SYSTEM_PROMPT_V2 = """You are a GUI agent. You are given a screenshot of the screen and the description of a target element. You need to click the target element using `pyautogui.click`. The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"""  # noqa: E501
+USER_INSTRUCTION_V2 = """Please click the following target element using `pyautogui.click`:\n{description}"""
+
+
+def parse_bbox_aguvis(response):
+    match = re.search(r"x=([\d.]+), y=([\d.]+)", response)
+    if match:
+        click_point = [float(match.group(1)), float(match.group(2))]
+    else:
+        click_point = [0.0, 0.0]
+    return click_point
+
+
+def compute_iou(box1, box2):
+    """
+    Compute the Intersection over Union (IoU) of two bounding boxes.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - float: IoU of box1 and box2.
+    """
+    # Determine the coordinates of the intersection rectangle
+    x_left = max(box1[0], box2[0])
+    y_top = max(box1[1], box2[1])
+    x_right = min(box1[2], box2[2])
+    y_bottom = min(box1[3], box2[3])
+
+    # Compute the area of intersection
+    intersection_area = max(0, x_right - x_left) * max(0, y_bottom - y_top)
+
+    # Compute the area of both bounding boxes
+    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
+    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
+
+    # Compute the area of the union
+    union_area = box1_area + box2_area - intersection_area
+
+    # Compute the Intersection over Union
+    iou = intersection_area / union_area
+
+    return iou
+
+
+def compute_accuracy(box1, box2, threshold=0.5):
+    """
+    Compute the accuracy of two bounding boxes based on a specified threshold.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - threshold (float): Threshold for the IoU to consider the prediction correct.
+
+    Returns:
+    - float: Accuracy of the prediction based on the IoU threshold.
+    """
+    iou = compute_iou(box1, box2)
+    return iou >= threshold
+
+
+def compute_center_accuracy(box1, box2):
+    """
+    Compute if the center point of box 2 is within box 1.
+
+    Parameters:
+    - box1 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+    - box2 (list of float): Bounding box [x_min, y_min, x_max, y_max].
+
+    Returns:
+    - bool: True if the center point of box 2 is within box 1, False otherwise.
+    """
+    # Compute the center point of box 2
+    center_x = (box2[0] + box2[2]) / 2
+    center_y = (box2[1] + box2[3]) / 2
+
+    # Check if the center point is within box 1
+    return box1[0] <= center_x <= box1[2] and box1[1] <= center_y <= box1[3]
+
+
+def convert_bbox(bbox, image_path, convert_xywh_to_x1y1x2y2=True):
+    new_bbox = bbox if isinstance(bbox, list) else ast.literal_eval(bbox)
+    if convert_xywh_to_x1y1x2y2:
+        new_bbox = [
+            new_bbox[0],
+            new_bbox[1],
+            new_bbox[0] + new_bbox[2],
+            new_bbox[1] + new_bbox[3],
+        ]
+    image = Image.open(image_path)
+    img_size = image.size
+    new_bbox = [
+        new_bbox[0] / img_size[0],
+        new_bbox[1] / img_size[1],
+        new_bbox[2] / img_size[0],
+        new_bbox[3] / img_size[1],
+    ]
+    return new_bbox
+
+
+class VBGD(ImageBaseDataset):
+    MODALITY = "IMAGE"
+    TYPE = "GUI"
+    DATASET_URL = {"VBGD": "https://huggingface.co/datasets/Zery/VBGD_Dataset/resolve/main/VBGD.tsv"}  # path
+    DATASET_MD5 = {"VBGD": "54615d8e27a93b3be13c71ddc09a6277"}
+    EVAL_TYPE = "point"  # point or rectangle
+    RE_TYPE = "functional"  # type of referring expressions: functional or composite
+
+    def __init__(
+        self,
+        dataset="VBGD_Development",
+        skip_noimg=True,
+        skeleton=False,
+        re_type="functional",
+    ):
+        # st()
+        ROOT = LMUDataRoot()
+        # You can override this variable to save image files to a different directory
+        self.dataset_name = dataset
+        self.img_root = osp.join(ROOT, "images", self.dataset_name)
+        self.RE_TYPE = re_type
+        if skeleton:
+            return
+
+        data = self.load_data(dataset)
+        self.skip_noimg = skip_noimg
+        if skip_noimg and "image" in data:
+            data = data[~pd.isna(data["image"])]
+
+        data["index"] = [str(idx + 1) for idx, x in enumerate(data["bbox"])]
+
+        self.meta_only = True
+        self.parse_response_func = parse_bbox_aguvis  # TODO: parse function can be specified through kwargs when initializing the dataset # noqa: E501
+
+        # The image field can store the base64 encoded image or another question index (for saving space) # noqa: E501
+        if "image" in data:
+            data["image"] = [str(x) for x in data["image"]]
+            image_map = {x: y for x, y in zip(data["index"], data["image"])}
+            for k in image_map:
+                if len(image_map[k]) <= 64:
+                    idx = image_map[k]
+                    assert idx in image_map and len(image_map[idx]) > 64
+                    image_map[k] = image_map[idx]
+
+            images = [toliststr(image_map[k]) for k in data["index"]]
+            data["image"] = [x[0] if len(x) == 1 else x for x in images]
+            self.meta_only = False
+
+        self.data = data
+
+    @classmethod
+    def get_action_space(self):
+        return ""
+
+    @classmethod
+    def get_trajectory(self, line):
+        traj_dict = {}
+        if self.RE_TYPE == "functional":
+            traj_dict["task"] = line["question"]
+        else:
+            traj_dict["task"] = line["description"]
+        return traj_dict
+
+    def build_prompt(self, line):
+        if isinstance(line, int):
+            line = self.data.iloc[line]
+        tgt_path = self.dump_image(line)
+
+        if self.RE_TYPE == "functional":
+            user_instruction = USER_INSTRUCTION.format(instruction=line["question"])
+        else:
+            user_instruction = USER_INSTRUCTION_V2.format(
+                description=line["description"]
+            )
+
+        msgs = []
+        # add system prompt
+        if self.RE_TYPE == "functional":
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT))
+        else:
+            msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT_V2))
+        if isinstance(tgt_path, list):
+            msgs.extend([dict(type="image", value=p) for p in tgt_path])
+        else:
+            msgs = [dict(type="image", value=tgt_path)]
+        msgs.append(dict(type="text", value=user_instruction))
+        return msgs
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        # st()
+        if self.EVAL_TYPE == "point":
+            return self.evaluate_point(eval_file, **judge_kwargs)
+
+        elif self.EVAL_TYPE == "rectangle":
+            return self.evaluate_rectangle(eval_file, **judge_kwargs)
+
+    def evaluate_rectangle(self, eval_file, **judge_kwargs):
+        scorers = {
+            "IoU": compute_iou,
+            "ACC@0.1": lambda x, y: compute_accuracy(x, y, 0.1),
+            "ACC@0.3": lambda x, y: compute_accuracy(x, y, 0.3),
+            "ACC@0.5": lambda x, y: compute_accuracy(x, y, 0.5),
+            "ACC@0.7": lambda x, y: compute_accuracy(x, y, 0.7),
+            "ACC@0.9": lambda x, y: compute_accuracy(x, y, 0.9),
+            "Center_ACC": compute_center_accuracy,
+        }
+        results_dict = {}
+        for key in scorers.keys():
+            results_dict.update(
+                {
+                    key: [],
+                    key + "_text": [],
+                    key + "_icon": [],
+                }
+            )
+
+        result = []
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = convert_bbox(
+                line["bbox"], os.path.join(self.img_root, line["image_path"]), convert_xywh_to_x1y1x2y2=False
+            )
+            prediction = str(line["prediction"])
+            try:
+                click_point = parse_bbox_aguvis(prediction)
+
+                match = {}
+                for score_key, score_value in scorers.items():
+                    score = score_value(bbox, click_point)
+                    if score_key != "IoU":
+                        match[score_key.replace("ACC", "match")] = score
+                    results_dict[score_key].append(score)
+                    if line["ui_type"] == "text":
+                        results_dict[score_key + "_text"].append(score)
+                    else:
+                        results_dict[score_key + "_icon"].append(score)
+            except Exception:
+                click_point = None
+                match = {score_key: False for score_key in scorers.keys() if score_key != "IoU"}
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "source": line["application"],
+                    "pred": click_point,
+                    "num_matched": sum(match.values()),
+                    **match,
+                }
+            )
+        for key in results_dict:
+            if len(results_dict[key]) == 0:
+                results_dict[key] = str(0)
+            else:
+                results_dict[key] = str(sum(results_dict[key]) / len(results_dict[key]))
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(results_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            failure_cases = [res for res in result if not res["match"] and res["is_wrong_format"]]
+            failure_cases.sort(key=lambda r: r["num_matched"], reverse=True)
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+        return results_dict
+
+    def evaluate_point(self, eval_file, **judge_kwargs):
+        # -1: format_err, 0: wrong, 1: correct
+        stats = defaultdict(list)
+        # Will include instance-level results
+        result = []
+
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = (
+                line["bbox"]
+                if isinstance(line["bbox"], list)
+                else ast.literal_eval(line["bbox"])
+            )
+            # The format of bbox is (x1, y1, x2, y2)
+
+            image = Image.open(os.path.join(self.img_root, line["image_path"]))
+            img_size = image.size
+
+            def make_safe(value):
+                if value == -1:
+                    # we can tolerate -1 as a special value and nomalize it to 0
+                    return 0
+                else:
+                    return value
+
+            bbox = [
+                make_safe(bbox[0]) / img_size[0],
+                make_safe(bbox[1]) / img_size[1],
+                make_safe(bbox[2]) / img_size[0],
+                make_safe(bbox[3]) / img_size[1],
+            ]
+
+            if any([x < 0 or x > 1 for x in bbox]):
+                raise ValueError(f"bbox out of range: {bbox} | {line['bbox']} | {img_size}")
+
+            key = line["category"] + ":" + line['ui_type']
+            prediction = str(line["prediction"])
+            try:
+                click_point = self.parse_response_func(prediction)
+                # Do Normalization By Default
+                if click_point[0] > 1 or click_point[1] > 1:
+                    click_point = (click_point[0] / img_size[0], click_point[1] / img_size[1])
+
+                match = (bbox[0] <= click_point[0] <= bbox[2]) and \
+                    (bbox[1] <= click_point[1] <= bbox[3])
+
+                if match:
+                    stats[key].append(1)
+                else:
+                    stats[key].append(0)
+                is_wrong_format = False
+
+            except Exception as e:
+                logger.warning(f"exception in screenspot eval:{e}")
+                stats[key].append(-1)
+                match, is_wrong_format, click_point = False, True, None
+
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "source": line["application"],
+                    "match": match,
+                    "is_wrong_format": is_wrong_format,
+                    "pred": click_point,
+                }
+            )
+
+        final_score_dict = {}
+        # Record the number of each category
+        final_score_dict.update({k + ':cnt': len(stats[k]) for k in stats})
+        # Calculate the Overall stats
+        full_stats = []
+        for v in stats.values():
+            full_stats.extend(v)
+        final_score_dict['Overall_Accuracy'] = np.mean([x > 0 for x in full_stats]) * 100
+        final_score_dict['Format_Err_Rate'] = np.mean([x < 0 for x in full_stats]) * 100
+        # Calculate the Accuracy of Text / Icon
+        text_stats = [v for k, v in stats.items() if k.split(":")[1] == "text" for x in v]
+        text_stats = itertools.chain(*text_stats)
+        final_score_dict['Text_Accuracy'] = np.mean([x > 0 for x in text_stats]) * 100
+        icon_stats = [v for k, v in stats.items() if k.split(":")[1] == "icon" for x in v]
+        icon_stats = itertools.chain(*icon_stats)
+        final_score_dict['Icon_Accuracy'] = np.mean([x > 0 for x in icon_stats]) * 100
+        # Calculate the Accuracy of Each Category
+        cates = list(set(data['category']))
+        for c in cates:
+            sub_stats = [v for k, v in stats.items() if k.split(":")[0] == c for x in v]
+            sub_stats = itertools.chain(*sub_stats)
+            final_score_dict[c + '_Accuracy'] = np.mean([x > 0 for x in sub_stats]) * 100
+
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(final_score_dict, score_pth)
+
+        failure_cases_path = os.environ.get("FAILURE_CASES_PATH", None)
+        if failure_cases_path is not None:
+            def click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                w, h = x2 - x1, y2 - y1
+                abs_shift_to_center = [abs(x - xc), abs(y - yc)]  # noqa: E501
+                width_outside, height_outside = [max(0, abs_shift_to_center[0] - w / 2), max(0, abs_shift_to_center[1] - h / 2)]  # noqa: E501
+                return (width_outside ** 2 + height_outside ** 2) ** 0.5  # noqa: E501
+
+            wrong_format_result = [res for res in result if res["is_wrong_format"]]
+            missed_result = [res for res in result if not res["match"] and not res["is_wrong_format"]]
+            missed_result.sort(key=lambda r: click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            failure_cases = wrong_format_result + missed_result
+
+            with open(failure_cases_path, "w") as f:
+                json.dump(failure_cases, f, indent=4, ensure_ascii=False)
+
+        successful_cases_path = os.environ.get("SUCCESSFUL_CASES_PATH", None)
+        if successful_cases_path is not None:
+            def _click_distance(bbox, click_point):
+                x, y = click_point
+                x1, y1, x2, y2 = bbox
+                xc, yc = (x1 + x2) / 2, (y1 + y2) / 2
+                x_shift, y_shift = x - xc, y - yc
+                return (x_shift ** 2 + y_shift ** 2) ** 0.5
+
+            successful_cases = [res for res in result if res["match"]]
+            successful_cases.sort(key=lambda r: _click_distance(r["parsed_bbox"], r["pred"]), reverse=True)
+            with open(successful_cases_path, "w") as f:
+                json.dump(successful_cases, f, indent=4, ensure_ascii=False)
+        return final_score_dict

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/GUI/venusbench.py

@@ -0,0 +1,186 @@
+import ast
+import os
+import os.path as osp
+import re
+from collections import defaultdict
+
+import numpy as np
+import pandas as pd
+from PIL import Image
+from tqdm import tqdm
+
+from vlmeval.dataset.image_base import ImageBaseDataset
+from vlmeval.smp import LMUDataRoot, dump, get_intermediate_file_path, get_logger, load, toliststr
+
+logger = get_logger(__name__)
+
+SYSTEM_PROMPT = "You are a GUI agent. You are given a task and a screenshot of the screen. " \
+    "You need to perform pyautogui click/moveTo action to complete the task. " \
+    "The answer format is `pyautogui.click(x=?, y=?), x and y is necessary`"
+
+USER_INSTRUCTION = "Please complete the following tasks by clicking using `pyautogui.click`:\n{instruction}"
+
+
+def parse_bbox_aguvis(response):
+    match = re.search(r"x=([\d.]+), y=([\d.]+)", response)
+    if match:
+        click_point = [float(match.group(1)), float(match.group(2))]
+    else:
+        click_point = [0.0, 0.0]
+    return click_point
+
+
+class VenusBench_GD(ImageBaseDataset):
+    MODALITY = "IMAGE"
+    TYPE = "GUI"
+    DATASET_URL = {
+        "VenusBench-GD": "https://huggingface.co/datasets/Zery/VBGD_Dataset/resolve/main/VenusBench.tsv",
+    }
+    DATASET_MD5 = {
+        'VenusBench-GD': '6a2fe92d3ecf5a3b6503a1fe4891c5ea'
+    }
+
+    def __init__(
+        self,
+        dataset="VenusBench-GD",
+        skip_noimg=True,
+        skeleton=False,
+    ):
+        ROOT = LMUDataRoot()
+        self.dataset_name = dataset
+        self.img_root = osp.join(ROOT, "images", self.dataset_name)
+
+        if skeleton:
+            return
+
+        data = self.load_data(dataset)
+        self.skip_noimg = skip_noimg
+        if skip_noimg and "image" in data:
+            data = data[~pd.isna(data["image"])]
+
+        # Verify we have index properly
+        if "index" not in data:
+            data["index"] = [str(idx + 1) for idx in range(len(data))]
+
+        self.meta_only = True
+        self.parse_response_func = parse_bbox_aguvis
+
+        if "image" in data:
+            data["image"] = [str(x) for x in data["image"]]
+            image_map = {x: y for x, y in zip(data["index"], data["image"])}
+            for k in image_map:
+                if len(image_map[k]) <= 64:
+                    idx = image_map[k]
+                    assert idx in image_map and len(image_map[idx]) > 64
+                    image_map[k] = image_map[idx]
+
+            images = [toliststr(image_map[k]) for k in data["index"]]
+            data["image"] = [x[0] if len(x) == 1 else x for x in images]
+            self.meta_only = False
+
+        self.data = data
+
+    @classmethod
+    def get_action_space(self):
+        return ""
+
+    @classmethod
+    def get_trajectory(self, line):
+        traj_dict = {}
+        traj_dict["task"] = line["question"]
+        return traj_dict
+
+    def build_prompt(self, line):
+        if isinstance(line, int):
+            line = self.data.iloc[line]
+        tgt_path = self.dump_image(line)
+        user_instruction = USER_INSTRUCTION.format(instruction=line["question"])
+        msgs = []
+        msgs.append(dict(role="system", type="text", value=SYSTEM_PROMPT))
+        if isinstance(tgt_path, list):
+            msgs.extend([dict(type="image", value=p) for p in tgt_path])
+        else:
+            msgs = [dict(type="image", value=tgt_path)]
+        msgs.append(dict(type="text", value=user_instruction))
+        return msgs
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        stats = defaultdict(list)
+        result = []
+
+        data = load(eval_file)
+        assert "bbox" in data and "prediction" in data
+        lt = len(data)
+        lines = [data.iloc[i] for i in range(lt)]
+
+        for i in tqdm(range(len(lines))):
+            line = lines[i]
+            bbox = (
+                line["bbox"]
+                if isinstance(line["bbox"], list)
+                else ast.literal_eval(line["bbox"])
+            )
+            # The format of bbox in VenusBench-GD is (x_min, y_min, x_max, y_max)
+            image = Image.open(os.path.join(self.img_root, line["image_path"]))
+            img_size = image.size
+
+            # Absolute to relative
+            bbox = [
+                bbox[0] / img_size[0],
+                bbox[1] / img_size[1],
+                bbox[2] / img_size[0],
+                bbox[3] / img_size[1],
+            ]
+
+            key = line["category"] + ":" + line['ui_type']
+            prediction = str(line["prediction"])
+            try:
+                click_point = self.parse_response_func(prediction)
+                if click_point[0] > 1 or click_point[1] > 1:
+                    click_point = (click_point[0] / img_size[0], click_point[1] / img_size[1])
+
+                match = (bbox[0] <= click_point[0] <= bbox[2]) and \
+                    (bbox[1] <= click_point[1] <= bbox[3])
+
+                if match:
+                    stats[key].append(1)
+                else:
+                    stats[key].append(0)
+                is_wrong_format = False
+            except Exception as e:
+                logger.warning(f"exception in venusbench eval:{e}")
+                stats[key].append(-1)
+                match, is_wrong_format, click_point = False, True, None
+
+            result.append(
+                {
+                    "img_path": os.path.join(self.img_root, line["image_path"]),
+                    "text": line["question"],
+                    "bbox": line["bbox"],
+                    "parsed_bbox": bbox,
+                    "type": line["ui_type"],
+                    "category": line["category"],
+                    "match": match,
+                    "is_wrong_format": is_wrong_format,
+                    "pred": click_point,
+                }
+            )
+
+        final_score_dict = {}
+        final_score_dict.update({k + ':cnt': len(stats[k]) for k in stats})
+
+        full_stats = []
+        for v in stats.values():
+            full_stats.extend(v)
+        final_score_dict['Overall_Accuracy'] = np.mean([x > 0 for x in full_stats]) * 100
+        final_score_dict['Format_Err_Rate'] = np.mean([x < 0 for x in full_stats]) * 100
+
+        cates = list(set([line["category"] for line in lines]))
+        for c in cates:
+            sub_stats = [v for k, v in stats.items() if k.split(":")[0] == c for x in v]
+            if len(sub_stats) > 0:
+                final_score_dict[c + '_Accuracy'] = np.mean([x[0] > 0 for x in [sub_stats]]) * 100
+
+        score_pth = get_intermediate_file_path(eval_file, '_score', 'json')
+        dump(final_score_dict, score_pth)
+        return final_score_dict

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/OmniDocBench/data_preprocess.py

@@ -0,0 +1,449 @@
+import html
+import os
+import re
+import shutil
+import subprocess
+import unicodedata
+import uuid
+
+from bs4 import BeautifulSoup
+from pylatexenc.latex2text import LatexNodes2Text
+
+
+def remove_markdown_fences(content):
+    content = re.sub(r'^```markdown\n?', '', content, flags=re.MULTILINE)
+    content = re.sub(r'```\n?$', '', content, flags=re.MULTILINE)
+    return content
+
+# Standardize all consecutive characters
+def replace_repeated_chars(input_str):
+    input_str = re.sub(r'_{4,}', '____', input_str) # Replace more than 4 consecutive underscores with 4 underscores
+    input_str = re.sub(r' {4,}', '    ', input_str)   # Replace more than 4 consecutive spaces with 4 spaces
+    return re.sub(r'([^a-zA-Z0-9])\1{10,}', r'\1\1\1\1', input_str) # For other consecutive symbols (except numbers and letters), replace more than 10 occurrences with 4
+
+# Special Unicode handling
+def fullwidth_to_halfwidth(s):
+    result = []
+    for char in s:
+        code = ord(char)
+        # Convert full-width space to half-width space
+        if code == 0x3000:
+            code = 0x0020
+        # Convert other full-width characters to half-width
+        elif 0xFF01 <= code <= 0xFF5E:
+            code -= 0xFEE0
+        result.append(chr(code))
+    return ''.join(result)
+
+def find_special_unicode(s):
+    special_chars = {}
+    for char in s:
+        if ord(char) > 127:  # Non-ASCII characters
+            # unicode_name = unicodedata.name(char, None)
+            unicode_name = unicodedata.category(char)
+            special_chars[char] = f'U+{ord(char):04X} ({unicode_name})'
+    return special_chars
+
+# # Define dictionary for Unicode character replacements
+# unicode_replacements = {
+#     "\u00A9": r"$\copyright$",  # Copyright symbol © to latex
+#     "\u00AE": r"$^\circledR$",  # Registered trademark ® to latex
+#     "\u2122": r"$^\text{TM}$",   # Trademark ™ to latex
+#     "\u2018": "'",             # Left single quote to straight quote
+#     "\u2019": "'",             # Right single quote to straight quote
+#     "\u201C": "\"",            # Left double quote to straight quote
+#     "\u201D": "\"",            # Right double quote to straight quote
+#     "\u2013": "-",             # En dash to hyphen
+#     "\u2014": "-",             # Em dash to hyphen
+#     "\u2026": "...",           # Unicode ellipsis to three dots
+#     "\u2103": r"$\textdegree C$",  # ℃
+#     "\u03B1": r"$\alpha$",         # α
+#     "\u03B2": r"$\beta$",          # β
+#     "\u03A3": r"$\Sigma$",         # Σ
+# }
+
+# # Use regex to replace Unicode characters
+# def replace_unicode(match):
+#     char = match.group(0)
+#     return unicode_replacements.get(char, char)
+
+inline_reg = re.compile(
+    r'\$(.*?)\$|'
+    r'\\\((.*?)\\\)',
+)
+
+def textblock2unicode(text):
+    inline_matches = inline_reg.finditer(text)
+    removal_positions = []
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        # print('-------- content-------', content)
+        # Remove escape characters \
+        clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+        try:
+            if any(char in clean_content for char in r'\^_'):
+                if clean_content.endswith('\\'):
+                    clean_content += ' '
+                # inline_array.append(match.group(0))
+                unicode_content = LatexNodes2Text().latex_to_text(clean_content)
+                removal_positions.append((position[0], position[1], unicode_content))
+        except:
+            continue
+
+    # Remove inline formulas from original text
+    for start, end, unicode_content in sorted(removal_positions, reverse=True):
+        text = text[:start] + unicode_content.strip() + text[end:]
+
+    return text
+
+def normalized_formula(text):
+    # Normalize math formulas before matching
+    filter_list = ['\\mathbf', '\\mathrm', '\\mathnormal', '\\mathit', '\\mathbb', '\\mathcal', '\\mathscr', '\\mathfrak', '\\mathsf', '\\mathtt',
+                   '\\textbf', '\\text', '\\boldmath', '\\boldsymbol', '\\operatorname', '\\bm',
+                   '\\symbfit', '\\mathbfcal', '\\symbf', '\\scriptscriptstyle', '\\notag',
+                   '\\setlength', '\\coloneqq', '\\space', '\\thickspace', '\\thinspace', '\\medspace', '\\nobreakspace', '\\negmedspace',
+                   '\\quad', '\\qquad', '\\enspace', '\\substackw', ' ']
+                #    '\\left', '\\right', '{', '}', ' ']
+
+    # delimiter_filter
+    pattern = re.compile(r"\\\[(.+?)(?<!\\)\\\]")
+    match = pattern.search(text)
+
+    if match:
+        text = match.group(1).strip()
+
+    tag_pattern = re.compile(r"\\tag\{.*?\}")
+    text = tag_pattern.sub('', text)
+    hspace_pattern = re.compile(r"\\hspace\{.*?\}")
+    text = hspace_pattern.sub('', text)
+    begin_pattern = re.compile(r"\\begin\{.*?\}")
+    text = begin_pattern.sub('', text)
+    end_pattern = re.compile(r"\\end\{.*?\}")
+    text = end_pattern.sub('', text)
+    col_sep = re.compile(r"\\arraycolsep.*?\}")
+    text = col_sep.sub('', text)
+    text = text.strip('.')
+
+    for filter_text in filter_list:
+        text = text.replace(filter_text, '')
+
+    # text = normalize_text(delimiter_filter(text))
+    # text = delimiter_filter(text)
+    text = text.lower()
+    return text
+
+def normalized_html_table(text):
+    def process_table_html(md_i):
+        """
+        pred_md format edit
+        """
+        def process_table_html(html_content):
+            soup = BeautifulSoup(html_content, 'html.parser')
+            th_tags = soup.find_all('th')
+            for th in th_tags:
+                th.name = 'td'
+            thead_tags = soup.find_all('thead')
+            for thead in thead_tags:
+                thead.unwrap()  # unwrap()会移除标签但保留其内容
+            math_tags = soup.find_all('math')
+            for math_tag in math_tags:
+                alttext = math_tag.get('alttext', '')
+                alttext = f'${alttext}$'
+                if alttext:
+                    math_tag.replace_with(alttext)
+            span_tags = soup.find_all('span')
+            for span in span_tags:
+                span.unwrap()
+            return str(soup)
+
+        table_res=''
+        table_res_no_space=''
+        if '<table' in md_i.replace(" ","").replace("'",'"'):
+            md_i = process_table_html(md_i)
+            table_res = html.unescape(md_i).replace('\n', '')
+            table_res = unicodedata.normalize('NFKC', table_res).strip()
+            pattern = r'<table\b[^>]*>(.*)</table>'
+            tables = re.findall(pattern, table_res, re.DOTALL | re.IGNORECASE)
+            table_res = ''.join(tables)
+            # table_res = re.sub('<table.*?>','',table_res)
+            table_res = re.sub('( style=".*?")', "", table_res)
+            table_res = re.sub('( height=".*?")', "", table_res)
+            table_res = re.sub('( width=".*?")', "", table_res)
+            table_res = re.sub('( align=".*?")', "", table_res)
+            table_res = re.sub('( class=".*?")', "", table_res)
+            table_res = re.sub('</?tbody>',"",table_res)
+
+            table_res = re.sub(r'\s+', " ", table_res)
+            table_res_no_space = '<html><body><table border="1" >' + table_res.replace(' ','') + '</table></body></html>'
+            # table_res_no_space = re.sub(' (style=".*?")',"",table_res_no_space)
+            # table_res_no_space = re.sub(r'[ ]', " ", table_res_no_space)
+            table_res_no_space = re.sub('colspan="', ' colspan="', table_res_no_space)
+            table_res_no_space = re.sub('rowspan="', ' rowspan="', table_res_no_space)
+            table_res_no_space = re.sub('border="', ' border="', table_res_no_space)
+
+            table_res = '<html><body><table border="1" >' + table_res + '</table></body></html>'
+            # table_flow.append(table_res)
+            # table_flow_no_space.append(table_res_no_space)
+
+        return table_res, table_res_no_space
+
+    def clean_table(input_str,flag=True):
+        if flag:
+            input_str = input_str.replace('<sup>', '').replace('</sup>', '')
+            input_str = input_str.replace('<sub>', '').replace('</sub>', '')
+            input_str = input_str.replace('<span>', '').replace('</span>', '')
+            input_str = input_str.replace('<div>', '').replace('</div>', '')
+            input_str = input_str.replace('<p>', '').replace('</p>', '')
+            input_str = input_str.replace('<spandata-span-identity="">', '')
+            input_str = re.sub('<colgroup>.*?</colgroup>','',input_str)
+        return input_str
+
+    norm_text, _ = process_table_html(text)
+    norm_text = clean_table(norm_text)
+    return norm_text
+
+def normalized_latex_table(text):
+    def latex_template(latex_code):
+        template = r'''
+        \documentclass[border=20pt]{article}
+        \usepackage{subcaption}
+        \usepackage{url}
+        \usepackage{graphicx}
+        \usepackage{caption}
+        \usepackage{multirow}
+        \usepackage{booktabs}
+        \usepackage{color}
+        \usepackage{colortbl}
+        \usepackage{xcolor,soul,framed}
+        \usepackage{fontspec}
+        \usepackage{amsmath,amssymb,mathtools,bm,mathrsfs,textcomp}
+        \setlength{\parindent}{0pt}''' + \
+        r'''
+        \begin{document}
+        ''' + \
+        latex_code + \
+        r'''
+        \end{document}'''
+
+        return template
+
+    def process_table_latex(latex_code):
+        SPECIAL_STRINGS= [
+            ['\\\\vspace\\{.*?\\}', ''],
+            ['\\\\hspace\\{.*?\\}', ''],
+            ['\\\\rule\{.*?\\}\\{.*?\\}', ''],
+            ['\\\\addlinespace\\[.*?\\]', ''],
+            ['\\\\addlinespace', ''],
+            ['\\\\renewcommand\\{\\\\arraystretch\\}\\{.*?\\}', ''],
+            ['\\\\arraystretch\\{.*?\\}', ''],
+            ['\\\\(row|column)?colors?\\{[^}]*\\}(\\{[^}]*\\}){0,2}', ''],
+            ['\\\\color\\{.*?\\}', ''],
+            ['\\\\textcolor\\{.*?\\}', ''],
+            ['\\\\rowcolor(\\[.*?\\])?\\{.*?\\}', ''],
+            ['\\\\columncolor(\\[.*?\\])?\\{.*?\\}', ''],
+            ['\\\\cellcolor(\\[.*?\\])?\\{.*?\\}', ''],
+            ['\\\\colorbox\\{.*?\\}', ''],
+            ['\\\\(tiny|scriptsize|footnotesize|small|normalsize|large|Large|LARGE|huge|Huge)', ''],
+            [r'\s+', ' '],
+            ['\\\\centering', ''],
+            ['\\\\begin\\{table\\}\\[.*?\\]', '\\\\begin{table}'],
+            ['\t', ''],
+            ['@{}', ''],
+            ['\\\\toprule(\\[.*?\\])?', '\\\\hline'],
+            ['\\\\bottomrule(\\[.*?\\])?', '\\\\hline'],
+            ['\\\\midrule(\\[.*?\\])?', '\\\\hline'],
+            ['p\\{[^}]*\\}', 'l'],
+            ['m\\{[^}]*\\}', 'c'],
+            ['\\\\scalebox\\{[^}]*\\}\\{([^}]*)\\}', '\\1'],
+            ['\\\\textbf\\{([^}]*)\\}', '\\1'],
+            ['\\\\textit\\{([^}]*)\\}', '\\1'],
+            ['\\\\cmidrule(\\[.*?\\])?\\(.*?\\)\\{([0-9]-[0-9])\\}', '\\\\cline{\\2}'],
+            ['\\\\hline', ''],
+            [r'\\multicolumn\{1\}\{[^}]*\}\{((?:[^{}]|(?:\{[^{}]*\}))*)\}', r'\1']
+        ]
+        pattern = r'\\begin\{tabular\}.*\\end\{tabular\}'  # 注意这里不用 .*?
+        matches = re.findall(pattern, latex_code, re.DOTALL)
+        latex_code = ' '.join(matches)
+
+        for special_str in SPECIAL_STRINGS:
+            latex_code = re.sub(fr'{special_str[0]}', fr'{special_str[1]}', latex_code)
+
+        return latex_code
+
+    def convert_latex_to_html(latex_content, cache_dir='./temp'):
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+
+        uuid_str = str(uuid.uuid1())
+        with open(f'{cache_dir}/{uuid_str}.tex', 'w') as f:
+            f.write(latex_template(latex_content))
+
+        cmd = ['latexmlc', '--quiet', '--nocomments', f'--log={cache_dir}/{uuid_str}.log',
+               f'{cache_dir}/{uuid_str}.tex', f'--dest={cache_dir}/{uuid_str}.html']
+        try:
+            subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+            with open(f'{cache_dir}/{uuid_str}.html', 'r') as f:
+                html_content = f.read()
+
+            pattern = r'<table\b[^>]*>(.*)</table>'
+            tables = re.findall(pattern, html_content, re.DOTALL | re.IGNORECASE)
+            tables = [f'<table>{table}</table>' for table in tables]
+            html_content = '\n'.join(tables)
+
+        except Exception as e:
+            html_content = ''
+
+        shutil.rmtree(cache_dir)
+        return html_content
+
+    html_text = convert_latex_to_html(text)
+    normlized_tables = normalized_html_table(html_text)
+    return normlized_tables
+
+
+def normalized_table(text, format='html'):
+    if format not in ['html', 'latex']:
+        raise ValueError('Invalid format: {}'.format(format))
+    else:
+        return globals()['normalized_{}_table'.format(format)](text)
+
+
+def textblock_with_norm_formula(text):
+    inline_matches = inline_reg.finditer(text)
+    removal_positions = []
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        # print('-------- content-------', content)
+
+        norm_content = normalized_formula(content)
+        removal_positions.append((position[0], position[1], norm_content))
+
+    # Remove inline formulas from original text
+    for start, end, norm_content in sorted(removal_positions, reverse=True):
+        text = text[:start] + norm_content.strip() + text[end:]
+
+    return text
+
+# def inline_filter_unicode(text):
+#     # Ensure text is string type
+#     if not isinstance(text, str):
+#         text = str(text)
+
+#     # Convert LaTeX content to Unicode representation
+#     text = LatexNodes2Text().latex_to_text(text)
+
+#     inline_array = []
+#     inline_matches = inline_reg.finditer(text)
+
+#     for match in inline_matches:
+#         position = [match.start(), match.end()]
+#         content = match.group(1) if match.group(1) is not None else match.group(2)
+
+#         # Remove escape characters \
+#         clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+#         if any(char in clean_content for char in r'\^_'):
+#             # inline_array.append(match.group(0))
+#             inline_array.append({
+#                 'category_type': 'equation_inline',
+#                 'position': position,
+#                 'content': match.group(0),
+#             })
+#             text = text.replace(match.group(0), '')
+#             # print('-----Found inline formula: ', match.group(0))
+#         else:
+#             text = text.replace(match.group(0), content)
+#         # # Add to inline_array
+#         # inline_array.append({
+#         #     'category_type': 'equation_inline',
+#         #     'position': position,
+#         #     'content': content,
+#         # })
+
+#         # # Remove matched formula from original text, can choose to replace with spaces or remove directly
+#         # text = text[:position[0]] + ' '*(position[1]-position[0]) + text[position[1]:]
+
+#     return text, inline_array
+
+def inline_filter_unicode(text):
+    # Ensure text is string type
+    if not isinstance(text, str):
+        text = str(text)
+
+    # Replace inline formula boundary markers
+    #print('--------text-------',text)
+    placeholder = '__INLINE_FORMULA_BOUNDARY__'
+    text_copy = text.replace('$', placeholder).replace('\\(', placeholder).replace('\\)', placeholder)
+    #print('--------text_copy-------',text_copy)
+    # Convert LaTeX content to Unicode representation
+    text_copy = LatexNodes2Text().latex_to_text(text_copy)
+    #print('--------text_copy---unicode----',text_copy)
+    # Restore boundary markers
+    text_copy = text_copy.replace(placeholder, '$')
+
+    inline_array = []
+    inline_matches = inline_reg.finditer(text_copy)
+    # Record positions of inline formulas to be removed
+    removal_positions = []
+
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        print('-------- content-------', content)
+        # Remove escape characters \
+        clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+        if any(char in clean_content for char in r'\^_'):
+            # inline_array.append(match.group(0))
+            inline_array.append({
+                'category_type': 'equation_inline',
+                'position': position,
+                'content': content,
+            })
+            removal_positions.append((position[0], position[1]))
+
+    # Remove inline formulas from original text
+    for start, end in sorted(removal_positions, reverse=True):
+        text = text[:start] + text[end:]
+
+    return text, inline_array
+
+def inline_filter(text):
+    # Ensure text is string type
+    if not isinstance(text, str):
+        text = str(text)
+
+    inline_array = []
+    inline_matches = inline_reg.finditer(text)
+
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        # print('inline_content: ', content)
+
+        # Remove escape characters \
+        clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+        if any(char in clean_content for char in r'\^_'):
+            # inline_array.append(match.group(0))
+            inline_array.append({
+                'category_type': 'equation_inline',
+                'position': position,
+                'content': match.group(0),
+            })
+            text = text.replace(match.group(0), '')
+            # print('-----Found inline formula: ', match.group(0))
+        else:
+            text = text.replace(match.group(0), content)
+
+    return text, inline_array
+
+# Text OCR quality check processing:
+def clean_string(input_string):
+    # Use regex to keep Chinese characters, English letters and numbers
+    input_string = input_string.replace('\\t', '').replace('\\n', '').replace('\t', '').replace('\n', '').replace('/t', '').replace('/n', '')
+    cleaned_string = re.sub(r'[^\w\u4e00-\u9fff]', '', input_string)
+    return cleaned_string

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/OmniDocBench/metrics.py

@@ -0,0 +1,486 @@
+import copy
+import json
+import pdb
+import random
+import time
+from collections import defaultdict, deque
+
+import evaluate
+import Levenshtein
+import pandas as pd
+from apted import APTED, Config
+from apted.helpers import Tree
+from lxml import etree, html
+from tabulate import tabulate
+from tqdm import tqdm
+
+from .utils import normalized_table, save_paired_result
+
+
+def show_result(results):
+    for metric_name in results.keys():
+        print(f'{metric_name}:')
+        score_table = [[k,v] for k,v in results[metric_name].items()]
+        print(tabulate(score_table))
+        print('='*100)
+
+def sort_nested_dict(d):
+    # If it's a dictionary, recursively sort it
+    if isinstance(d, dict):
+        # Sort the current dictionary
+        sorted_dict = {k: sort_nested_dict(v) for k, v in sorted(d.items())}
+        return sorted_dict
+    # If not a dictionary, return directly
+    return d
+
+def get_full_labels_results(samples:dict):
+    if not samples:
+        return {}
+    label_group_dict = defaultdict(lambda: defaultdict(list))
+    for sample in samples:
+        label_list = []
+        if not sample.get("gt_attribute"):
+            continue
+        for anno in sample["gt_attribute"]:
+            for k,v in anno.items():
+                label_list.append(k+": "+str(v))
+        for label_name in list(set(label_list)):  # Currently if there are merged cases, calculate based on the set of all labels involved after merging
+            for metric, score in sample['metric'].items():
+                label_group_dict[label_name][metric].append(score)
+
+    print('----Anno Attribute---------------')
+    result = {}
+    result['sample_count'] = {}
+    for attribute in label_group_dict.keys():
+        for metric, scores in label_group_dict[attribute].items():
+            mean_score = sum(scores) / len(scores)
+            if not result.get(metric):
+                result[metric] = {}
+            result[metric][attribute] = mean_score
+            result['sample_count'][attribute] = len(scores)
+    result = sort_nested_dict(result)
+    show_result(result)
+    return result
+
+
+def get_page_split(samples, page_info):   # Page level metric
+    if not page_info:
+        return {}
+    result_list = defaultdict(list)
+
+
+    for sample in samples:
+        img_name = sample['img_id'] if sample['img_id'].endswith('.jpg') else '_'.join(sample['img_id'].split('_')[:-1])
+        page_info_s = page_info[img_name]
+        if not sample.get('metric'):
+            continue
+        for metric, score in sample['metric'].items():
+            gt = sample['norm_gt'] if sample.get('norm_gt') else sample['gt']
+            pred = sample['norm_pred'] if sample.get('norm_pred') else sample['pred']
+            result_list[metric].append({
+                'image_name': img_name,
+                'metric': metric,
+                'attribute': 'ALL',
+                'score': score,
+                'upper_len': max(len(gt), len(pred))
+            })
+            for k,v in page_info_s.items():
+                if isinstance(v, list): # special issue
+                    for special_issue in v:
+                        if 'table' not in special_issue:  # Table-related special fields have duplicates
+                            result_list[metric].append({
+                                'image_name': img_name,
+                                'metric': metric,
+                                'attribute': special_issue,
+                                'score': score,
+                                'upper_len': max(len(gt), len(pred))
+                            })
+                else:
+                    result_list[metric].append({
+                        'image_name': img_name,
+                        'metric': metric,
+                        'attribute': k+": "+str(v),
+                        'score': score,
+                        'upper_len': max(len(gt), len(pred))
+                    })
+
+    # Page level logic, accumulation is only done within pages, and mean operation is performed between pages
+    result = {}
+    if result_list.get('Edit_dist'):
+        df = pd.DataFrame(result_list['Edit_dist'])
+        up_total_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: (x["score"]*x['upper_len']).sum() / x['upper_len'].sum()).groupby('attribute').mean()  # At page level, accumulate edits, denominator is sum of max(gt, pred) from each sample
+        result['Edit_dist'] = up_total_avg.to_dict()
+    for metric in result_list.keys():
+        if metric == 'Edit_dist':
+            continue
+        df = pd.DataFrame(result_list[metric])
+        page_avg = df.groupby(["image_name", "attribute"]).apply(lambda x: x["score"].mean()).groupby('attribute').mean()
+        result[metric] = page_avg.to_dict()
+
+    result = sort_nested_dict(result)
+    # print('----Page Attribute---------------')
+    show_result(result)
+    return result
+
+
+def get_groups(samples, group_info):
+    group_samples = defaultdict(list)
+    for sample in samples:
+        group_samples['all'].append(sample)
+        for group in group_info:
+            select_flag = True
+            for k, v in group.items():
+                for gt_attribute in sample['gt_attribute']:   # gt_attribute is a list containing all merged gt attributes
+                    if not gt_attribute:   # if no GT attributes, don't include in calculation
+                        select_flag = False
+                    elif gt_attribute[k] != v:  # if any gt attribute doesn't meet criteria, don't select
+                        select_flag = False
+            if select_flag:
+                group_samples[str(group)].append(sample)
+    return group_samples
+
+
+class Registry:
+    def __init__(self):
+        self._registry = {}
+    def register(self, name):
+        def decorator(item):
+            if name in self._registry:
+                raise ValueError(f"Item {name} already registered.")
+            self._registry[name] = item
+            return item
+        return decorator
+    def get(self, name):
+        if name not in self._registry:
+            raise ValueError(f"Item {name} not found in registry.")
+        return self._registry[name]
+    def list_items(self):
+        return list(self._registry.keys())
+
+METRIC_REGISTRY = Registry()
+
+
+@METRIC_REGISTRY.register("TEDS")
+class call_TEDS():
+    def __init__(self, samples):
+        self.samples = samples
+    def evaluate(self, group_info=[], save_name='default'):
+        teds = TEDS(structure_only=False)
+        teds_structure_only = TEDS(structure_only=True)
+
+        group_scores = defaultdict(list)
+        group_scores_structure_only = defaultdict(list)
+
+        samples = self.samples
+        for sample in samples:
+            gt = sample['norm_gt'] if sample.get('norm_gt') else sample['gt']
+            pred = sample['norm_pred'] if sample.get('norm_pred') else sample['pred']
+
+            score = teds.evaluate(pred, gt)
+            score_structure_only = teds_structure_only.evaluate(pred, gt)
+            # print('TEDS score:', score)
+            group_scores['all'].append(score)
+            group_scores_structure_only['all'].append(score_structure_only)
+
+            if not sample.get('metric'):
+                sample['metric'] = {}
+            sample['metric']['TEDS'] = score
+            sample['metric']['TEDS_structure_only'] = score_structure_only
+
+            for group in group_info:
+                select_flag = True
+                for k, v in group.items():
+                    for gt_attribute in sample['gt_attribute']:   # gt_attribute is a list containing all merged gt attributes
+                        if not gt_attribute:   # if no GT attributes, don't include in calculation
+                            select_flag = False
+                        elif gt_attribute[k] != v:  # if any gt attribute doesn't meet criteria, don't select
+                            select_flag = False
+                if select_flag:
+                    group_scores[str(group)].append(score)
+
+        result = {}
+        for group_name, scores in group_scores.items():
+            if len(scores) > 0:
+                result[group_name] = sum(scores) / len(scores)    # average of normalized scores at sample level
+            else:
+                result[group_name] = 'NaN'
+                print(f'Warning: Empyty matched samples for {group_name}.')
+
+        structure_only_result = {}
+        for group_name, scores in group_scores_structure_only.items():
+            if len(scores) > 0:
+                structure_only_result[group_name] = sum(scores) / len(scores)    # average of normalized scores at sample level
+            else:
+                structure_only_result[group_name] = 'NaN'
+                print(f'Warning: Empyty matched samples for {group_name}.')
+
+        return samples,{'TEDS': result, 'TEDS_structure_only': structure_only_result}
+
+
+@METRIC_REGISTRY.register("BLEU")
+class call_BLEU():
+    def __init__(self, samples):
+        self.samples = samples
+    def evaluate(self, group_info=[], save_name='default'):
+        group_samples = get_groups(self.samples, group_info)
+        result = {}
+        bleu = evaluate.load("bleu", keep_in_memory=True, experiment_id=random.randint(1,1e8))
+
+        for group_name, samples in group_samples.items():
+            predictions, references = [], []
+            for sample in samples:
+                gt = sample['norm_gt'] if sample.get('norm_gt') else sample['gt']
+                pred = sample['norm_pred'] if sample.get('norm_pred') else sample['pred']
+                predictions.append(pred)
+                references.append(gt)
+
+        if not predictions or not any(predictions) or not references or not any(references):
+            bleu_score = 0
+        else:
+            try:
+                bleu_results = bleu.compute(predictions=predictions, references=references)
+                bleu_score = bleu_results["bleu"]
+            except ZeroDivisionError:
+                bleu_score = 0
+
+        result[group_name] = bleu_score
+
+        return self.samples,{'BLEU': result}
+
+@METRIC_REGISTRY.register("METEOR")
+class call_METEOR():
+    def __init__(self, samples):
+        self.samples = samples
+    def evaluate(self, group_info=[], save_name='default'):
+        group_samples = get_groups(self.samples, group_info)
+        result = {}
+        for group_name, samples in group_samples.items():
+            predictions, references = [], []
+            for sample in samples:
+                gt = sample['norm_gt'] if sample.get('norm_gt') else sample['gt']
+                pred = sample['norm_pred'] if sample.get('norm_pred') else sample['pred']
+                predictions.append(gt)
+                references.append(pred)
+            meteor = evaluate.load('meteor', keep_in_memory=True, experiment_id=random.randint(1,1e8))
+            meteor_results = meteor.compute(predictions=predictions, references=references)
+            result[group_name] = meteor_results['meteor']
+
+        return self.samples,{'METEOR': result}
+
+
+@METRIC_REGISTRY.register("Edit_dist")
+class call_Edit_dist():
+    def __init__(self, samples):
+        self.samples = samples
+    def evaluate(self, group_info=[], save_name='default'):
+        samples = self.samples
+        for sample in samples:
+            img_name = sample['img_id'] if sample['img_id'].endswith('.jpg') else '_'.join(sample['img_id'].split('_')[:-1])
+            sample['image_name'] = img_name
+            gt = sample['norm_gt'] if sample.get('norm_gt') else sample['gt']
+            pred = sample['norm_pred'] if sample.get('norm_pred') else sample['pred']
+            upper_len = max(len(pred), len(gt))
+            sample['upper_len'] = upper_len
+            if len(pred) > 0 or len(gt) > 0:
+                edit_dist = Levenshtein.distance(pred, gt)
+                if not sample.get('metric'):
+                    sample['metric'] = {}
+                sample['metric']['Edit_dist'] = edit_dist / upper_len
+                sample['Edit_num'] = edit_dist
+
+        if isinstance(samples, list):
+            saved_samples = samples
+        else:
+            saved_samples = samples.samples
+
+        if not saved_samples:
+            return {'Edit_dist': {'ALL_page_avg': 'NaN'}}
+
+        df = pd.DataFrame(saved_samples)
+        up_total_avg = df.groupby("image_name").apply(lambda x: x['Edit_num'].sum() / x['upper_len'].sum()) # page level, sum of edits divided by sum of max(gt,pred) lengths for each sample
+        per_img_score = up_total_avg.to_dict()
+
+        return samples,{'Edit_dist': {'ALL_page_avg': up_total_avg.mean()}}
+
+
+@METRIC_REGISTRY.register("CDM")
+class call_CDM():
+    def __init__(self, samples):
+        self.samples = samples
+    def evaluate(self, group_info=[], save_name='default'):
+        if isinstance(self.samples, list):
+            cdm_samples = copy.deepcopy(self.samples)
+        else:
+            cdm_samples = copy.deepcopy(self.samples.samples)
+        for idx, sample in enumerate(cdm_samples):
+            sample['img_name'] = sample['img_id']
+            sample['img_id'] = str(idx)
+            sample['gt'] = sample['gt'].lstrip("$$").rstrip("$$").strip()
+            sample['pred'] = sample['pred'].split("```latex")[-1].split("```")[0]
+            sample['pred'] = sample['pred'].lstrip("$$").rstrip("$$").strip()
+
+        return  self.samples,False
+
+
+class TEDS(object):
+    ''' Tree Edit Distance basead Similarity
+    '''
+    def __init__(self, structure_only=False, n_jobs=1, ignore_nodes=None):
+        assert isinstance(n_jobs, int) and (n_jobs >= 1), 'n_jobs must be an integer greather than 1'
+        self.structure_only = structure_only
+        self.n_jobs = n_jobs
+        self.ignore_nodes = ignore_nodes
+        self.__tokens__ = []
+
+    def tokenize(self, node):
+        ''' Tokenizes table cells
+        '''
+        self.__tokens__.append('<%s>' % node.tag)
+        if node.text is not None:
+            self.__tokens__ += list(node.text)
+        for n in node.getchildren():
+            self.tokenize(n)
+        if node.tag != 'unk':
+            self.__tokens__.append('</%s>' % node.tag)
+        if node.tag != 'td' and node.tail is not None:
+            self.__tokens__ += list(node.tail)
+
+    def load_html_tree(self, node, parent=None):
+        ''' Converts HTML tree to the format required by apted
+        '''
+        global __tokens__
+        if node.tag == 'td':
+            if self.structure_only:
+                cell = []
+            else:
+                self.__tokens__ = []
+                self.tokenize(node)
+                cell = self.__tokens__[1:-1].copy()
+            new_node = TableTree(node.tag,
+                                 int(node.attrib.get('colspan', '1')),
+                                 int(node.attrib.get('rowspan', '1')),
+                                 cell, *deque())
+        else:
+            new_node = TableTree(node.tag, None, None, None, *deque())
+        if parent is not None:
+            parent.children.append(new_node)
+        if node.tag != 'td':
+            for n in node.getchildren():
+                self.load_html_tree(n, new_node)
+        if parent is None:
+            return new_node
+
+    def evaluate(self, pred, true):
+        ''' Computes TEDS score between the prediction and the ground truth of a
+            given sample
+        '''
+        if (not pred) or (not true):
+            return 0.0
+        parser = html.HTMLParser(remove_comments=True, encoding='utf-8')
+        pred = html.fromstring(pred, parser=parser)
+        true = html.fromstring(true, parser=parser)
+        if pred.xpath('body/table') and true.xpath('body/table'):
+            pred = pred.xpath('body/table')[0]
+            true = true.xpath('body/table')[0]
+            if self.ignore_nodes:
+                etree.strip_tags(pred, *self.ignore_nodes)
+                etree.strip_tags(true, *self.ignore_nodes)
+            n_nodes_pred = len(pred.xpath(".//*"))
+            n_nodes_true = len(true.xpath(".//*"))
+            n_nodes = max(n_nodes_pred, n_nodes_true)
+            tree_pred = self.load_html_tree(pred)
+            tree_true = self.load_html_tree(true)
+            distance = APTED(tree_pred, tree_true, CustomConfig()).compute_edit_distance()
+            return 1.0 - (float(distance) / n_nodes)
+        else:
+            return 0.0
+
+    def batch_evaluate(self, pred_json, true_json):
+        ''' Computes TEDS score between the prediction and the ground truth of
+            a batch of samples
+            @params pred_json: {'FILENAME': 'HTML CODE', ...}
+            @params true_json: {'FILENAME': {'html': 'HTML CODE'}, ...}
+            @output: {'FILENAME': 'TEDS SCORE', ...}
+        '''
+        samples = true_json.keys()
+        # if self.n_jobs == 1:
+        scores = [self.evaluate(pred_json.get(filename, ''), true_json[filename]['html']) for filename in tqdm(samples)]
+        # else:
+        #     inputs = [{'pred': pred_json.get(filename, ''), 'true': true_json[filename]['html']} for filename in samples]
+        #     scores = parallel_process(inputs, self.evaluate, use_kwargs=True, n_jobs=self.n_jobs, front_num=1)
+        scores = dict(zip(samples, scores))
+        return scores
+
+
+class CustomConfig(Config):
+    @staticmethod
+    def maximum(*sequences):
+        """Get maximum possible value
+        """
+        return max(map(len, sequences))
+
+    def normalized_distance(self, *sequences):
+        """Get distance from 0 to 1
+        """
+        return float(Levenshtein.distance(*sequences)) / self.maximum(*sequences)
+
+    def rename(self, node1, node2):
+        """Compares attributes of trees"""
+        if (node1.tag != node2.tag) or (node1.colspan != node2.colspan) or (node1.rowspan != node2.rowspan):
+            return 1.
+        if node1.tag == 'td':
+            if node1.content or node2.content:
+                return self.normalized_distance(node1.content, node2.content)
+        return 0.
+
+
+class TableTree(Tree):
+    def __init__(self, tag, colspan=None, rowspan=None, content=None, *children):
+        self.tag = tag
+        self.colspan = colspan
+        self.rowspan = rowspan
+        self.content = content
+        self.children = list(children)
+
+    def bracket(self):
+        """Show tree using brackets notation"""
+        if self.tag == 'td':
+            result = '"tag": %s, "colspan": %d, "rowspan": %d, "text": %s' % \
+                     (self.tag, self.colspan, self.rowspan, self.content)
+        else:
+            result = '"tag": %s' % self.tag
+        for child in self.children:
+            result += child.bracket()
+        return "{{{}}}".format(result)
+
+
+class recogition_end2end_base_dataset():
+    def __init__(self, samples):
+        img_id = 0
+        for sample in samples:
+            if not sample.get('img_id'):
+                sample['img_id'] = img_id
+            img_id += 1
+        self.samples = samples
+    def __getitem__(self, idx):
+        return self.samples[idx]
+
+
+class recogition_end2end_table_dataset(recogition_end2end_base_dataset):
+    def __init__(self, samples, table_format):
+        self.pred_table_format = table_format
+        self.samples = self.normalize_data(samples)
+
+    def normalize_data(self, samples):
+        img_id = 0
+        for sample in samples:
+            p = sample['pred']
+            r = sample['gt']
+            p = normalized_table(p, self.pred_table_format)
+            r = normalized_table(r)
+            sample['norm_gt'] = r
+            sample['norm_pred'] = p
+            sample['img_id'] = sample['img_id'] if sample.get('img_id') else img_id
+            img_id += 1
+
+        return samples

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/OmniDocBench/omnidocbench.py

@@ -0,0 +1,557 @@
+import base64
+import copy
+import json
+import os
+import tempfile
+
+import numpy as np
+import pandas as pd
+import torch.distributed as dist
+from tqdm import tqdm
+
+from vlmeval.smp import dump, get_intermediate_file_path, load
+from ..image_base import ImageBaseDataset
+
+# from ..utils import get_intermediate_file_path, load, dump
+
+
+class OmniDocBench(ImageBaseDataset):
+
+    MODALITY = 'IMAGE'
+    TYPE = 'QA'
+
+    DATASET_URL = {'OmniDocBench':'https://huggingface.co/datasets/ouyanglinke/OmniDocBench_tsv/resolve/main/OmniDocBench.tsv'}
+    DATASET_MD5 = {'OmniDocBench': '0fa5ccf31e682e219cb9ca83da741a59'}
+
+
+    system_prompt = r'''You are an AI assistant specialized in converting PDF images to Markdown format. Please follow these instructions for the conversion:
+
+    1. Text Processing:
+    - Accurately recognize all text content in the PDF image without guessing or inferring.
+    - Convert the recognized text into Markdown format.
+    - Maintain the original document structure, including headings, paragraphs, lists, etc.
+
+    2. Mathematical Formula Processing:
+    - Convert all mathematical formulas to LaTeX format.
+    # - Enclose inline formulas with \( \). For example: This is an inline formula \( E = mc^2 \)
+    - Enclose block formulas with \\[ \\]. For example: \[ \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} \]
+
+    3. Table Processing:
+    - Convert tables to HTML format.
+    - Wrap the entire table with <table> and </table>.
+
+    4. Figure Handling:
+    - Ignore figures content in the PDF image. Do not attempt to describe or convert images.
+
+    5. Output Format:
+    - Ensure the output Markdown document has a clear structure with appropriate line breaks between elements.
+    - For complex layouts, try to maintain the original document's structure and format as closely as possible.
+
+    Please strictly follow these guidelines to ensure accuracy and consistency in the conversion. Your task is to accurately convert the content of the PDF image into Markdown format without adding any extra explanations or comments.
+    '''
+
+    def __init__(self,dataset='OmniDocBench',**kwargs):
+        super().__init__(dataset,**kwargs)
+        print(f'self.img_root:{self.img_root}')
+
+    def build_prompt(self, line):
+
+        image_path = self.dump_image(line)[0]
+        msg = [
+            dict(type='image', value=image_path),
+            dict(type='text', value=self.system_prompt)
+        ]
+        return msg
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        tsv_path=self.data_path
+        End2end_evaluator=end2end_evaluator(eval_file,tsv_path)
+        Table_evalutor=table_evalutor(eval_file,tsv_path)
+
+        metrics_all=End2end_evaluator.score()
+        metircs_table=Table_evalutor.score()
+
+        return metrics_all
+
+
+class end2end_evaluator():
+    def __init__(self,
+                 eval_file,
+                 tsv_path,
+                 match_method:str='quick_match',
+                 filter_types:dict=None):
+        self.eval_file=eval_file
+        self.match_method=match_method
+        self.references=[]
+        self.predictions = load(eval_file)['prediction'].tolist()
+        self.dafault_metircs_dict={
+            'text_block':
+                {'metric': ['Edit_dist', 'BLEU', 'METEOR']},
+            'display_formula':
+                {'metric': ['Edit_dist', 'CDM']},
+            'table':
+                {'metric': ['TEDS', 'Edit_dist']},
+            'reading_order':
+                {'metric': ['Edit_dist']}
+            }
+
+        references = load(tsv_path)['answer'].tolist()
+
+        load_success,load_fail=0,0
+        for i,ans in tqdm(enumerate(references),desc='Loading data'):
+            try:
+                ans = json.loads(ans)
+                load_success+=1
+                self.references.append(ans) #[{},{}]
+            except json.JSONDecodeError as e:
+                load_fail+=1
+                continue
+        print(f'load_success:{load_success},load_fail:{load_fail}')
+
+        filtered_gt_samples = []
+        if filter_types:
+            for gt_sample in self.references:
+                select_flag = True
+                for k, v in filter_types.items():
+                    if gt_sample["page_info"]["page_attribute"][k] != v:
+                        select_flag = False
+                if select_flag:
+                    filtered_gt_samples.append(gt_sample)
+        else:
+            filtered_gt_samples = self.references #[{},{},{}]
+        self.references=filtered_gt_samples
+
+
+    def score(self)->dict:
+        samples=self.get_matched_elements(self.references,self.predictions)
+        metrics=self.process_generated_metric_results(samples)
+        return metrics
+
+    def get_page_elements(self, selected_annos):
+        saved_element_dict = defaultdict(list)
+        related_truncated = []
+        truncated_all = {}
+        for relation in selected_annos["extra"]["relation"]:   # Handle truncated text issues
+            if relation["relation_type"] == 'truncated':
+                truncated_all[relation["source_anno_id"]] = ""
+                truncated_all[relation["target_anno_id"]] = ""
+                exist_flag = False
+                for merge_list in related_truncated:
+                    if relation["source_anno_id"] in merge_list or relation["target_anno_id"] in merge_list:  # Consider cases where three text blocks may need to be merged
+                        merge_list.append(relation["source_anno_id"])
+                        merge_list.append(relation["target_anno_id"])
+                        exist_flag = True
+                if not exist_flag:
+                    related_truncated.append([relation["source_anno_id"], relation["target_anno_id"]])
+
+        for item in selected_annos['layout_dets']:
+            if item['anno_id'] not in truncated_all.keys():
+                saved_element_dict[item["category_type"]].append(item)
+            else:
+                truncated_all[item['anno_id']] = item
+
+        for merge_list in related_truncated:
+            text_block_list = [truncated_all[key] for key in merge_list]
+            sorted_block = sorted(text_block_list, key=lambda x: x['order'])
+            text = ""
+            for block in sorted_block:
+                text += block['text']
+            merged_block = {
+                "category_type": sorted_block[0]["category_type"], # Directly use information from the first block
+                "order": sorted_block[0]["order"],
+                "anno_id": sorted_block[0]["anno_id"],
+                "text": text,
+                "merge_list": sorted_block
+            }
+            saved_element_dict[sorted_block[0]["category_type"]].append(merged_block)
+
+        return saved_element_dict
+
+    def get_page_elements_list(self, gt_page_elements, category_list):
+        element_list = []
+        for category_type in category_list:
+            if gt_page_elements.get(category_type):
+                element_list.extend(gt_page_elements[category_type])
+        return element_list
+
+    def get_sorted_text_list(self, selected_annos):
+        # txt_type: text, latex, html
+        text_list = []
+        for item in selected_annos:
+            if item.get('order'):
+                order = item['order']
+            else:
+                order = 0
+            # 【txt_type,selecte_annos]
+            text_list.append((order, item))
+        sorted_text_list = sorted(text_list, key=lambda x: x[0])
+        return [_[1] for _ in sorted_text_list]
+
+    def filtered_out_ignore(self, items, ignore_category_list):
+        filted_items = []
+        for item in items:
+            if item['gt_category_type'] not in ignore_category_list:
+                filted_items.append(item)
+        return filted_items
+
+    def get_order_paired(self, order_match_s, img_name):
+        matched = [(item['gt_position'], item['pred_position']) for item in order_match_s if (item['gt_position'] != [""] and item['pred_position'] != "")]
+        gt_idx_all = [item['gt_position'] for item in order_match_s if (item['gt_position'] != [""])]
+        read_order_pred = [i[0] for i in sorted(matched, key=lambda x: x[1])]
+        read_order_gt = sum(gt_idx_all, []) # Convert to one-dimensional list
+        read_order_gt = [x for x in read_order_gt if x]
+        gt = sorted(read_order_gt)
+        pred = sum(read_order_pred, [])
+        pred = [x for x in pred if x]
+        if len(pred) > 0 or len(gt) > 0:
+            import Levenshtein
+            edit = Levenshtein.distance(gt, pred)/ max(len(pred), len(gt))
+            return {
+                'gt': gt,
+                'pred': pred,
+                'img_id': img_name,
+                'edit': edit
+            }
+        else:
+            return {}  # If both GT and pred are empty for the page, return empty
+
+    def formula_format(self, formula_matches, img_name):
+        # formated_list = []
+        for i, item in enumerate(formula_matches):
+            item["img_id"] = img_name + '_' + str(i)
+        return formula_matches
+
+    def get_matched_elements(self,references:list,predictions:list)->dict:
+        from .metrics import recogition_end2end_base_dataset, recogition_end2end_table_dataset
+
+        plain_text_match = []
+        display_formula_match = []
+        html_table_match = []
+        latex_table_match = []
+        order_match = []
+
+
+        for i,sample in enumerate(references):
+            img_name = os.path.basename(sample["page_info"]["image_path"])
+            pred_content = predictions[i]
+            result = self.process_get_matched_elements(sample, pred_content, img_name)
+            [plain_text_match_clean, formated_display_formula, latex_table_match_s, html_table_match_s, order_match_single] = result
+
+            if order_match_single:
+                order_match.append(order_match_single)
+            if plain_text_match_clean:
+                plain_text_match.extend(plain_text_match_clean)
+            if formated_display_formula:
+                display_formula_match.extend(formated_display_formula)
+            if latex_table_match_s:
+                latex_table_match.extend(latex_table_match_s)
+            if html_table_match_s:
+                html_table_match.extend(html_table_match_s)
+
+        if len(latex_table_match) > len(html_table_match):
+            table_match = latex_table_match
+            table_format = 'latex'
+        else:
+            table_match = html_table_match
+            table_format = 'html'
+
+        matched_samples_all = {
+            "text_block": recogition_end2end_base_dataset(plain_text_match),
+            "display_formula": recogition_end2end_base_dataset(display_formula_match),
+            "table": recogition_end2end_table_dataset(table_match, table_format),
+            "reading_order": recogition_end2end_base_dataset(order_match)
+        }
+
+        return matched_samples_all
+
+    def process_get_matched_elements(self, sample, pred_content, img_name):
+        from func_timeout import FunctionTimedOut, func_timeout
+
+        from .utils import (match_gt2pred_no_split, match_gt2pred_quick, match_gt2pred_simple,
+                            md_tex_filter)
+
+        if self.match_method == 'simple_match':   # add match choice
+            match_gt2pred = match_gt2pred_simple
+        elif self.match_method == 'quick_match':
+            match_gt2pred = match_gt2pred_quick
+        elif self.match_method == 'no_split':
+            match_gt2pred = match_gt2pred_no_split
+        else:
+            # print('Invalid match method name. The quick_match will be used.')
+            match_gt2pred = match_gt2pred_quick
+
+        pred_dataset = md_tex_filter(pred_content)
+        gt_page_elements = self.get_page_elements(sample)
+
+        text_all = self.get_page_elements_list(gt_page_elements, ['text_block', 'title', 'code_txt', 'code_txt_caption', 'reference', 'equation_caption',
+                                                'figure_caption', 'figure_footnote', 'table_caption', 'table_footnote', 'code_algorithm', 'code_algorithm_caption',
+                                                'header', 'footer', 'page_footnote', 'page_number'])
+
+
+        display_formula_match_s = []
+        plain_text_match_clean = []
+        latex_table_match_s = []
+        html_table_match_s = []
+        order_match_single = []
+        if text_all:
+            gt_text_list = self.get_sorted_text_list(text_all)
+            try:
+                plain_text_match_s = func_timeout(
+                    30, match_gt2pred, args=(gt_text_list, pred_dataset['text_all'], 'text', img_name)
+                )
+            except FunctionTimedOut as e1:
+                print(f'Time out for plain text match of {img_name}, match_gt2pred_simple will be used.')
+                plain_text_match_s = match_gt2pred_simple(gt_text_list, pred_dataset['text_all'], 'text', img_name)
+            except Exception as e:
+                print(str(e))
+                sys.exit()
+
+            if not plain_text_match_s:
+                print(f'No text match of {img_name}. The plain text match will be empty.')
+            else:
+                plain_text_match_clean = self.filtered_out_ignore(plain_text_match_s, ['figure_caption', 'figure_footnote', 'table_caption', 'table_footnote', 'code_algorithm', 'code_algorithm_caption', 'header', 'footer', 'page_footnote', 'page_number', 'equation_caption'])
+
+
+        if gt_page_elements.get('equation_isolated'):
+            gt_display_list = self.get_sorted_text_list(gt_page_elements['equation_isolated'])
+            display_formula_match_s = match_gt2pred(gt_display_list, pred_dataset['equation_isolated'], 'formula', img_name)
+            display_formula_match_s = [x for x in display_formula_match_s if x['gt_idx'] != [""]]
+            if not display_formula_match_s:
+                print(f'No display_formula_match of {img_name}. The display_formula_match will be empty.')
+
+        if gt_page_elements.get('table'):
+            gt_table_list = self.get_sorted_text_list(gt_page_elements['table'])
+            if pred_dataset['latex_table']:
+                latex_table_match_s = match_gt2pred_simple(gt_table_list, pred_dataset['latex_table'], 'latex_table', img_name)
+                latex_table_match_s = [x for x in latex_table_match_s if x['gt_idx'] != [""]]
+            if pred_dataset['html_table']:
+                html_table_match_s = match_gt2pred_simple(gt_table_list, pred_dataset['html_table'], 'html_table', img_name)
+                html_table_match_s = [x for x in html_table_match_s if x['gt_idx'] != [""]]
+            else:
+                html_table_match_s = match_gt2pred_simple(gt_table_list, [], 'html_table', img_name)
+                html_table_match_s = [x for x in html_table_match_s if x['gt_idx'] != [""]]
+
+
+        order_match_s = plain_text_match_clean
+        if order_match_s:
+            order_match_single = self.get_order_paired(order_match_s, img_name)
+
+        return [plain_text_match_clean, display_formula_match_s, latex_table_match_s, html_table_match_s, order_match_single]
+
+    def process_generated_metric_results(self,samples,save_name:str='end2end_quick_match'):
+        from .metrics import METRIC_REGISTRY, get_full_labels_results, get_page_split, show_result
+
+        result_all={}
+        page_info={}
+        metircs_dict=self.dafault_metircs_dict
+        pages=self.references #gt_samples list
+
+        for page in pages:
+                img_path=os.path.basename(page['page_info']['image_path'])
+                page_info[img_path]=page['page_info']['page_attribute']
+
+        for element in metircs_dict.keys():
+
+            result={}
+            group_info=metircs_dict[element].get('group',[])
+            # samples = samples.get(element) ##
+            cur_samples = samples[element]
+
+            for metric in metircs_dict[element]['metric']:
+                metric_val = METRIC_REGISTRY.get(metric)
+
+                cur_samples,result_s = metric_val(cur_samples).evaluate(group_info, f"{save_name}_{element}")
+                if result_s:
+                    result.update(result_s)
+
+            if result:
+                print(f"{element}")
+                show_result(result)
+            result_all[element]={}
+
+
+            group_result=get_full_labels_results(cur_samples)
+            page_result=get_page_split(cur_samples,page_info)
+
+            result_all[element]={
+                'all':result,
+                'group':group_result,
+                'page':page_result
+            }
+            if isinstance(cur_samples,list):
+                saved_samples=cur_samples
+            else:
+                saved_samples=cur_samples.samples
+            # NOTE: The original code has a bug here, it will overwrite the result file in each iteration.
+            # I will fix it by adding element to the filename.
+            # NOTE: Fixed typo .josn -> .json
+            result_file = get_intermediate_file_path(self.eval_file, f'_{save_name}_{element}_result', 'json')
+            dump(saved_samples, result_file)
+
+        metric_result_file = get_intermediate_file_path(self.eval_file, f'_{save_name}_metric_result', 'json')
+        dump(result_all, metric_result_file)
+
+        dict_list = []
+        save_dict={}
+        en_overall=[]
+        ch_overall=[]
+        for category_type, metric in [("text_block", "Edit_dist"), ("display_formula", "Edit_dist"), ("display_formula", "CDM"), ("table", "TEDS"), ("table", "Edit_dist"), ("reading_order", "Edit_dist")]:
+            if metric == 'CDM':
+                save_dict[category_type+'_'+metric+'_EN'] = '-'
+                save_dict[category_type+'_'+metric+'_CH'] = '-'
+            elif metric == "TEDS":
+                save_dict[category_type+'_'+metric+'_EN'] = result_all[category_type]["page"][metric]["language: english"] * 100
+                save_dict[category_type+'_'+metric+'_CH'] = result_all[category_type]["page"][metric]["language: simplified_chinese"] * 100
+            else:
+                save_dict[category_type+'_'+metric+'_EN'] = result_all[category_type]["page"][metric].get("language: english", np.nan)
+                save_dict[category_type+'_'+metric+'_CH'] = result_all[category_type]["page"][metric].get("language: simplified_chinese",np.nan)
+            if metric == "Edit_dist":
+                en_overall.append(result_all[category_type]["page"][metric].get("language: english", np.nan))
+                ch_overall.append(result_all[category_type]["page"][metric].get("language: simplified_chinese",np.nan))
+
+        save_dict['overall_EN'] = sum(en_overall) / len(en_overall)
+        save_dict['overall_CH'] = sum(ch_overall) / len(ch_overall)
+        dict_list.append(save_dict)
+        df = pd.DataFrame(dict_list,index=['end2end',]).round(3)
+
+        e2e_eval_file = get_intermediate_file_path(self.eval_file, '_End2End_Evaluation', 'json')
+        dump(result_all, e2e_eval_file)
+        
+        overall_file = get_intermediate_file_path(self.eval_file, '_overall')
+        dump(df, overall_file)
+
+        print(f"The save path of End2End_Evaluation is: {e2e_eval_file}")
+        print(f"The save path of overall metrics is: {overall_file}")
+        return df
+
+
+class table_evalutor():
+    def __init__(self,eval_file,tsv_path):
+        self.eval_file = eval_file
+        gt_key='html'
+        pred_key='pred'
+        self.category_filter='table'
+        self.category_type='table'
+        self.metircs_list=['TEDS','Edit_dist']
+        self.gt_samples,self.table_samples=self.load_data(eval_file,tsv_path,pred_key,gt_key)
+
+    def load_data(self,eval_file,gt_file,pred_key,gt_key):
+        from .data_preprocess import (clean_string, normalized_formula, normalized_table,
+                                      textblock2unicode)
+        samples=[]
+        preds=[]
+        predictions=load(eval_file)['prediction'].tolist()
+        gt_samples=load(gt_file)['answer'].tolist()
+        load_success,load_fail=0,0
+        for i,gt_sample in tqdm(enumerate(gt_samples),desc='Loading data'):
+            try:
+                ans=json.loads(gt_sample)
+                for item in ans['layout_dets']:
+                    if item['category_type']=="table":
+                        item['pred']=predictions[i]
+                        load_success+=1
+                        preds.append(ans)
+
+            except json.JSONDecodeError as e:
+                load_fail+=1
+                continue
+        print(f'load_table_success:{load_success},load_table_fail:{load_fail}')
+
+        count=0
+        for pred in preds:
+            img_name = os.path.basename(pred['page_info']['image_path'])
+            for i, ann in enumerate(pred['layout_dets']):
+                if not ann.get(gt_key):
+                    continue
+                if self.category_filter:
+                    if ann['category_type'] not in self.category_filter:
+                        continue
+                if not ann.get(pred_key):
+                    # print(f'Cannot find pred for {img_name}. ann is {ann}')
+                    # pdb.set_trace()
+                    count += 1
+                    continue
+                else:
+                    gt_text = ann[gt_key]
+                    norm_gt = gt_text
+                    pred_text = ann[pred_key]
+                    norm_pred = pred_text
+                    if self.category_type:
+                        if self.category_type == 'text':
+                            norm_gt = clean_string(textblock2unicode(ann[gt_key]))
+                            norm_pred = clean_string(textblock2unicode(ann[pred_key]))
+                        elif self.category_type == 'formula':
+                            norm_gt = normalized_formula(ann[gt_key])
+                            norm_pred = normalized_formula(ann[pred_key])
+                        elif self.category_type == 'table':
+                            norm_gt = normalized_table(ann[gt_key], gt_key)
+                            norm_pred = normalized_table(ann[pred_key], gt_key)
+                        else:
+                            raise ValueError(f'Invalid category type: {self.category_type}')
+
+                samples.append({
+                    "gt": gt_text,
+                    "norm_gt": norm_gt,
+                    "gt_attribute": [ann['attribute']],
+                    'pred': pred_text,
+                    "norm_pred": norm_pred,
+                    'img_id': img_name
+                })
+
+        print(f'Cannot find pred for {count} samples.')
+        return preds,samples
+
+    def score(self)->dict:
+        metrics=self.process_generated_metric_results()
+        return metrics
+
+    def process_generated_metric_results(self,save_name:str='OmniDocBench_table'):
+        from .metrics import METRIC_REGISTRY, get_full_labels_results, get_page_split, show_result
+
+        p_scores={}
+        page_info={}
+        no_page_flag=False
+        samples=self.table_samples
+        pages=self.gt_samples
+
+        for page in pages:
+            if 'page_info' not in page:
+                no_page_flag=True
+                break
+            img_path=os.path.basename(page['page_info']['image_path'])
+            page_info[img_path]=page['page_info']['page_attribute']
+
+        for metric in self.metircs_list:
+            metric_val=METRIC_REGISTRY.get(metric)
+            samples, result = metric_val(samples).evaluate({}, save_name)
+            if result:
+                p_scores.update(result)
+        show_result(p_scores)
+        group_result=get_full_labels_results(samples)
+        if no_page_flag:
+            page_result={}
+        else:
+            page_result=get_page_split(samples,page_info)
+
+        result_all={
+            'all':p_scores,
+            'group':group_result,
+            'page':page_result
+        }
+
+        metric_result_file = get_intermediate_file_path(self.eval_file, f'_{save_name}_metric_result', 'json')
+        dump(result_all, metric_result_file)
+
+        dict_list=[]
+        dict_list.append(result_all["group"]["TEDS"])
+
+        df4 = pd.DataFrame(dict_list, index=['OmniDocBench_table'])
+        df4 = df4 * 100
+        df4 = df4.round(1)
+        selected_columns = df4[["language: table_en", "language: table_simplified_chinese", "language: table_en_ch_mixed", "line: full_line", "line: less_line", "line: fewer_line", "line: wireless_line",
+                        "with_span: True", "with_span: False", "include_equation: True", "include_equation: False", "include_background: True", "include_background: False", "table_layout: vertical", "table_layout: horizontal"]]
+
+        table_attr_file = get_intermediate_file_path(self.eval_file, '_table_attribute')
+        dump(selected_columns, table_attr_file)
+        print(f'The save path of table_attribute is :{table_attr_file}')
+        return selected_columns

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/OmniDocBench/requirements.txt

@@ -0,0 +1,13 @@
+accelerate>=0.26.0
+apted
+BeautifulSoup4
+evaluate
+func_timeout
+jmespath
+Levenshtein
+lxml
+nltk
+pylatexenc
+qwen_vl_utils
+scipy
+torchvision

reference/code/Fast-dLLM/third_party/VLMEvalKit/vlmeval/dataset/OmniDocBench/utils.py

@@ -0,0 +1,1918 @@
+import copy
+import html
+import json
+import os
+import pdb
+import re
+import shutil
+import subprocess
+import sys
+import unicodedata
+import uuid
+from collections import defaultdict
+
+import Levenshtein
+import numpy as np
+from bs4 import BeautifulSoup
+from pylatexenc.latex2text import LatexNodes2Text
+from pylatexenc.latexencode import unicode_to_latex
+from pylatexenc.latexwalker import (LatexCharsNode, LatexEnvironmentNode, LatexGroupNode,
+                                    LatexMacroNode, LatexSpecialsNode, LatexWalker)
+from scipy.optimize import linear_sum_assignment
+
+
+def read_md_file(filepath):
+    with open(filepath, 'r', encoding='utf-8') as file:
+        content = file.read()
+
+    return content
+
+def save_paired_result(preds, gts, save_path):
+    save_result = []
+    formula_id = 0
+    for gt, pred in zip(gts, preds):
+        save_result.append({
+            "gt": gt,
+            "pred": pred,
+            "img_id": formula_id
+        })
+        formula_id += 1
+    with open(save_path, 'w', encoding='utf-8') as f:
+        json.dump(save_result, f, indent=4, ensure_ascii=False)
+
+def remove_markdown_fences(content):
+    content = re.sub(r'^```markdown\n?', '', content, flags=re.MULTILINE)
+    content = re.sub(r'```\n?$', '', content, flags=re.MULTILINE)
+    return content
+
+# Standardize all consecutive characters
+def replace_repeated_chars(input_str):
+    input_str = re.sub(r'_{4,}', '____', input_str) # Replace more than 4 consecutive underscores with 4 underscores
+    input_str = re.sub(r' {4,}', '    ', input_str)   # Replace more than 4 consecutive spaces with 4 spaces
+    return re.sub(r'([^a-zA-Z0-9])\1{10,}', r'\1\1\1\1', input_str) # For other consecutive symbols (except numbers and letters), replace more than 10 occurrences with 4
+
+# Special Unicode handling
+def fullwidth_to_halfwidth(s):
+    result = []
+    for char in s:
+        code = ord(char)
+        # Convert full-width space to half-width space
+        if code == 0x3000:
+            code = 0x0020
+        # Convert other full-width characters to half-width
+        elif 0xFF01 <= code <= 0xFF5E:
+            code -= 0xFEE0
+        result.append(chr(code))
+    return ''.join(result)
+
+def find_special_unicode(s):
+    special_chars = {}
+    for char in s:
+        if ord(char) > 127:  # Non-ASCII characters
+            # unicode_name = unicodedata.name(char, None)
+            unicode_name = unicodedata.category(char)
+            special_chars[char] = f'U+{ord(char):04X} ({unicode_name})'
+    return special_chars
+
+
+inline_reg = re.compile(
+    r'\$(.*?)\$|'
+    r'\\\((.*?)\\\)',
+)
+
+def textblock2unicode(text):
+    inline_matches = inline_reg.finditer(text)
+    removal_positions = []
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        # print('-------- content-------', content)
+        # Remove escape characters \
+        clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+        try:
+            if any(char in clean_content for char in r'\^_'):
+                if clean_content.endswith('\\'):
+                    clean_content += ' '
+                # inline_array.append(match.group(0))
+                unicode_content = LatexNodes2Text().latex_to_text(clean_content)
+                removal_positions.append((position[0], position[1], unicode_content))
+        except:
+            continue
+
+    # Remove inline formulas from original text
+    for start, end, unicode_content in sorted(removal_positions, reverse=True):
+        text = text[:start] + unicode_content.strip() + text[end:]
+
+    return text
+
+def normalized_formula(text):
+    # Normalize math formulas before matching
+    filter_list = ['\\mathbf', '\\mathrm', '\\mathnormal', '\\mathit', '\\mathbb', '\\mathcal', '\\mathscr', '\\mathfrak', '\\mathsf', '\\mathtt',
+                   '\\textbf', '\\text', '\\boldmath', '\\boldsymbol', '\\operatorname', '\\bm',
+                   '\\symbfit', '\\mathbfcal', '\\symbf', '\\scriptscriptstyle', '\\notag',
+                   '\\setlength', '\\coloneqq', '\\space', '\\thickspace', '\\thinspace', '\\medspace', '\\nobreakspace', '\\negmedspace',
+                   '\\quad', '\\qquad', '\\enspace', '\\substackw', ' ']
+                #    '\\left', '\\right', '{', '}', ' ']
+
+    # delimiter_filter
+    pattern = re.compile(r"\\\[(.+?)(?<!\\)\\\]")
+    match = pattern.search(text)
+
+    if match:
+        text = match.group(1).strip()
+
+    tag_pattern = re.compile(r"\\tag\{.*?\}")
+    text = tag_pattern.sub('', text)
+    hspace_pattern = re.compile(r"\\hspace\{.*?\}")
+    text = hspace_pattern.sub('', text)
+    begin_pattern = re.compile(r"\\begin\{.*?\}")
+    text = begin_pattern.sub('', text)
+    end_pattern = re.compile(r"\\end\{.*?\}")
+    text = end_pattern.sub('', text)
+    col_sep = re.compile(r"\\arraycolsep.*?\}")
+    text = col_sep.sub('', text)
+    text = text.strip('.')
+
+    for filter_text in filter_list:
+        text = text.replace(filter_text, '')
+
+    # text = normalize_text(delimiter_filter(text))
+    # text = delimiter_filter(text)
+    text = text.lower()
+    return text
+
+def normalized_html_table(text):
+    def process_table_html(md_i):
+        """
+        pred_md format edit
+        """
+        def process_table_html(html_content):
+            soup = BeautifulSoup(html_content, 'html.parser')
+            th_tags = soup.find_all('th')
+            for th in th_tags:
+                th.name = 'td'
+            thead_tags = soup.find_all('thead')
+            for thead in thead_tags:
+                thead.unwrap()  # unwrap()会移除标签但保留其内容
+            math_tags = soup.find_all('math')
+            for math_tag in math_tags:
+                alttext = math_tag.get('alttext', '')
+                alttext = f'${alttext}$'
+                if alttext:
+                    math_tag.replace_with(alttext)
+            span_tags = soup.find_all('span')
+            for span in span_tags:
+                span.unwrap()
+            return str(soup)
+
+        table_res=''
+        table_res_no_space=''
+        if '<table' in md_i.replace(" ","").replace("'",'"'):
+            md_i = process_table_html(md_i)
+            table_res = html.unescape(md_i).replace('\n', '')
+            table_res = unicodedata.normalize('NFKC', table_res).strip()
+            pattern = r'<table\b[^>]*>(.*)</table>'
+            tables = re.findall(pattern, table_res, re.DOTALL | re.IGNORECASE)
+            table_res = ''.join(tables)
+            # table_res = re.sub('<table.*?>','',table_res)
+            table_res = re.sub('( style=".*?")', "", table_res)
+            table_res = re.sub('( height=".*?")', "", table_res)
+            table_res = re.sub('( width=".*?")', "", table_res)
+            table_res = re.sub('( align=".*?")', "", table_res)
+            table_res = re.sub('( class=".*?")', "", table_res)
+            table_res = re.sub('</?tbody>',"",table_res)
+
+            table_res = re.sub(r'\s+', " ", table_res)
+            table_res_no_space = '<html><body><table border="1" >' + table_res.replace(' ','') + '</table></body></html>'
+            # table_res_no_space = re.sub(' (style=".*?")',"",table_res_no_space)
+            # table_res_no_space = re.sub(r'[ ]', " ", table_res_no_space)
+            table_res_no_space = re.sub('colspan="', ' colspan="', table_res_no_space)
+            table_res_no_space = re.sub('rowspan="', ' rowspan="', table_res_no_space)
+            table_res_no_space = re.sub('border="', ' border="', table_res_no_space)
+
+            table_res = '<html><body><table border="1" >' + table_res + '</table></body></html>'
+            # table_flow.append(table_res)
+            # table_flow_no_space.append(table_res_no_space)
+
+        return table_res, table_res_no_space
+
+    def clean_table(input_str,flag=True):
+        if flag:
+            input_str = input_str.replace('<sup>', '').replace('</sup>', '')
+            input_str = input_str.replace('<sub>', '').replace('</sub>', '')
+            input_str = input_str.replace('<span>', '').replace('</span>', '')
+            input_str = input_str.replace('<div>', '').replace('</div>', '')
+            input_str = input_str.replace('<p>', '').replace('</p>', '')
+            input_str = input_str.replace('<spandata-span-identity="">', '')
+            input_str = re.sub('<colgroup>.*?</colgroup>','',input_str)
+        return input_str
+
+    norm_text, _ = process_table_html(text)
+    norm_text = clean_table(norm_text)
+    return norm_text
+
+def normalized_latex_table(text):
+    def latex_template(latex_code):
+        template = r'''
+        \documentclass[border=20pt]{article}
+        \usepackage{subcaption}
+        \usepackage{url}
+        \usepackage{graphicx}
+        \usepackage{caption}
+        \usepackage{multirow}
+        \usepackage{booktabs}
+        \usepackage{color}
+        \usepackage{colortbl}
+        \usepackage{xcolor,soul,framed}
+        \usepackage{fontspec}
+        \usepackage{amsmath,amssymb,mathtools,bm,mathrsfs,textcomp}
+        \setlength{\parindent}{0pt}''' + \
+        r'''
+        \begin{document}
+        ''' + \
+        latex_code + \
+        r'''
+        \end{document}'''
+
+        return template
+
+    def process_table_latex(latex_code):
+        SPECIAL_STRINGS= [
+            ['\\\\vspace\\{.*?\\}', ''],
+            ['\\\\hspace\\{.*?\\}', ''],
+            ['\\\\rule\{.*?\\}\\{.*?\\}', ''],
+            ['\\\\addlinespace\\[.*?\\]', ''],
+            ['\\\\addlinespace', ''],
+            ['\\\\renewcommand\\{\\\\arraystretch\\}\\{.*?\\}', ''],
+            ['\\\\arraystretch\\{.*?\\}', ''],
+            ['\\\\(row|column)?colors?\\{[^}]*\\}(\\{[^}]*\\}){0,2}', ''],
+            ['\\\\color\\{.*?\\}', ''],
+            ['\\\\textcolor\\{.*?\\}', ''],
+            ['\\\\rowcolor(\\[.*?\\])?\\{.*?\\}', ''],
+            ['\\\\columncolor(\\[.*?\\])?\\{.*?\\}', ''],
+            ['\\\\cellcolor(\\[.*?\\])?\\{.*?\\}', ''],
+            ['\\\\colorbox\\{.*?\\}', ''],
+            ['\\\\(tiny|scriptsize|footnotesize|small|normalsize|large|Large|LARGE|huge|Huge)', ''],
+            [r'\s+', ' '],
+            ['\\\\centering', ''],
+            ['\\\\begin\\{table\\}\\[.*?\\]', '\\\\begin{table}'],
+            ['\t', ''],
+            ['@{}', ''],
+            ['\\\\toprule(\\[.*?\\])?', '\\\\hline'],
+            ['\\\\bottomrule(\\[.*?\\])?', '\\\\hline'],
+            ['\\\\midrule(\\[.*?\\])?', '\\\\hline'],
+            ['p\\{[^}]*\\}', 'l'],
+            ['m\\{[^}]*\\}', 'c'],
+            ['\\\\scalebox\\{[^}]*\\}\\{([^}]*)\\}', '\\1'],
+            ['\\\\textbf\\{([^}]*)\\}', '\\1'],
+            ['\\\\textit\\{([^}]*)\\}', '\\1'],
+            ['\\\\cmidrule(\\[.*?\\])?\\(.*?\\)\\{([0-9]-[0-9])\\}', '\\\\cline{\\2}'],
+            ['\\\\hline', ''],
+            [r'\\multicolumn\{1\}\{[^}]*\}\{((?:[^{}]|(?:\{[^{}]*\}))*)\}', r'\1']
+        ]
+        pattern = r'\\begin\{tabular\}.*\\end\{tabular\}'  # 注意这里不用 .*?
+        matches = re.findall(pattern, latex_code, re.DOTALL)
+        latex_code = ' '.join(matches)
+
+        for special_str in SPECIAL_STRINGS:
+            latex_code = re.sub(fr'{special_str[0]}', fr'{special_str[1]}', latex_code)
+
+        return latex_code
+
+    def convert_latex_to_html(latex_content, cache_dir='./temp'):
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+
+        uuid_str = str(uuid.uuid1())
+        with open(f'{cache_dir}/{uuid_str}.tex', 'w') as f:
+            f.write(latex_template(latex_content))
+
+        cmd = ['latexmlc', '--quiet', '--nocomments', f'--log={cache_dir}/{uuid_str}.log',
+               f'{cache_dir}/{uuid_str}.tex', f'--dest={cache_dir}/{uuid_str}.html']
+        try:
+            subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+            with open(f'{cache_dir}/{uuid_str}.html', 'r') as f:
+                html_content = f.read()
+
+            pattern = r'<table\b[^>]*>(.*)</table>'
+            tables = re.findall(pattern, html_content, re.DOTALL | re.IGNORECASE)
+            tables = [f'<table>{table}</table>' for table in tables]
+            html_content = '\n'.join(tables)
+
+        except Exception as e:
+            html_content = ''
+
+        shutil.rmtree(cache_dir)
+        return html_content
+
+    html_text = convert_latex_to_html(text)
+    normlized_tables = normalized_html_table(html_text)
+    return normlized_tables
+
+
+def normalized_table(text, format='html'):
+    if format not in ['html', 'latex']:
+        raise ValueError('Invalid format: {}'.format(format))
+    else:
+        return globals()['normalized_{}_table'.format(format)](text)
+
+
+def textblock_with_norm_formula(text):
+    inline_matches = inline_reg.finditer(text)
+    removal_positions = []
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        # print('-------- content-------', content)
+
+        norm_content = normalized_formula(content)
+        removal_positions.append((position[0], position[1], norm_content))
+
+    # Remove inline formulas from original text
+    for start, end, norm_content in sorted(removal_positions, reverse=True):
+        text = text[:start] + norm_content.strip() + text[end:]
+
+    return text
+
+
+def inline_filter_unicode(text):
+    # Ensure text is string type
+    if not isinstance(text, str):
+        text = str(text)
+
+    # Replace inline formula boundary markers
+    #print('--------text-------',text)
+    placeholder = '__INLINE_FORMULA_BOUNDARY__'
+    text_copy = text.replace('$', placeholder).replace('\\(', placeholder).replace('\\)', placeholder)
+    #print('--------text_copy-------',text_copy)
+    # Convert LaTeX content to Unicode representation
+    text_copy = LatexNodes2Text().latex_to_text(text_copy)
+    #print('--------text_copy---unicode----',text_copy)
+    # Restore boundary markers
+    text_copy = text_copy.replace(placeholder, '$')
+
+    inline_array = []
+    inline_matches = inline_reg.finditer(text_copy)
+    # Record positions of inline formulas to be removed
+    removal_positions = []
+
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        print('-------- content-------', content)
+        # Remove escape characters \
+        clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+        if any(char in clean_content for char in r'\^_'):
+            # inline_array.append(match.group(0))
+            inline_array.append({
+                'category_type': 'equation_inline',
+                'position': position,
+                'content': content,
+            })
+            removal_positions.append((position[0], position[1]))
+
+    # Remove inline formulas from original text
+    for start, end in sorted(removal_positions, reverse=True):
+        text = text[:start] + text[end:]
+
+    return text, inline_array
+
+def inline_filter(text):
+    # Ensure text is string type
+    if not isinstance(text, str):
+        text = str(text)
+
+    inline_array = []
+    inline_matches = inline_reg.finditer(text)
+
+    for match in inline_matches:
+        position = [match.start(), match.end()]
+        content = match.group(1) if match.group(1) is not None else match.group(2)
+        # print('inline_content: ', content)
+
+        # Remove escape characters \
+        clean_content = re.sub(r'\\([\\_&%^])', '', content)
+
+        if any(char in clean_content for char in r'\^_'):
+            # inline_array.append(match.group(0))
+            inline_array.append({
+                'category_type': 'equation_inline',
+                'position': position,
+                'content': match.group(0),
+            })
+            text = text.replace(match.group(0), '')
+            # print('-----Found inline formula: ', match.group(0))
+        else:
+            text = text.replace(match.group(0), content)
+
+    return text, inline_array
+
+# Text OCR quality check processing:
+def clean_string(input_string):
+    # Use regex to keep Chinese characters, English letters and numbers
+    input_string = input_string.replace('\\t', '').replace('\\n', '').replace('\t', '').replace('\n', '').replace('/t', '').replace('/n', '')
+    cleaned_string = re.sub(r'[^\w\u4e00-\u9fff]', '', input_string)
+    return cleaned_string
+
+def extract_tabular(text):
+    begin_pattern = r'\\begin{tabular}'
+    end_pattern = r'\\end{tabular}'
+
+    tabulars = []
+    positions = []
+    current_pos = 0
+    stack = []
+
+    while current_pos < len(text):
+        begin_match = re.search(begin_pattern, text[current_pos:])
+        end_match = re.search(end_pattern, text[current_pos:])
+
+        if not begin_match and not end_match:
+            break
+
+        if begin_match and (not end_match or begin_match.start() < end_match.start()):
+            stack.append(current_pos + begin_match.start())
+            current_pos += begin_match.start() + len(end_pattern)
+        elif end_match:
+            if stack:
+                start_pos = stack.pop()
+                if not stack:
+                    end_pos = current_pos + end_match.start() + len(end_pattern)
+                    tabular_code = text[start_pos:end_pos]
+                    tabulars.append(tabular_code)
+                    positions.append((start_pos, end_pos))
+            current_pos += end_match.start() + len(end_pattern)
+        else:
+            current_pos += 1
+
+    if stack:
+        new_start = stack[0] + len(begin_pattern)
+        new_tabulars, new_positions = extract_tabular(text[new_start:])
+        new_positions = [(start + new_start, end + new_start) for start, end in new_positions]
+        tabulars.extend(new_tabulars)
+        positions.extend(new_positions)
+
+    return tabulars, positions
+
+# math reg
+    # r'\\begin{equation\*?}(.*?)\\end{equation\*?}|'
+    # r'\\begin{align\*?}(.*?)\\end{align\*?}|'
+    # r'\\begin{gather\*?}(.*?)\\end{gather\*?}|'
+display_reg = re.compile(
+    r'\$\$(.*?)\$\$|'
+    r'\\\[(.*?)\\\]|'
+    r'\$(.*?)\$|'
+    r'\\\((.*?)\\\)',
+    re.DOTALL
+)
+
+# inline_reg = re.compile(
+#     r'(?<!\$)\$(?!\$)(.*?)(?<!\$)\$(?!\$)|'
+#     r'\\\((.*?)\\\)',
+# )
+inline_reg = re.compile(
+    r'\$(.*?)\$|'
+    r'\\\((.*?)\\\)',
+)
+
+# table
+table_reg = re.compile(
+    r'\\begin{table\*?}(.*?)\\end{table\*?}|'
+    r'\\begin{tabular\*?}(.*?)\\end{tabular\*?}',
+    re.DOTALL
+)
+md_table_reg = re.compile(
+    r'\|\s*.*?\s*\|\n',
+    re.DOTALL)
+html_table_reg = re.compile(
+    r'(<table.*?</table>)',
+    re.DOTALL
+)
+
+# title
+title_reg = re.compile(
+    r'^\s*#.*$',
+    re.MULTILINE)
+
+# img
+img_pattern = r'!\[.*?\]\(.*?\)'
+
+# code block
+code_block_reg = re.compile(
+    r'```(\w+)\n(.*?)```',
+    re.DOTALL
+)
+
+
+def md_tex_filter(content):
+    '''
+    Input: 1 page md or tex content - String
+    Output: text, display, inline, table, title, code - list
+    '''
+    content = re.sub(img_pattern, '', content)  # remove image
+    content = remove_markdown_fences(content)   # remove markdown fences
+    content = replace_repeated_chars(content) # replace all consecutive characters
+
+
+
+    pred_all = []
+    latex_table_array, table_positions = extract_tex_table(content)
+    for latex_table, position in zip(latex_table_array, table_positions):
+        position = [position[0], position[0]+len(latex_table)]   # !!!
+        pred_all.append({
+            'category_type': 'latex_table',
+            'position': position,
+            'content': latex_table
+        })
+        content = content[:position[0]] + ' '*(position[1]-position[0]) + content[position[1]:]  # replace latex table with space
+
+
+    # extract html table
+    html_table_array, table_positions = extract_html_table(content)
+    for html_table, position in zip(html_table_array, table_positions):
+        position = [position[0], position[0]+len(html_table)]
+        pred_all.append({
+            'category_type': 'html_table',
+            'position': position,
+            'content': html_table
+        })
+        content = content[:position[0]] + ' '*(position[1]-position[0]) + content[position[1]:]  # replace html table with space
+
+    # extract interline formula
+    display_matches = display_reg.finditer(content)
+    for match in display_matches:
+        matched = match.group(0)
+        if matched:
+            single_line = ''.join(matched.split())
+            position = [match.start(), match.end()]
+            # replace $$ with \[\]
+            dollar_pattern = re.compile(r'\$\$(.*?)\$\$|\$(.*?)\$|\\\((.*?)\\\)', re.DOTALL)
+            sub_match = dollar_pattern.search(single_line)
+            if sub_match is None:
+                # pass
+                content = content[:position[0]] + ' '*(position[1]-position[0]) + content[position[1]:]
+                pred_all.append({
+                    'category_type': 'equation_isolated',
+                    'position': position,
+                    'content': single_line
+                })
+            elif sub_match.group(1):
+                single_line = re.sub(dollar_pattern, r'\\[\1\\]', single_line)
+                content = content[:position[0]] + ' '*(position[1]-position[0]) + content[position[1]:]  # replace equation with space
+                pred_all.append({
+                    'category_type': 'equation_isolated',
+                    'position': position,
+                    'content': single_line
+                })
+            else:
+                single_line = re.sub(dollar_pattern, r'\\[\2\3\\]', single_line)
+                pred_all.append({
+                    'category_type': 'equation_isolated',
+                    'position': position,
+                    'content': single_line,
+                    'fine_category_type': 'equation_inline'
+                })
+
+
+    # extract md table with ||
+    md_table_mathces = md_table_reg.findall(content+'\n')
+    if len(md_table_mathces) >= 2:
+        # print("md table found!")
+        # print("content:", content)
+        content = convert_markdown_to_html(content)
+        # print('----------content after converting md table to html:', content)
+        html_table_matches = html_table_reg.finditer(content)
+        if html_table_matches:
+            for match in html_table_matches:
+                matched = match.group(0)
+                position = [match.start(), match.end()]
+                # content = content.replace(match, '')
+                # print('content after removing the md table:', content)
+                content = content[:position[0]] + ' '*(position[1]-position[0]) + content[position[1]:]  # replace md table with space
+                pred_all.append({
+                    'category_type': 'html_table',
+                    'position': position,
+                    'content': matched.strip(),
+                    'fine_category_type': 'md2html_table'
+                })
+    # print('---------After md table: \n', content)
+
+    # extract code blocks
+    code_matches = code_block_reg.finditer(content)
+    if code_matches:
+        for match in code_matches:
+            position = [match.start(), match.end()]
+            language = match.group(1)
+            code = match.group(2).strip()
+            # content = content.replace(match.group(0), '')
+            content = content[:position[0]] + ' '*(position[1]-position[0]) + content[position[1]:]  # replace code block with space
+            pred_all.append({
+                'category_type': 'text_all',
+                'position': position,
+                'content': code,
+                'language': language,
+                'fine_category_type': 'code'
+            })
+
+
+    # Remove latex style
+    content = re.sub(r'\\title\{(.*?)\}', r'\1', content)
+    content = re.sub(r'\\title\s*\{\s*(.*?)\s*\}', r'\1', content, flags=re.DOTALL)
+    content = re.sub(r'\\text\s*\{\s*(.*?)\s*\}', r'\1', content, flags=re.DOTALL)
+    content = re.sub(r'\\section\*?\{(.*?)\}', r'\1', content)
+    content = re.sub(r'\\section\*?\{\s*(.*?)\s*\}', r'\1', content, flags=re.DOTALL)
+
+    # extract texts
+    res = content.split('\n\n')
+    if len(res) == 1:
+        res = content.split('\n')  # some models do not use double newlines, so use single newlines to split
+
+    content_position = 0
+    for text in res:
+        position = [content_position, content_position+len(text)]
+        content_position += len(text)
+        text = text.strip()
+        text = text.strip('\n')
+        # print('ori_text: ', text)
+        text = '\n'.join([_.strip() for _ in text.split('\n') if _.strip()])   # avoid some single newline content with many spaces
+        # print('after strip text: ', text)
+
+        if text:  # Check if the stripped text is not empty
+            if text.startswith('<table') and text.endswith('</table>'):
+                pred_all.append({
+                    'category_type': 'html_table',
+                    'position': position,
+                    'content': text,
+                })
+
+            elif text.startswith('$') and text.endswith('$'):
+                if text.replace('$', '').strip():
+                    pred_all.append({
+                        'category_type': 'equation_isolated',
+                        'position': position,
+                        'content': text.strip(),
+                    })
+            else:
+                text = text.strip()
+                if text:
+                    pred_all.append({
+                        'category_type': 'text_all',
+                        'position': position,
+                        'content': text,
+                        'fine_category_type': 'text_block'
+                    })
+
+    pred_dataset = defaultdict(list)
+    pred_all = sorted(pred_all, key=lambda x: x['position'][0])
+    for item in pred_all:
+        pred_dataset[item['category_type']].append(item)
+    # pdb.set_trace()
+    return pred_dataset
+
+
+def extract_tex_table(content):
+    tables = []
+    tables_positions = []
+
+    pattern = r'\\begin{table}(.*?)\\end{table}'
+    for match in re.finditer(pattern, content, re.DOTALL):
+        start_pos = match.start()
+        end_pos = match.end()
+        table_content = match.group(0)
+        tables.append(table_content)
+        tables_positions.append((start_pos, end_pos))
+        content = content[:start_pos] + ' '*(end_pos-start_pos) + content[end_pos:]
+
+    tabulars, tabular_positions = extract_tabular(content)
+    all_tables = tables + tabulars
+    all_positions = tables_positions + tabular_positions
+
+    all_result = sorted([[pos, table]for pos, table in zip(all_positions, all_tables)], key=lambda x: x[0][0])
+    all_tables = [x[1] for x in all_result]
+    all_positions = [x[0] for x in all_result]
+
+    return all_tables, all_positions
+
+
+def extract_html_table(text):
+    begin_pattern = r'<table(?:[^>]*)>'
+    end_pattern = r'</table>'
+
+    tabulars = []
+    positions = []
+    current_pos = 0
+    stack = []
+
+    while current_pos < len(text):
+        begin_match = re.search(begin_pattern, text[current_pos:])
+        end_match = re.search(end_pattern, text[current_pos:])
+
+        if not begin_match and not end_match:
+            break
+
+        if begin_match and (not end_match or begin_match.start() < end_match.start()):
+            stack.append(current_pos + begin_match.start())
+            current_pos += begin_match.start() + len(end_pattern)
+        elif end_match:
+            if stack:
+                start_pos = stack.pop()
+                if not stack:
+                    end_pos = current_pos + end_match.start() + len(end_pattern)
+                    tabular_code = text[start_pos:end_pos]
+                    tabulars.append(tabular_code)
+                    positions.append((start_pos, end_pos))
+            current_pos += end_match.start() + len(end_pattern)
+        else:
+            current_pos += 1
+
+    if stack:
+        new_start = stack[0] + len(begin_pattern)
+        new_tabulars, new_positions = extract_html_table(text[new_start:])
+        new_positions = [(start + new_start, end + new_start) for start, end in new_positions]
+        tabulars.extend(new_tabulars)
+        positions.extend(new_positions)
+
+    return tabulars, positions
+
+
+def extract_node_content(node):
+    """ Recursively extract content from LatexEnvironmentNode and rebuild LaTeX table representation """
+    if isinstance(node, LatexCharsNode):
+        return node.chars  # Use chars attribute
+    elif isinstance(node, LatexGroupNode):
+        return "{" + "".join(extract_node_content(n) for n in node.nodelist) + "}"
+    elif isinstance(node, LatexMacroNode):
+        # Extract macro command and its arguments
+        macro_content = "\\" + node.macroname
+        if node.nodeargs:
+            macro_content += "".join([extract_node_content(arg) for arg in node.nodeargs])
+        return macro_content
+    elif isinstance(node, LatexEnvironmentNode):
+        # Extract environment, preserve environment name and arguments
+        content = "\\begin{" + node.environmentname + "}"
+        if node.nodeargd and node.nodeargd.argnlist:
+            # content += "".join("{" + extract_node_content(arg) + "}" for arg in node.nodeargd)
+            # content += "".join("{" + extract_node_content(node.nodeargd) + "}")
+            content += "{" + extract_node_content(node.nodeargd.argnlist[0]) + "}"
+        if node.nodelist:
+            content += "".join(extract_node_content(n) for n in node.nodelist)
+        content += "\\end{" + node.environmentname + "}"
+        return content
+    elif isinstance(node, LatexSpecialsNode):  # Changed to LatexSpecialsNode
+        return node.specials_chars
+    else:
+        return ""
+
+def get_node_end_pos(node):
+    """Recursively determine the end position of a node"""
+    if hasattr(node, 'nodelist') and node.nodelist:
+        # If the node has child nodes, recursively find the end position of the last child node
+        return get_node_end_pos(node.nodelist[-1])
+    elif hasattr(node, 'pos_end'):
+        # If the node has pos_end attribute, return it directly
+        return node.pos_end
+    else:
+        # If there are no child nodes, assume the node ends at the last character of its content
+        return node.pos + len(str(node))
+
+def remove_tex_table(content):
+    tables, positions = extract_tex_table(content)
+
+    # Delete in reverse order by position to avoid affecting unprocessed start positions
+    for start, end in sorted(positions, reverse=True):
+        content = content[:start] + content[end:]  # Remove table content
+
+    return content
+
+
+
+def get_pred_category_type(pred_idx, pred_items):
+    # if pred_idx:
+    if pred_items[pred_idx].get('fine_category_type'):
+        pred_pred_category_type = pred_items[pred_idx]['fine_category_type']
+    else:
+        pred_pred_category_type = pred_items[pred_idx]['category_type']
+    # else:
+    #     pred_pred_category_type = ""
+    return pred_pred_category_type
+
+
+def compute_edit_distance_matrix_new(gt_lines, matched_lines):
+    try:
+        distance_matrix = np.zeros((len(gt_lines), len(matched_lines)))
+        for i, gt_line in enumerate(gt_lines):
+            for j, matched_line in enumerate(matched_lines):
+                if len(gt_line) == 0 and len(matched_line) == 0:
+                    distance_matrix[i][j] = 0
+                else:
+                    distance_matrix[i][j] = Levenshtein.distance(gt_line, matched_line) / max(len(matched_line), len(gt_line))
+        return distance_matrix
+    except ZeroDivisionError:
+        #print("ZeroDivisionError occurred. Outputting norm_gt_lines and norm_pred_lines:")
+        # print("norm_gt_lines:", gt_lines)
+        # print("norm_pred_lines:", matched_lines)
+        raise
+
+def get_gt_pred_lines(gt_items, pred_items, line_type):
+    norm_html_lines = []
+    gt_lines = []
+    gt_cat_list = []
+    for item in gt_items:
+        if item.get('fine_category_type'):
+            gt_cat_list.append(item['fine_category_type'])
+        else:
+            gt_cat_list.append(item['category_type'])
+        if item.get('content'):
+            gt_lines.append(str(item['content']))
+            norm_html_lines.append(str(item['content']))
+        elif line_type == 'text':
+            gt_lines.append(str(item['text']))
+        elif line_type == 'html_table':
+            gt_lines.append(str(item['html']))
+        elif line_type == 'formula':
+            gt_lines.append(str(item['latex']))
+        elif line_type == 'latex_table':
+            gt_lines.append(str(item['latex']))
+            norm_html_lines.append(str(item['html']))
+
+    pred_lines = [str(item['content']) for item in pred_items]
+
+
+    if line_type == 'formula':
+        norm_gt_lines = [normalized_formula(_) for _ in gt_lines]
+        norm_pred_lines = [normalized_formula(_) for _ in pred_lines]
+    elif line_type == 'text':
+        # norm_gt_lines = [textblock_with_norm_formula(_) for _ in gt_lines]
+        # norm_pred_lines = [textblock_with_norm_formula(_) for _ in pred_lines]
+        norm_gt_lines = [clean_string(textblock2unicode(_)) for _ in gt_lines]
+        norm_pred_lines = [clean_string(textblock2unicode(_)) for _ in pred_lines]
+        # norm_gt_lines = get_norm_text_lines(gt_lines)
+        # norm_pred_lines = get_norm_text_lines(pred_lines)
+    else:
+        norm_gt_lines = gt_lines
+        norm_pred_lines = pred_lines
+
+    if line_type == 'latex_table':
+        gt_lines = norm_html_lines
+
+
+    filtered_lists = [(a, b, c) for a, b, c in zip(gt_lines, norm_gt_lines, gt_cat_list) if a and b]
+
+    # decompress to three lists
+    if filtered_lists:
+        gt_lines_c, norm_gt_lines_c, gt_cat_list_c = zip(*filtered_lists)
+
+        # convert to lists
+        gt_lines_c = list(gt_lines_c)
+        norm_gt_lines_c = list(norm_gt_lines_c)
+        gt_cat_list_c = list(gt_cat_list_c)
+    else:
+        gt_lines_c = []
+        norm_gt_lines_c = []
+        gt_cat_list_c = []
+
+    # pred's empty values
+    filtered_lists = [(a, b) for a, b in zip(pred_lines, norm_pred_lines) if a and b]
+
+    # decompress to two lists
+    if filtered_lists:
+        pred_lines_c, norm_pred_lines_c = zip(*filtered_lists)
+
+        # convert to lists
+        pred_lines_c = list(pred_lines_c)
+        norm_pred_lines_c = list(norm_pred_lines_c)
+    else:
+        pred_lines_c = []
+        norm_pred_lines_c = []
+
+    return gt_lines_c, norm_gt_lines_c, gt_cat_list_c, pred_lines_c, norm_pred_lines_c
+    # return gt_lines, norm_gt_lines, gt_cat_list, pred_lines, norm_pred_lines
+
+
+def match_gt2pred_simple(gt_items, pred_items, line_type, img_name):
+
+    gt_lines, norm_gt_lines, gt_cat_list, pred_lines, norm_pred_lines = get_gt_pred_lines(gt_items, pred_items, line_type)
+
+    match_list = []
+    if not norm_gt_lines: # not matched pred should be concatenated
+        # print("One of the lists is empty. Returning an empty gt result.")
+        # for pred_idx in range(len(norm_pred_lines)):
+        pred_idx_list = range(len(norm_pred_lines))
+        match_list.append({
+            'gt_idx': [""],
+            'gt': "",
+            'pred_idx': pred_idx_list,
+            'pred': ''.join(pred_lines[_] for _ in pred_idx_list),
+            'gt_position': [""],
+            'pred_position': pred_items[pred_idx_list[0]]['position'][0],  # get the first pred's position
+            'norm_gt': "",
+            'norm_pred': ''.join(norm_pred_lines[_] for _ in pred_idx_list),
+            'gt_category_type': "",
+            'pred_category_type': get_pred_category_type(pred_idx_list[0], pred_items), # get the first pred's category
+            'gt_attribute': [{}],
+            'edit': 1,
+            'img_id': img_name
+        })
+        return match_list
+    elif not norm_pred_lines: # not matched gt should be separated
+        # print("One of the lists is empty. Returning an empty pred result.")
+        for gt_idx in range(len(norm_gt_lines)):
+            match_list.append({
+                'gt_idx': [gt_idx],
+                'gt': gt_lines[gt_idx],
+                'pred_idx': [""],
+                'pred': "",
+                'gt_position': [gt_items[gt_idx].get('order') if gt_items[gt_idx].get('order') else gt_items[gt_idx].get('position', [""])[0]],
+                'pred_position': "",
+                'norm_gt': norm_gt_lines[gt_idx],
+                'norm_pred': "",
+                'gt_category_type': gt_cat_list[gt_idx],
+                'pred_category_type': "",
+                'gt_attribute': [gt_items[gt_idx].get("attribute", {})],
+                'edit': 1,
+                'img_id': img_name
+            })
+        return match_list
+
+    cost_matrix = compute_edit_distance_matrix_new(norm_gt_lines, norm_pred_lines)
+
+    row_ind, col_ind = linear_sum_assignment(cost_matrix)
+
+
+    for gt_idx in range(len(norm_gt_lines)):
+        if gt_idx in row_ind:
+            row_i = list(row_ind).index(gt_idx)
+            pred_idx = int(col_ind[row_i])
+            pred_line = pred_lines[pred_idx]
+            norm_pred_line = norm_pred_lines[pred_idx]
+            edit = cost_matrix[gt_idx][pred_idx]
+            # print('edit_dist', edit)
+            # if edit > 0.7:
+            #     print('! Not match')
+        else:
+            # print('No match pred')
+            pred_idx = ""
+            pred_line = ""
+            norm_pred_line = ""
+            edit = 1
+
+        match_list.append({
+            'gt_idx': [gt_idx],
+            'gt': gt_lines[gt_idx],
+            'norm_gt': norm_gt_lines[gt_idx],
+            'gt_category_type': gt_cat_list[gt_idx],
+            'gt_position': [gt_items[gt_idx].get('order') if gt_items[gt_idx].get('order') else gt_items[gt_idx].get('position', [""])[0]],
+            'gt_attribute': [gt_items[gt_idx].get("attribute", {})],
+            'pred_idx': [pred_idx],
+            'pred': pred_line,
+            'norm_pred': norm_pred_line,
+            'pred_category_type': get_pred_category_type(pred_idx, pred_items) if pred_idx else "",
+            'pred_position': pred_items[pred_idx]['position'][0] if pred_idx else "",
+            'edit': edit,
+            'img_id': img_name
+        })
+        # print('-'*10)
+        # [([0,1], 0),(2, 1), (1,2)] --> [0,2,1]/[0,1,2]
+
+    pred_idx_list = [pred_idx for pred_idx in range(len(norm_pred_lines)) if pred_idx not in col_ind] # get not matched preds
+    if pred_idx_list: # if there are still remaining pred_idx, concatenate all preds
+        match_list.append({
+            'gt_idx': [""],
+            'gt': "",
+            'pred_idx': pred_idx_list,
+            'pred': ''.join(pred_lines[_] for _ in pred_idx_list),
+            'gt_position': [""],
+            'pred_position': pred_items[pred_idx_list[0]]['position'][0],  # get the first pred's position
+            'norm_gt': "",
+            'norm_pred': ''.join(norm_pred_lines[_] for _ in pred_idx_list),
+            'gt_category_type': "",
+            'pred_category_type': get_pred_category_type(pred_idx_list[0], pred_items), # get the first pred's category
+            'gt_attribute': [{}],
+            'edit': 1,
+            'img_id': img_name
+        })
+    return match_list
+
+
+def match_gt2pred_no_split(gt_items, pred_items, line_type, img_name):
+    # directly concatenate gt and pred by position
+    gt_lines, norm_gt_lines, gt_cat_list, pred_lines, norm_pred_lines = get_gt_pred_lines(gt_items, pred_items, line_type)
+    gt_line_with_position = []
+    for gt_line, norm_gt_line, gt_item in zip(gt_lines, norm_gt_lines, gt_items):
+        gt_position = gt_item['order'] if gt_item.get('order') else gt_item.get('position', [""])[0]
+        if gt_position:
+            gt_line_with_position.append((gt_position, gt_line, norm_gt_line))
+    sorted_gt_lines = sorted(gt_line_with_position, key=lambda x: x[0])
+    gt = '\n\n'.join([_[1] for _ in sorted_gt_lines])
+    norm_gt = '\n\n'.join([_[2] for _ in sorted_gt_lines])
+    pred_line_with_position = [(pred_item['position'], pred_line, pred_norm_line) for pred_line, pred_norm_line, pred_item in zip(pred_lines, norm_pred_lines, pred_items)]
+    sorted_pred_lines = sorted(pred_line_with_position, key=lambda x: x[0])
+    pred = '\n\n'.join([_[1] for _ in sorted_pred_lines])
+    norm_pred = '\n\n'.join([_[2] for _ in sorted_pred_lines])
+    # edit = Levenshtein.distance(norm_gt, norm_pred)/max(len(norm_gt), len(norm_pred))
+    if norm_gt or norm_pred:
+        return [{
+                'gt_idx': [0],
+                'gt': gt,
+                'norm_gt': norm_gt,
+                'gt_category_type': "text_merge",
+                'gt_position': [""],
+                'gt_attribute': [{}],
+                'pred_idx': [0],
+                'pred': pred,
+                'norm_pred': norm_pred,
+                'pred_category_type': "text_merge",
+                'pred_position': "",
+                # 'edit': edit,
+                'img_id': img_name
+            }]
+    else:
+        return []
+
+
+import copy
+import pdb
+from collections import Counter, defaultdict
+
+import evaluate
+# from rapidfuzz.distance import Levenshtein
+import Levenshtein
+import numpy as np
+from Levenshtein import distance as Levenshtein_distance
+from scipy.optimize import linear_sum_assignment
+
+
+def match_gt2pred_quick(gt_items, pred_items, line_type, img_name):
+
+    gt_lines, norm_gt_lines, gt_cat_list, pred_lines, norm_pred_lines= get_gt_pred_lines(gt_items, pred_items, line_type)
+    all_gt_indices = set(range(len(norm_gt_lines)))
+    all_pred_indices = set(range(len(norm_pred_lines)))
+
+    if not norm_gt_lines:
+        match_list = []
+        for pred_idx in range(len(norm_pred_lines)):
+            match_list.append({
+                'gt_idx': [""],
+                'gt': "",
+                'pred_idx': [pred_idx],
+                'pred': pred_lines[pred_idx],
+                'gt_position': "",
+                'pred_position': pred_items[pred_idx]['position'][0],
+                'norm_gt': "",
+                'norm_pred': norm_pred_lines[pred_idx],
+                'gt_category_type': "",
+                'pred_category_type': get_pred_category_type(pred_idx, pred_items),
+                'gt_attribute': [{}],
+                'edit': 1,
+                'img_id': img_name
+            })
+        return match_list
+    elif not norm_pred_lines:
+        match_list = []
+        for gt_idx in range(len(norm_gt_lines)):
+            match_list.append({
+                'gt_idx': [gt_idx],
+                'gt': gt_lines[gt_idx],
+                'pred_idx': [""],
+                'pred': "",
+                'gt_position': [gt_items[gt_idx].get('order') if gt_items[gt_idx].get('order') else gt_items[gt_idx].get('position', [""])[0]],
+                'pred_position': "",
+                'norm_gt': norm_gt_lines[gt_idx],
+                'norm_pred': "",
+                'gt_category_type': gt_cat_list[gt_idx],
+                'pred_category_type': "",
+                'gt_attribute': [gt_items[gt_idx].get("attribute", {})],
+                'edit': 1,
+                'img_id': img_name
+            })
+        return match_list
+    elif len(norm_gt_lines) == 1 and len(norm_pred_lines) == 1:
+        edit_distance = Levenshtein_distance(norm_gt_lines[0], norm_pred_lines[0])
+        normalized_edit_distance = edit_distance / max(len(norm_gt_lines[0]), len(norm_pred_lines[0]))
+        return [{
+            'gt_idx': [0],
+            'gt': gt_lines[0],
+            'pred_idx': [0],
+            'pred': pred_lines[0],
+            'gt_position': [gt_items[0].get('order') if gt_items[0].get('order') else gt_items[0].get('position', [""])[0]],
+            'pred_position': pred_items[0]['position'][0],
+            'norm_gt': norm_gt_lines[0],
+            'norm_pred': norm_pred_lines[0],
+            'gt_category_type': gt_cat_list[0],
+            'pred_category_type': get_pred_category_type(0, pred_items),
+            'gt_attribute': [gt_items[0].get("attribute", {})],
+            'edit': normalized_edit_distance,
+            'img_id': img_name
+        }]
+
+    cost_matrix = compute_edit_distance_matrix_new(norm_gt_lines, norm_pred_lines)
+
+    matched_col_idx, row_ind, cost_list = cal_final_match(cost_matrix, norm_gt_lines, norm_pred_lines)
+
+    gt_lens_dict, pred_lens_dict = initialize_indices(norm_gt_lines, norm_pred_lines)
+
+    matches, unmatched_gt_indices, unmatched_pred_indices = process_matches(matched_col_idx, row_ind, cost_list, norm_gt_lines, norm_pred_lines, pred_lines)
+
+    matching_dict = fuzzy_match_unmatched_items(unmatched_gt_indices, norm_gt_lines, norm_pred_lines)
+
+    final_matches = merge_matches(matches, matching_dict)
+
+    recalculate_edit_distances(final_matches, gt_lens_dict, norm_gt_lines, norm_pred_lines)
+
+    converted_results = convert_final_matches(final_matches, norm_gt_lines, norm_pred_lines)
+
+    merged_results = merge_duplicates_add_unmatched(converted_results, norm_gt_lines, norm_pred_lines, gt_lines, pred_lines, all_gt_indices, all_pred_indices)
+
+    for entry in merged_results:
+            entry['gt_idx'] = [entry['gt_idx']] if not isinstance(entry['gt_idx'], list) else entry['gt_idx']
+            entry['pred_idx'] = [entry['pred_idx']] if not isinstance(entry['pred_idx'], list) else entry['pred_idx']
+            entry['gt_position'] = [gt_items[_].get('order') if gt_items[_].get('order') else gt_items[_].get('position', [""])[0] for _ in entry['gt_idx']] if entry['gt_idx'] != [""] else [""]
+            entry['pred_position'] = pred_items[entry['pred_idx'][0]]['position'][0] if entry['pred_idx'] != [""] else ""
+            entry['gt'] = ''.join([gt_lines[_] for _ in entry['gt_idx']]) if entry['gt_idx'] != [""] else ""
+            entry['pred'] = ''.join([pred_lines[_] for _ in entry['pred_idx']]) if entry['pred_idx'] != [""] else ""
+            entry['norm_gt'] = ''.join([norm_gt_lines[_] for _ in entry['gt_idx']]) if entry['gt_idx'] != [""] else ""
+            entry['norm_pred'] = ''.join([norm_pred_lines[_] for _ in entry['pred_idx']]) if entry['pred_idx'] != [""] else ""
+
+            if entry['gt_idx'] != [""]:
+                ignore_type = ['figure_caption', 'figure_footnote', 'table_caption', 'table_footnote', 'code_algorithm', 'code_algorithm_caption', 'header', 'footer', 'page_footnote', 'page_number', 'equation_caption']
+                gt_cagegory_clean = [gt_cat_list[_] for _ in entry['gt_idx'] if gt_cat_list[_] not in ignore_type]
+                if gt_cagegory_clean:
+                    entry['gt_category_type'] = Counter(gt_cagegory_clean).most_common(1)[0][0]
+                else:
+                    entry['gt_category_type'] = Counter([gt_cat_list[_] for _ in entry['gt_idx']]).most_common(1)[0][0]
+            else:
+                entry['gt_category_type'] = ""
+            entry['pred_category_type'] = get_pred_category_type(entry['pred_idx'][0], pred_items) if entry['pred_idx'] != [""] else ""
+            entry['gt_attribute'] = [gt_items[_].get("attribute", {}) for _ in entry['gt_idx']] if entry['gt_idx'] != [""] else [{}]
+            entry['img_id'] = img_name
+
+    return merged_results
+
+
+def merge_duplicates_add_unmatched(converted_results, norm_gt_lines, norm_pred_lines, gt_lines, pred_lines, all_gt_indices, all_pred_indices):
+    merged_results = []
+    processed_pred = set()
+    processed_gt = set()
+
+    for entry in converted_results:
+        pred_idx = tuple(entry['pred_idx']) if isinstance(entry['pred_idx'], list) else (entry['pred_idx'],)
+        if pred_idx not in processed_pred and pred_idx != ("",):
+            merged_entry = {
+                'gt_idx': [entry['gt_idx']],
+                'gt': entry['gt'],
+                'pred_idx': entry['pred_idx'],
+                'pred': entry['pred'],
+                'edit': entry['edit']
+            }
+            for other_entry in converted_results:
+                other_pred_idx = tuple(other_entry['pred_idx']) if isinstance(other_entry['pred_idx'], list) else (other_entry['pred_idx'],)
+                if other_pred_idx == pred_idx and other_entry is not entry:
+                    merged_entry['gt_idx'].append(other_entry['gt_idx'])
+                    merged_entry['gt'] += other_entry['gt']
+                    processed_gt.add(other_entry['gt_idx'])
+            merged_results.append(merged_entry)
+            processed_pred.add(pred_idx)
+            processed_gt.add(entry['gt_idx'])
+
+    for entry in converted_results:
+        if entry['gt_idx'] not in processed_gt:
+            merged_results.append(entry)
+
+    for gt_idx in range(len(norm_gt_lines)):
+        if gt_idx not in processed_gt:
+            merged_results.append({
+                'gt_idx': [gt_idx],
+                'gt': gt_lines[gt_idx],
+                'pred_idx': [""],
+                'pred': "",
+                'edit': 1
+            })
+    return merged_results
+
+
+
+
+def formula_format(formula_matches, img_name):
+    return [
+        {
+            "gt": item["gt"],
+            "pred": item["pred"],
+            "img_id": f"{img_name}_{i}"
+        }
+        for i, item in enumerate(formula_matches)
+    ]
+
+
+def merge_lists_with_sublists(main_list, sub_lists):
+    main_list_final = list(copy.deepcopy(main_list))
+    for sub_list in sub_lists:
+        pop_idx = main_list_final.index(sub_list[0])
+        for _ in sub_list:
+            main_list_final.pop(pop_idx)
+        main_list_final.insert(pop_idx, sub_list)
+    return main_list_final
+
+
+def sub_pred_fuzzy_matching(gt, pred):
+
+    min_d = float('inf')
+    # pos = -1
+
+    gt_len = len(gt)
+    pred_len = len(pred)
+
+    if gt_len >= pred_len and pred_len > 0:
+        for i in range(gt_len - pred_len + 1):
+            sub = gt[i:i + pred_len]
+            dist = Levenshtein_distance(sub, pred)/pred_len
+            if dist < min_d:
+                min_d = dist
+                pos = i
+
+        return min_d
+    else:
+        return False
+
+def sub_gt_fuzzy_matching(pred, gt):
+
+    min_d = float('inf')
+    pos = ""
+    matched_sub = ""
+    gt_len = len(gt)
+    pred_len = len(pred)
+
+    if pred_len >= gt_len and gt_len > 0:
+        for i in range(pred_len - gt_len + 1):
+            sub = pred[i:i + gt_len]
+            dist = Levenshtein.distance(sub, gt)  /gt_len
+            if dist < min_d:
+                min_d = dist
+                pos = i
+                matched_sub = sub
+        return min_d, pos, gt_len, matched_sub
+    else:
+        return 1, "", gt_len, ""
+
+
+def get_final_subset(subset_certain, subset_certain_cost):
+    if not subset_certain or not subset_certain_cost:
+        return []
+
+    subset_turple = sorted([(a, b) for a, b in zip(subset_certain, subset_certain_cost)], key=lambda x: x[0][0])
+
+    group_list = defaultdict(list)
+    group_idx = 0
+    group_list[group_idx].append(subset_turple[0])
+
+    for item in subset_turple[1:]:
+        overlap_flag = False
+        for subset in group_list[group_idx]:
+            for idx in item[0]:
+                if idx in subset[0]:
+                    overlap_flag = True
+                    break
+            if overlap_flag:
+                break
+        if overlap_flag:
+            group_list[group_idx].append(item)
+        else:
+            group_idx += 1
+            group_list[group_idx].append(item)
+
+    final_subset = []
+    for _, group in group_list.items():
+        if len(group) == 1:
+            final_subset.append(group[0][0])
+        else:
+            path_dict = defaultdict(list)
+            path_idx = 0
+            path_dict[path_idx].append(group[0])
+
+            for subset in group[1:]:
+                new_path = True
+                for path_idx_s, path_items in path_dict.items():
+                    is_dup = False
+                    is_same = False
+                    for path_item in path_items:
+                        if path_item[0] == subset[0]:
+                            is_dup = True
+                            is_same = True
+                            if path_item[1] > subset[1]:
+                                path_dict[path_idx_s].pop(path_dict[path_idx_s].index(path_item))
+                                path_dict[path_idx_s].append(subset)
+                        else:
+                            for num_1 in path_item[0]:
+                                for num_2 in subset[0]:
+                                    if num_1 == num_2:
+                                        is_dup = True
+                    if not is_dup:
+                        path_dict[path_idx_s].append(subset)
+                        new_path = False
+                    if is_same:
+                        new_path = False
+                if new_path:
+                    path_idx = len(path_dict.keys())
+                    path_dict[path_idx].append(subset)
+
+            saved_cost = float('inf')
+            saved_subset = []
+            for path_idx, path in path_dict.items():
+                avg_cost = sum([i[1] for i in path]) / len(path)
+                if avg_cost < saved_cost:
+                    saved_subset = [i[0] for i in path]
+                    saved_cost = avg_cost
+
+            final_subset.extend(saved_subset)
+
+    return final_subset
+
+def judge_pred_merge(gt_list, pred_list, threshold=0.6):
+    if len(pred_list) == 1:
+        return False, False
+
+    cur_pred = ' '.join(pred_list[:-1])
+    merged_pred = ' '.join(pred_list)
+
+    cur_dist = Levenshtein.distance(gt_list[0], cur_pred) / max(len(gt_list[0]), len(cur_pred))
+    merged_dist = Levenshtein.distance(gt_list[0], merged_pred) / max(len(gt_list[0]), len(merged_pred))
+
+    if merged_dist > cur_dist:
+        return False, False
+
+    cur_fuzzy_dists = [sub_pred_fuzzy_matching(gt_list[0], cur_pred) for cur_pred in pred_list[:-1]]
+    if any(dist is False or dist > threshold for dist in cur_fuzzy_dists):
+        return False, False
+
+    add_fuzzy_dist = sub_pred_fuzzy_matching(gt_list[0], pred_list[-1])
+    if add_fuzzy_dist is False:
+        return False, False
+
+    merged_pred_flag = add_fuzzy_dist < threshold
+    continue_flag = len(merged_pred) <= len(gt_list[0])
+
+    return merged_pred_flag, continue_flag
+
+def deal_with_truncated(cost_matrix, norm_gt_lines, norm_pred_lines):
+    matched_first = np.argwhere(cost_matrix < 0.25)
+    masked_gt_idx = [i[0] for i in matched_first]
+    unmasked_gt_idx = [i for i in range(cost_matrix.shape[0]) if i not in masked_gt_idx]
+    masked_pred_idx = [i[1] for i in matched_first]
+    unmasked_pred_idx = [i for i in range(cost_matrix.shape[1]) if i not in masked_pred_idx]
+
+    merges_gt_dict = {}
+    merges_pred_dict = {}
+    merged_gt_subsets = []
+
+    for gt_idx in unmasked_gt_idx:
+        check_merge_subset = []
+        merged_dist = []
+
+        for pred_idx in unmasked_pred_idx:
+            step = 1
+            merged_pred = [norm_pred_lines[pred_idx]]
+
+            while True:
+                if pred_idx + step in masked_pred_idx or pred_idx + step >= len(norm_pred_lines):
+                    break
+                else:
+                    merged_pred.append(norm_pred_lines[pred_idx + step])
+                    merged_pred_flag, continue_flag = judge_pred_merge([norm_gt_lines[gt_idx]], merged_pred)
+                    if not merged_pred_flag:
+                        break
+                    else:
+                        step += 1
+                    if not continue_flag:
+                        break
+
+            check_merge_subset.append(list(range(pred_idx, pred_idx + step)))
+            matched_line = ' '.join([norm_pred_lines[i] for i in range(pred_idx, pred_idx + step)])
+            dist = Levenshtein_distance(norm_gt_lines[gt_idx], matched_line) / max(len(matched_line), len(norm_gt_lines[gt_idx]))
+            merged_dist.append(dist)
+
+        if not merged_dist:
+            subset_certain = []
+            min_cost_idx = ""
+            min_cost = float('inf')
+        else:
+            min_cost = min(merged_dist)
+            min_cost_idx = merged_dist.index(min_cost)
+            subset_certain = check_merge_subset[min_cost_idx]
+
+        merges_gt_dict[gt_idx] = {
+            'merge_subset': check_merge_subset,
+            'merged_cost': merged_dist,
+            'min_cost_idx': min_cost_idx,
+            'subset_certain': subset_certain,
+            'min_cost': min_cost
+        }
+
+    subset_certain = [merges_gt_dict[gt_idx]['subset_certain'] for gt_idx in unmasked_gt_idx if merges_gt_dict[gt_idx]['subset_certain']]
+    subset_certain_cost = [merges_gt_dict[gt_idx]['min_cost'] for gt_idx in unmasked_gt_idx if merges_gt_dict[gt_idx]['subset_certain']]
+
+    subset_certain_final = get_final_subset(subset_certain, subset_certain_cost)
+
+    if not subset_certain_final:
+        return cost_matrix, norm_pred_lines, range(len(norm_pred_lines))
+
+    final_pred_idx_list = merge_lists_with_sublists(range(len(norm_pred_lines)), subset_certain_final)
+    final_norm_pred_lines = [' '.join(norm_pred_lines[idx_list[0]:idx_list[-1]+1]) if isinstance(idx_list, list) else norm_pred_lines[idx_list] for idx_list in final_pred_idx_list]
+
+    new_cost_matrix = compute_edit_distance_matrix_new(norm_gt_lines, final_norm_pred_lines)
+
+    return new_cost_matrix, final_norm_pred_lines, final_pred_idx_list
+
+def cal_move_dist(gt, pred):
+    assert len(gt) == len(pred), 'Not right length'
+    step = 0
+    for i, gt_c in enumerate(gt):
+        if gt_c != pred[i]:
+            step += abs(i - pred.index(gt_c))
+            pred[i], pred[pred.index(gt_c)] = pred[pred.index(gt_c)], pred[i]
+    return step / len(gt)
+
+def cal_final_match(cost_matrix, norm_gt_lines, norm_pred_lines):
+    min_indice = cost_matrix.argmax(axis=1)
+
+    new_cost_matrix, final_norm_pred_lines, final_pred_idx_list = deal_with_truncated(cost_matrix, norm_gt_lines, norm_pred_lines)
+
+    row_ind, col_ind = linear_sum_assignment(new_cost_matrix)
+
+    cost_list = [new_cost_matrix[r][c] for r, c in zip(row_ind, col_ind)]
+    matched_col_idx = [final_pred_idx_list[i] for i in col_ind]
+
+    return matched_col_idx, row_ind, cost_list
+
+def initialize_indices(norm_gt_lines, norm_pred_lines):
+    gt_lens_dict = {idx: len(gt_line) for idx, gt_line in enumerate(norm_gt_lines)}
+    pred_lens_dict = {idx: len(pred_line) for idx, pred_line in enumerate(norm_pred_lines)}
+    return gt_lens_dict, pred_lens_dict
+
+def process_matches(matched_col_idx, row_ind, cost_list, norm_gt_lines, norm_pred_lines, pred_lines):
+    matches = {}
+    unmatched_gt_indices = []
+    unmatched_pred_indices = []
+
+    for i in range(len(norm_gt_lines)):
+        if i in row_ind:
+            idx = list(row_ind).index(i)
+            pred_idx = matched_col_idx[idx]
+
+            if pred_idx is None or (isinstance(pred_idx, list) and None in pred_idx):
+                unmatched_pred_indices.append(pred_idx)
+                continue
+
+            if isinstance(pred_idx, list):
+                pred_line = ' | '.join(norm_pred_lines[pred_idx[0]:pred_idx[-1]+1])
+                ori_pred_line = ' | '.join(pred_lines[pred_idx[0]:pred_idx[-1]+1])
+                matched_pred_indices_range = list(range(pred_idx[0], pred_idx[-1]+1))
+            else:
+                pred_line = norm_pred_lines[pred_idx]
+                ori_pred_line = pred_lines[pred_idx]
+                matched_pred_indices_range = [pred_idx]
+
+            edit = cost_list[idx]
+
+            if edit > 0.7:
+                unmatched_pred_indices.extend(matched_pred_indices_range)
+                unmatched_gt_indices.append(i)
+            else:
+                matches[i] = {
+                    'pred_indices': matched_pred_indices_range,
+                    'edit_distance': edit,
+                }
+                for matched_pred_idx in matched_pred_indices_range:
+                    if matched_pred_idx in unmatched_pred_indices:
+                        unmatched_pred_indices.remove(matched_pred_idx)
+        else:
+            unmatched_gt_indices.append(i)
+
+    return matches, unmatched_gt_indices, unmatched_pred_indices
+
+def fuzzy_match_unmatched_items(unmatched_gt_indices, norm_gt_lines, norm_pred_lines):
+    matching_dict = {}
+
+    for pred_idx, pred_content in enumerate(norm_pred_lines):
+        if isinstance(pred_idx, list):
+            continue
+
+        matching_indices = []
+
+        for unmatched_gt_idx in unmatched_gt_indices:
+            gt_content = norm_gt_lines[unmatched_gt_idx]
+            cur_fuzzy_dist_unmatch, cur_pos, gt_lens, matched_field = sub_gt_fuzzy_matching(pred_content, gt_content)
+            if cur_fuzzy_dist_unmatch < 0.4:
+                matching_indices.append(unmatched_gt_idx)
+
+        if matching_indices:
+            matching_dict[pred_idx] = matching_indices
+
+    return matching_dict
+
+def merge_matches(matches, matching_dict):
+    final_matches = {}
+    processed_gt_indices = set()
+
+    for gt_idx, match_info in matches.items():
+        pred_indices = match_info['pred_indices']
+        edit_distance = match_info['edit_distance']
+
+        pred_key = tuple(sorted(pred_indices))
+
+        if pred_key in final_matches:
+            if gt_idx not in processed_gt_indices:
+                final_matches[pred_key]['gt_indices'].append(gt_idx)
+                processed_gt_indices.add(gt_idx)
+        else:
+            final_matches[pred_key] = {
+                'gt_indices': [gt_idx],
+                'edit_distance': edit_distance
+            }
+            processed_gt_indices.add(gt_idx)
+
+    for pred_idx, gt_indices in matching_dict.items():
+        pred_key = (pred_idx,) if not isinstance(pred_idx, (list, tuple)) else tuple(sorted(pred_idx))
+
+        if pred_key in final_matches:
+            for gt_idx in gt_indices:
+                if gt_idx not in processed_gt_indices:
+                    final_matches[pred_key]['gt_indices'].append(gt_idx)
+                    processed_gt_indices.add(gt_idx)
+        else:
+            final_matches[pred_key] = {
+                'gt_indices': [gt_idx for gt_idx in gt_indices if gt_idx not in processed_gt_indices],
+                'edit_distance': None
+            }
+            processed_gt_indices.update(final_matches[pred_key]['gt_indices'])
+
+    return final_matches
+
+
+
+def recalculate_edit_distances(final_matches, gt_lens_dict, norm_gt_lines, norm_pred_lines):
+    for pred_key, info in final_matches.items():
+        gt_indices = sorted(set(info['gt_indices']))
+
+        if not gt_indices:
+            info['edit_distance'] = 1
+            continue
+
+        if len(gt_indices) > 1:
+            merged_gt_content = ''.join(norm_gt_lines[gt_idx] for gt_idx in gt_indices)
+            pred_content = norm_pred_lines[pred_key[0]] if isinstance(pred_key[0], int) else ''
+
+            try:
+                edit_distance = Levenshtein_distance(merged_gt_content, pred_content)
+                normalized_edit_distance = edit_distance / max(len(merged_gt_content), len(pred_content))
+            except ZeroDivisionError:
+                normalized_edit_distance = 1
+
+            info['edit_distance'] = normalized_edit_distance
+        else:
+            gt_idx = gt_indices[0]
+            pred_content = ' '.join(norm_pred_lines[pred_idx] for pred_idx in pred_key if isinstance(pred_idx, int))
+
+            try:
+                edit_distance = Levenshtein_distance(norm_gt_lines[gt_idx], pred_content)
+                normalized_edit_distance = edit_distance / max(len(norm_gt_lines[gt_idx]), len(pred_content))
+            except ZeroDivisionError:
+                normalized_edit_distance = 1
+
+            info['edit_distance'] = normalized_edit_distance
+            info['pred_content'] = pred_content
+
+
+def convert_final_matches(final_matches, norm_gt_lines, norm_pred_lines):
+    converted_results = []
+
+    all_gt_indices = set(range(len(norm_gt_lines)))
+    all_pred_indices = set(range(len(norm_pred_lines)))
+
+    for pred_key, info in final_matches.items():
+        pred_content = ' '.join(norm_pred_lines[pred_idx] for pred_idx in pred_key if isinstance(pred_idx, int))
+
+        for gt_idx in sorted(set(info['gt_indices'])):
+            result_entry = {
+                'gt_idx': int(gt_idx),
+                'gt': norm_gt_lines[gt_idx],
+                'pred_idx': list(pred_key),
+                'pred': pred_content,
+                'edit': info['edit_distance']
+            }
+            converted_results.append(result_entry)
+
+    matched_gt_indices = set().union(*[set(info['gt_indices']) for info in final_matches.values()])
+    unmatched_gt_indices = all_gt_indices - matched_gt_indices
+    matched_pred_indices = set(idx for pred_key in final_matches.keys() for idx in pred_key if isinstance(idx, int))
+    unmatched_pred_indices = all_pred_indices - matched_pred_indices
+
+    if unmatched_pred_indices:
+        if unmatched_gt_indices:
+            distance_matrix = [
+                [Levenshtein_distance(norm_gt_lines[gt_idx], norm_pred_lines[pred_idx]) for pred_idx in unmatched_pred_indices]
+                for gt_idx in unmatched_gt_indices
+            ]
+
+            row_ind, col_ind = linear_sum_assignment(distance_matrix)
+
+            for i, j in zip(row_ind, col_ind):
+                gt_idx = list(unmatched_gt_indices)[i]
+                pred_idx = list(unmatched_pred_indices)[j]
+                result_entry = {
+                    'gt_idx': int(gt_idx),
+                    'gt': norm_gt_lines[gt_idx],
+                    'pred_idx': [pred_idx],
+                    'pred': norm_pred_lines[pred_idx],
+                    'edit': 1
+                }
+                converted_results.append(result_entry)
+
+            matched_gt_indices.update(list(unmatched_gt_indices)[i] for i in row_ind)
+        else:
+            result_entry = {
+                'gt_idx': "",
+                'gt': '',
+                'pred_idx': list(unmatched_pred_indices),
+                'pred': ' '.join(norm_pred_lines[pred_idx] for pred_idx in unmatched_pred_indices),
+                'edit': 1
+            }
+            converted_results.append(result_entry)
+    else:
+        for gt_idx in unmatched_gt_indices:
+            result_entry = {
+                'gt_idx': int(gt_idx),
+                'gt': norm_gt_lines[gt_idx],
+                'pred_idx': "",
+                'pred': '',
+                'edit': 1
+            }
+            converted_results.append(result_entry)
+
+    return converted_results
+
+import json
+
+
+def read_md_file(filepath):
+    with open(filepath, 'r', encoding='utf-8') as file:
+        content = file.read()
+
+    return content
+
+def save_paired_result(preds, gts, save_path):
+    save_result = []
+    formula_id = 0
+    for gt, pred in zip(gts, preds):
+        save_result.append({
+            "gt": gt,
+            "pred": pred,
+            "img_id": formula_id
+        })
+        formula_id += 1
+    with open(save_path, 'w', encoding='utf-8') as f:
+        json.dump(save_result, f, indent=4, ensure_ascii=False)
+
+
+import os
+import re
+
+import matplotlib.font_manager as fm
+import matplotlib.pyplot as plt
+import numpy as np
+
+font = fm.FontProperties(fname=r'font/SimHei.ttf')
+
+
+def print_aligned_dict(data):
+    # Find the maximum length of all keys
+    max_key_length = max(len(key) for key in data['testcase1'])
+
+    # Print header
+    print(f"{' ' * (max_key_length + 4)}", end="")
+    for key in data:
+        print(f"{key:>{max_key_length}}", end="")
+    print()
+
+    # Print dictionary content
+    for subkey in data['testcase1']:
+        print(f"{subkey:<{max_key_length + 4}}", end="")
+        for key in data:
+            print(f"{data[key][subkey]:>{max_key_length}}", end="")
+        print()
+def create_dict_from_folders(directory):
+    body = {}
+    for folder_name in os.listdir(directory):
+        folder_path = os.path.join(directory, folder_name)
+        if os.path.isdir(folder_path):
+            body[folder_name] = {}
+    return body
+
+
+def create_radar_chart(df, title, filename):
+    labels = df.columns
+
+    # Calculate angles
+    angles = np.linspace(0, 2 * np.pi, len(labels), endpoint=False).tolist()
+    angles += angles[:1]
+
+    # Initialize radar chart
+    fig, ax = plt.subplots(figsize=(10, 6), subplot_kw=dict(polar=True), dpi=200)
+    # ax.spines['polar'].set_visible(False)
+
+    # Draw radar chart for each dataset
+    for index, row in df.iterrows():
+        values = row.tolist()
+        values += values[:1]
+        ax.fill(angles, values, alpha=0.1)
+        ax.plot(angles, values, label=index)
+
+        # Add percentage labels next to each data point
+        for angle, value in zip(angles, values):
+            ax.text(angle, value, '{:.1%}'.format(value), ha='center', va='center', fontsize=7, alpha=0.7)
+
+    # Set labels
+    ax.set_yticklabels([])
+    ax.set_xticks(angles[:-1])
+    ax.set_xticklabels(labels, fontproperties=font)
+    ax.spines['polar'].set_visible(False)  # Hide the outermost circle
+    ax.grid(False)
+    for j in np.arange(0, 1.2, 0.2):
+        ax.plot(angles, len(values) * [j], '-.', lw=0.5, color='black', alpha=0.5)
+    for j in range(len(values)):
+        ax.plot([angles[j], angles[j]], [0, 1], '-.', lw=0.5, color='black', alpha=0.5)
+
+    # Add title and legend
+    plt.legend(loc='upper right', bbox_to_anchor=(0.1, 0.1))
+
+    ax.tick_params(pad=30)
+    ax.set_theta_zero_location('N')
+    # Save chart to file
+    plt.savefig(filename)
+
+# The function is from https://github.com/intsig-textin/markdown_tester
+def markdown_to_html(markdown_table):
+    rows = [row.strip() for row in markdown_table.strip().split('\n')]
+    num_columns = len(rows[0].split('|')) - 2
+
+    html_table = '<table>\n  <thead>\n    <tr>\n'
+
+    header_cells = [cell.strip() for cell in rows[0].split('|')[1:-1]]
+    for cell in header_cells:
+        html_table += f'      <th>{cell}</th>\n'
+    html_table += '    </tr>\n  </thead>\n  <tbody>\n'
+
+    for row in rows[2:]:
+        cells = [cell.strip() for cell in row.split('|')[1:-1]]
+        html_table += '    <tr>\n'
+        for cell in cells:
+            html_table += f'      <td>{cell}</td>\n'
+        html_table += '    </tr>\n'
+
+    html_table += '  </tbody>\n</table>\n'
+    return html_table
+def convert_markdown_to_html(self, markdown_content, md_type):
+    # Define a regex pattern to find Markdown tables with newlines
+    markdown_content = markdown_content.replace('\r', '')
+    pattern = re.compile(r'\|\s*.*?\s*\|\n', re.DOTALL)
+
+    # Find all matches in the Markdown content
+    matches = pattern.findall(markdown_content)
+    for match in matches:
+        html_table = markdown_to_html(match)
+        markdown_content = markdown_content.replace(match, html_table, 1)  # Only replace the first occurrence
+    res_html = convert_table(replace_table_with_placeholder(markdown_content))
+
+    return res_html
+def convert_table_str(s):
+    s = re.sub(r'<table.*?>','<table>',s)
+    s = re.sub(r'<th','<td',s)
+    s = re.sub(r'</th>','</td>',s)
+    # s = re.sub(r'<td rowspan="(.)">',lambda x:f'<td colspan="1" rowspan="{x.group(1)}">',s)
+    # s = re.sub(r'<td colspan="(.)">',lambda x:f'<td colspan="{x.group(1)}" rowspan="1">',s)
+    res = ''
+    res += '\n\n'
+    temp_item = ''
+    for c in s:
+        temp_item += c
+        if c == '>' and not re.search(r'<td.*?>\$',temp_item):
+            res += temp_item+'\n'
+            temp_item = ''
+    return res+'\n'
+def merge_table(md):
+    table_temp = ''
+    for line in md:
+        table_temp += line
+    return convert_table_str(table_temp)
+def find_md_table_mode(line):
+    if re.search(r'-*?:',line) or re.search(r'---',line) or re.search(r':-*?',line):
+        return True
+    return False
+def delete_table_and_body(input_list):
+    res = []
+    for line in input_list:
+        if not re.search(r'</?t(able|head|body)>',line):
+            res.append(line)
+    return res
+def merge_tables(input_str):
+    # Delete HTML comments
+    input_str = re.sub(r'<!--[\s\S]*?-->', '', input_str)
+
+    # Use regex to find each <table> block
+    table_blocks = re.findall(r'<table>[\s\S]*?</table>', input_str)
+
+    # Process each <table> block, replace <th> with <td>
+    output_lines = []
+    for block in table_blocks:
+        block_lines = block.split('\n')
+        for i, line in enumerate(block_lines):
+            if '<th>' in line:
+                block_lines[i] = line.replace('<th>', '<td>').replace('</th>', '</td>')
+        final_tr = delete_table_and_body(block_lines)
+        if len(final_tr) > 2:
+            output_lines.extend(final_tr)  # Ignore <table> and </table> tags, keep only table content
+
+    # Rejoin the processed strings
+    merged_output = '<table>\n{}\n</table>'.format('\n'.join(output_lines))
+
+    return "\n\n" + merged_output + "\n\n"
+
+def replace_table_with_placeholder(input_string):
+    lines = input_string.split('\n')
+    output_lines = []
+
+    in_table_block = False
+    temp_block = ""
+    last_line = ""
+
+    org_table_list = []
+    in_org_table = False
+
+    for idx, line in enumerate(lines):
+        # if not in_org_table:
+        # if "<table>" not in last_line and in_table_block == False and temp_block != "":
+        #     output_lines.append(merge_tables(temp_block))
+        #     temp_block = ""
+        if "<table>" in line:
+            # if "<table><tr" in line:
+            #     org_table_list.append(line)
+            #     in_org_table = True
+            #     output_lines.append(last_line)
+            #     continue
+            # else:
+            in_table_block = True
+            temp_block += last_line
+        elif in_table_block:
+            if not find_md_table_mode(last_line) and "</thead>" not in last_line:
+                temp_block += "\n" + last_line
+            if "</table>" in last_line:
+                if "<table>" not in line:
+                    in_table_block = False
+                    output_lines.append(merge_tables(temp_block))
+                    temp_block = ""
+        else:
+            output_lines.append(last_line)
+
+        last_line = line
+        # else:
+        #     org_table_list.append(line)
+        #     if "</table" in line:
+        #         in_org_table = False
+        #         last_line = merge_table(org_table_list)
+        #         org_table_list = []
+
+    if last_line:
+        if in_table_block or "</table>" in last_line:
+            temp_block += "\n" + last_line
+            output_lines.append(merge_tables(temp_block))
+        else:
+            output_lines.append(last_line)
+    # if "</table>" in last_line:
+    #     output_lines.append(merge_tables(temp_block))
+
+    return '\n'.join(output_lines)
+
+def convert_table(input_str):
+    # Replace <table>
+    output_str = input_str.replace("<table>", "<table border=\"1\" >")
+
+    # Replace <td>
+    output_str = output_str.replace("<td>", "<td colspan=\"1\" rowspan=\"1\">")
+
+    return output_str
+
+def convert_markdown_to_html(markdown_content):
+    # Define a regex pattern to find Markdown tables with newlines
+    markdown_content = markdown_content.replace('\r', '')+'\n'
+    pattern = re.compile(r'\|\s*.*?\s*\|\n', re.DOTALL)
+
+    # Find all matches in the Markdown content
+    matches = pattern.findall(markdown_content)
+
+    for match in matches:
+        html_table = markdown_to_html(match)
+        markdown_content = markdown_content.replace(match, html_table, 1)  # Only replace the first occurrence
+
+    res_html = convert_table(replace_table_with_placeholder(markdown_content))
+
+    return res_html