+ The page you are looking for cannot be found. +
++ If you just switched documentation versions, it is likely that the page you were on is moved. You can look for it in + the content table left, or go to the homepage. +
+ + +{% endblock %} diff --git a/docs/en/_templates/autosummary/class.rst b/docs/en/_templates/autosummary/class.rst new file mode 100644 index 0000000000000000000000000000000000000000..4c3a7a9abf5c5b14ac3ef3b00a2f070480295358 --- /dev/null +++ b/docs/en/_templates/autosummary/class.rst @@ -0,0 +1,13 @@ +.. role:: hidden + :class: hidden-section +.. currentmodule:: {{ module }} + + +{{ name | underline}} + +.. autoclass:: {{ name }} + :members: + +.. + autogenerated from _templates/autosummary/class.rst + note it does not have :inherited-members: diff --git a/docs/en/_templates/callable.rst b/docs/en/_templates/callable.rst new file mode 100644 index 0000000000000000000000000000000000000000..3a7b9d2b96c76dfa3eb1d8bef56f58f219fe7760 --- /dev/null +++ b/docs/en/_templates/callable.rst @@ -0,0 +1,14 @@ +.. role:: hidden + :class: hidden-section +.. currentmodule:: {{ module }} + + +{{ name | underline}} + +.. autoclass:: {{ name }} + :members: + :special-members: __call__ + +.. + autogenerated from _templates/callable.rst + note it does not have :inherited-members: diff --git a/docs/en/advanced_guides/accelerator_intro.md b/docs/en/advanced_guides/accelerator_intro.md new file mode 100644 index 0000000000000000000000000000000000000000..cca84083aa791f712de953d5daf387c6389681ba --- /dev/null +++ b/docs/en/advanced_guides/accelerator_intro.md @@ -0,0 +1,142 @@ +# Accelerate Evaluation Inference with vLLM or LMDeploy + +## Background + +During the OpenCompass evaluation process, the Huggingface transformers library is used for inference by default. While this is a very general solution, there are scenarios where more efficient inference methods are needed to speed up the process, such as leveraging VLLM or LMDeploy. + +- [LMDeploy](https://github.com/InternLM/lmdeploy) is a toolkit designed for compressing, deploying, and serving large language models (LLMs), developed by the [MMRazor](https://github.com/open-mmlab/mmrazor) and [MMDeploy](https://github.com/open-mmlab/mmdeploy) teams. +- [vLLM](https://github.com/vllm-project/vllm) is a fast and user-friendly library for LLM inference and serving, featuring advanced serving throughput, efficient PagedAttention memory management, continuous batching of requests, fast model execution via CUDA/HIP graphs, quantization techniques (e.g., GPTQ, AWQ, SqueezeLLM, FP8 KV Cache), and optimized CUDA kernels. + +## Preparation for Acceleration + +First, check whether the model you want to evaluate supports inference acceleration using vLLM or LMDeploy. Additionally, ensure you have installed vLLM or LMDeploy as per their official documentation. Below are the installation methods for reference: + +### LMDeploy Installation Method + +Install LMDeploy using pip (Python 3.8+) or from [source](https://github.com/InternLM/lmdeploy/blob/main/docs/en/build.md): + +```bash +pip install lmdeploy +``` + +### VLLM Installation Method + +Install vLLM using pip or from [source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source): + +```bash +pip install vllm +``` + +## Accelerated Evaluation Using VLLM or LMDeploy + +### Method 1: Using Command Line Parameters to Change the Inference Backend + +OpenCompass offers one-click evaluation acceleration. During evaluation, it can automatically convert Huggingface transformer models to VLLM or LMDeploy models for use. Below is an example code for evaluating the GSM8k dataset using the default Huggingface version of the llama3-8b-instruct model: + +```python +# eval_gsm8k.py +from mmengine.config import read_base + +with read_base(): + # Select a dataset list + from .datasets.gsm8k.gsm8k_0shot_gen_a58960 import gsm8k_datasets as datasets + # Select an interested model + from ..models.hf_llama.hf_llama3_8b_instruct import models +``` + +Here, `hf_llama3_8b_instruct` specifies the original Huggingface model configuration, as shown below: + +```python +from opencompass.models import HuggingFacewithChatTemplate + +models = [ + dict( + type=HuggingFacewithChatTemplate, + abbr='llama-3-8b-instruct-hf', + path='meta-llama/Meta-Llama-3-8B-Instruct', + max_out_len=1024, + batch_size=8, + run_cfg=dict(num_gpus=1), + stop_words=['<|end_of_text|>', '<|eot_id|>'], + ) +] +``` + +To evaluate the GSM8k dataset using the default Huggingface version of the llama3-8b-instruct model, use: + +```bash +python run.py config/eval_gsm8k.py +``` + +To accelerate the evaluation using vLLM or LMDeploy, you can use the following script: + +```bash +python run.py config/eval_gsm8k.py -a vllm +``` + +or + +```bash +python run.py config/eval_gsm8k.py -a lmdeploy +``` + +### Method 2: Accelerating Evaluation via Deployed Inference Acceleration Service API + +OpenCompass also supports accelerating evaluation by deploying vLLM or LMDeploy inference acceleration service APIs. Follow these steps: + +1. Install the openai package: + +```bash +pip install openai +``` + +2. Deploy the inference acceleration service API for vLLM or LMDeploy. Below is an example for LMDeploy: + +```bash +lmdeploy serve api_server meta-llama/Meta-Llama-3-8B-Instruct --model-name Meta-Llama-3-8B-Instruct --server-port 23333 +``` + +Parameters for starting the api_server can be checked using `lmdeploy serve api_server -h`, such as --tp for tensor parallelism, --session-len for the maximum context window length, --cache-max-entry-count for adjusting the k/v cache memory usage ratio, etc. + +3. Once the service is successfully deployed, modify the evaluation script by changing the model configuration path to the service address, as shown below: + +```python +from opencompass.models import OpenAISDK + +api_meta_template = dict( + round=[ + dict(role='HUMAN', api_role='HUMAN'), + dict(role='BOT', api_role='BOT', generate=True), + ], + reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')], +) + +models = [ + dict( + abbr='Meta-Llama-3-8B-Instruct-LMDeploy-API', + type=OpenAISDK, + key='EMPTY', # API key + openai_api_base='http://0.0.0.0:23333/v1', # Service address + path='Meta-Llama-3-8B-Instruct', # Model name for service request + tokenizer_path='meta-llama/Meta-Llama-3.1-8B-Instruct', # The tokenizer name or path, if set to `None`, uses the default `gpt-4` tokenizer + rpm_verbose=True, # Whether to print request rate + meta_template=api_meta_template, # Service request template + query_per_second=1, # Service request rate + max_out_len=1024, # Maximum output length + max_seq_len=4096, # Maximum input length + temperature=0.01, # Generation temperature + batch_size=8, # Batch size + retry=3, # Number of retries + ) +] +``` + +## Acceleration Effect and Performance Comparison + +Below is a comparison table of the acceleration effect and performance when using VLLM or LMDeploy on a single A800 GPU for evaluating the Llama-3-8B-Instruct model on the GSM8k dataset: + +| Inference Backend | Accuracy | Inference Time (minutes:seconds) | Speedup (relative to Huggingface) | +| ----------------- | -------- | -------------------------------- | --------------------------------- | +| Huggingface | 74.22 | 24:26 | 1.0 | +| LMDeploy | 73.69 | 11:15 | 2.2 | +| VLLM | 72.63 | 07:52 | 3.1 | diff --git a/docs/en/advanced_guides/circular_eval.md b/docs/en/advanced_guides/circular_eval.md new file mode 100644 index 0000000000000000000000000000000000000000..5251dde9cbe1dd6905e008232e56ed325b35d076 --- /dev/null +++ b/docs/en/advanced_guides/circular_eval.md @@ -0,0 +1,113 @@ +# CircularEval + +## Background + +For multiple-choice questions, when a Language Model (LLM) provides the correct option, it does not necessarily imply a true understanding and reasoning of the question. It could be a guess. To differentiate these scenarios and reduce LLM bias towards options, CircularEval (CircularEval) can be utilized. A multiple-choice question is augmented by shuffling its options, and if the LLM correctly answers all variations of the augmented question, it is considered correct under CircularEval. + +## Adding Your Own CircularEval Dataset + +Generally, to evaluate a dataset using CircularEval, both its loading and evaluation methods need to be rewritten. Modifications are required in both the OpenCompass main library and configuration files. We will use C-Eval as an example for explanation. + +OpenCompass main library: + +```python +from opencompass.datasets.ceval import CEvalDataset +from opencompass.datasets.circular import CircularDatasetMeta + +class CircularCEvalDataset(CEvalDataset, metaclass=CircularDatasetMeta): + # The overloaded dataset class + dataset_class = CEvalDataset + + # Splits of the DatasetDict that need CircularEval. For CEvalDataset, which loads [dev, val, test], we only need 'val' and 'test' for CircularEval, not 'dev' + default_circular_splits = ['val', 'test'] + + # List of keys to be shuffled + default_option_keys = ['A', 'B', 'C', 'D'] + + # If the content of 'answer_key' is one of ['A', 'B', 'C', 'D'], representing the correct answer. This field indicates how to update the correct answer after shuffling options. Choose either this or default_answer_key_switch_method + default_answer_key = 'answer' + + # If 'answer_key' content is not one of ['A', 'B', 'C', 'D'], a function can be used to specify the correct answer after shuffling options. Choose either this or default_answer_key + # def default_answer_key_switch_method(item, circular_pattern): + # # 'item' is the original data item + # # 'circular_pattern' is a tuple indicating the order after shuffling options, e.g., ('D', 'A', 'B', 'C') means the original option A is now D, and so on + # item['answer'] = circular_pattern['ABCD'.index(item['answer'])] + # return item +``` + +`CircularCEvalDataset` accepts the `circular_pattern` parameter with two values: + +- `circular`: Indicates a single cycle. It is the default value. ABCD is expanded to ABCD, BCDA, CDAB, DABC, a total of 4 variations. +- `all_possible`: Indicates all permutations. ABCD is expanded to ABCD, ABDC, ACBD, ACDB, ADBC, ADCB, BACD, ..., a total of 24 variations. + +Additionally, we provide a `CircularEvaluator` to replace `AccEvaluator`. This Evaluator also accepts `circular_pattern`, and it should be consistent with the above. It produces the following metrics: + +- `acc_{origin|circular|all_possible}`: Treating each question with shuffled options as separate, calculating accuracy. +- `perf_{origin|circular|all_possible}`: Following Circular logic, a question is considered correct only if all its variations with shuffled options are answered correctly, calculating accuracy. +- `more_{num}_{origin|circular|all_possible}`: According to Circular logic, a question is deemed correct if the number of its variations answered correctly is greater than or equal to num, calculating accuracy. + +OpenCompass configuration file: + +```python +from mmengine.config import read_base +from opencompass.datasets.circular import CircularCEvalDataset + +with read_base(): + from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets + +for d in ceval_datasets: + # Overloading the load method + d['type'] = CircularCEvalDataset + # Renaming for differentiation from non-circular evaluation versions + d['abbr'] = d['abbr'] + '-circular-4' + # Overloading the evaluation method + d['eval_cfg']['evaluator'] = {'type': CircularEvaluator} + +# The dataset after the above operations looks like this: +# dict( +# type=CircularCEvalDataset, +# path='./data/ceval/formal_ceval', # Unchanged +# name='computer_network', # Unchanged +# abbr='ceval-computer_network-circular-4', +# reader_cfg=dict(...), # Unchanged +# infer_cfg=dict(...), # Unchanged +# eval_cfg=dict(evaluator=dict(type=CircularEvaluator), ...), +# ) +``` + +Additionally, for better presentation of results in CircularEval, consider using the following summarizer: + +```python + + +from mmengine.config import read_base +from opencompass.summarizers import CircularSummarizer + +with read_base(): + from ...summarizers.groups.ceval.ceval_summary_groups + +new_summary_groups = [] +for item in ceval_summary_groups: + new_summary_groups.append( + { + 'name': item['name'] + '-circular-4', + 'subsets': [i + '-circular-4' for i in item['subsets']], + } + ) + +summarizer = dict( + type=CircularSummarizer, + # Select specific metrics to view + metric_types=['acc_origin', 'perf_circular'], + dataset_abbrs = [ + 'ceval-circular-4', + 'ceval-humanities-circular-4', + 'ceval-stem-circular-4', + 'ceval-social-science-circular-4', + 'ceval-other-circular-4', + ], + summary_groups=new_summary_groups, +) +``` + +For more complex evaluation examples, refer to this sample code: https://github.com/open-compass/opencompass/tree/main/examples/eval_circular.py diff --git a/docs/en/advanced_guides/code_eval.md b/docs/en/advanced_guides/code_eval.md new file mode 100644 index 0000000000000000000000000000000000000000..1e8602dd5f59f685312abe5378584b4ca6dee1ce --- /dev/null +++ b/docs/en/advanced_guides/code_eval.md @@ -0,0 +1,104 @@ +# Code Evaluation Tutorial + +This tutorial primarily focuses on evaluating a model's coding proficiency, using `humaneval` and `mbpp` as examples. + +## pass@1 + +If you only need to generate a single response to evaluate the pass@1 performance, you can directly use [configs/datasets/humaneval/humaneval_gen_8e312c.py](https://github.com/open-compass/opencompass/blob/main/configs/datasets/humaneval/humaneval_gen_8e312c.py) and [configs/datasets/mbpp/deprecated_mbpp_gen_1e1056.py](https://github.com/open-compass/opencompass/blob/main/configs/datasets/mbpp/deprecated_mbpp_gen_1e1056.py), referring to the general [quick start tutorial](../get_started/quick_start.md). + +For multilingual evaluation, please refer to the [Multilingual Code Evaluation Tutorial](./code_eval_service.md). + +## pass@k + +If you need to generate multiple responses for a single example to evaluate the pass@k performance, consider the following two situations. Here we take 10 responses as an example: + +### Typical Situation + +For most models that support the `num_return_sequences` parameter in HF's generation, we can use it directly to obtain multiple responses. Refer to the following configuration file: + +```python +from opencompass.datasets import MBPPDatasetV2, MBPPPassKEvaluator + +with read_base(): + from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets + from .datasets.mbpp.deprecated_mbpp_gen_1e1056 import mbpp_datasets + +mbpp_datasets[0]['type'] = MBPPDatasetV2 +mbpp_datasets[0]['eval_cfg']['evaluator']['type'] = MBPPPassKEvaluator +mbpp_datasets[0]['reader_cfg']['output_column'] = 'test_column' + +datasets = [] +datasets += humaneval_datasets +datasets += mbpp_datasets + +models = [ + dict( + type=HuggingFaceCausalLM, + ..., + generation_kwargs=dict( + num_return_sequences=10, + do_sample=True, + top_p=0.95, + temperature=0.8, + ), + ..., + ) +] +``` + +For `mbpp`, new changes are needed in the dataset and evaluation, so we simultaneously modify the `type`, `eval_cfg.evaluator.type`, `reader_cfg.output_column` fields to accommodate these requirements. + +We also need model responses with randomness, thus setting the `generation_kwargs` parameter is necessary. Note that we need to set `num_return_sequences` to get the number of responses. + +Note: `num_return_sequences` must be greater than or equal to k, as pass@k itself is a probability estimate. + +You can specifically refer to the following configuration file [examples/eval_code_passk.py](https://github.com/open-compass/opencompass/blob/main/examples/eval_code_passk.py) + +### For Models That Do Not Support Multiple Responses + +This applies to some HF models with poorly designed APIs or missing features. In this case, we need to repeatedly construct datasets to achieve multiple response effects. Refer to the following configuration: + +```python +from opencompass.datasets import MBPPDatasetV2, MBPPPassKEvaluator + +with read_base(): + from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets + from .datasets.mbpp.deprecated_mbpp_gen_1e1056 import mbpp_datasets + +humaneval_datasets[0]['abbr'] = 'openai_humaneval_pass10' +humaneval_datasets[0]['num_repeats'] = 10 +mbpp_datasets[0]['abbr'] = 'mbpp_pass10' +mbpp_datasets[0]['num_repeats'] = 10 +mbpp_datasets[0]['type'] = MBPPDatasetV2 +mbpp_datasets[0]['eval_cfg']['evaluator']['type'] = MBPPPassKEvaluator +mbpp_datasets[0]['reader_cfg']['output_column'] = 'test_column' + +datasets = [] +datasets += humaneval_datasets +datasets += mbpp_datasets + +models = [ + dict( + type=HuggingFaceCausalLM, + ..., + generation_kwargs=dict( + do_sample=True, + top_p=0.95, + temperature=0.8, + ), + ..., + ) +] +``` + +Since the dataset's prompt has not been modified, we need to replace the corresponding fields to achieve the purpose of repeating the dataset. +You need to modify these fields: + +- `num_repeats`: the number of times the dataset is repeated +- `abbr`: It's best to modify the dataset abbreviation along with the number of repetitions because the number of datasets will change, preventing potential issues arising from discrepancies with the values in `.cache/dataset_size.json`. + +For `mbpp`, modify the `type`, `eval_cfg.evaluator.type`, `reader_cfg.output_column` fields as well. + +We also need model responses with randomness, thus setting the `generation_kwargs` parameter is necessary. + +You can specifically refer to the following configuration file [examples/eval_code_passk_repeat_dataset.py](https://github.com/open-compass/opencompass/blob/main/examples/eval_code_passk_repeat_dataset.py) diff --git a/docs/en/advanced_guides/code_eval_service.md b/docs/en/advanced_guides/code_eval_service.md new file mode 100644 index 0000000000000000000000000000000000000000..8c827a1a5390485ce3c3f523486400c559f72cbe --- /dev/null +++ b/docs/en/advanced_guides/code_eval_service.md @@ -0,0 +1,224 @@ +# Code Evaluation Docker Tutorial + +To complete the LLM code capability evaluation, we need to build a separate evaluation environment to avoid executing erroneous code in the development environment, which would inevitably cause losses. The code evaluation service currently used by OpenCompass can refer to the [code-evaluator](https://github.com/open-compass/code-evaluator) project. The following will introduce evaluation tutorials around the code evaluation service. + +1. humaneval-x + +This is a multi-programming language dataset [humaneval-x](https://huggingface.co/datasets/THUDM/humaneval-x). +You can download the dataset from this [download link](https://github.com/THUDM/CodeGeeX2/tree/main/benchmark/humanevalx). Please download the language file (××.jsonl.gz) that needs to be evaluated and place it in the `./data/humanevalx` folder. + +The currently supported languages are `python`, `cpp`, `go`, `java`, `js`. + +2. DS1000 + +This is a Python multi-algorithm library dataset [ds1000](https://github.com/xlang-ai/DS-1000). +You can download the dataset from this [download link](https://github.com/xlang-ai/DS-1000/blob/main/ds1000_data.zip). + +The currently supported algorithm libraries are `Pandas`, `Numpy`, `Tensorflow`, `Scipy`, `Sklearn`, `Pytorch`, `Matplotlib`. + +## Launching the Code Evaluation Service + +1. Ensure you have installed Docker, please refer to [Docker installation document](https://docs.docker.com/engine/install/). +2. Pull the source code of the code evaluation service project and build the Docker image. + +Choose the dockerfile corresponding to the dataset you need, and replace `humanevalx` or `ds1000` in the command below. + +```shell +git clone https://github.com/open-compass/code-evaluator.git +docker build -t code-eval-{your-dataset}:latest -f docker/{your-dataset}/Dockerfile . +``` + +3. Create a container with the following commands: + +```shell +# Log output format +docker run -it -p 5000:5000 code-eval-{your-dataset}:latest python server.py + +# Run the program in the background +# docker run -itd -p 5000:5000 code-eval-{your-dataset}:latest python server.py + +# Using different ports +# docker run -itd -p 5001:5001 code-eval-{your-dataset}:latest python server.py --port 5001 +``` + +**Note:** + +- If you encounter a timeout during the evaluation of Go, please use the following command when creating the container. + +```shell +docker run -it -p 5000:5000 -e GO111MODULE=on -e GOPROXY=https://goproxy.io code-eval-{your-dataset}:latest python server.py +``` + +4. To ensure you have access to the service, use the following command to check the inference environment and evaluation service connection status. (If both inferences and code evaluations run on the same host, skip this step.) + +```shell +ping your_service_ip_address +telnet your_service_ip_address your_service_port +``` + +## Local Code Evaluation + +When the model inference and code evaluation services are running on the same host or within the same local area network, direct code reasoning and evaluation can be performed. **Note: DS1000 is currently not supported, please proceed with remote evaluation.** + +### Configuration File + +We provide [the configuration file](https://github.com/open-compass/opencompass/blob/main/examples/eval_codegeex2.py) of using `humanevalx` for evaluation on `codegeex2` as reference. + +The dataset and related post-processing configurations files can be found at this [link](https://github.com/open-compass/opencompass/tree/main/configs/datasets/humanevalx) with attention paid to the `evaluator` field in the humanevalx_eval_cfg_dict. + +```python +from opencompass.openicl.icl_prompt_template import PromptTemplate +from opencompass.openicl.icl_retriever import ZeroRetriever +from opencompass.openicl.icl_inferencer import GenInferencer +from opencompass.datasets import HumanevalXDataset, HumanevalXEvaluator + +humanevalx_reader_cfg = dict( + input_columns=['prompt'], output_column='task_id', train_split='test') + +humanevalx_infer_cfg = dict( + prompt_template=dict( + type=PromptTemplate, + template='{prompt}'), + retriever=dict(type=ZeroRetriever), + inferencer=dict(type=GenInferencer, max_out_len=1024)) + +humanevalx_eval_cfg_dict = { + lang : dict( + evaluator=dict( + type=HumanevalXEvaluator, + language=lang, + ip_address="localhost", # replace to your code_eval_server ip_address, port + port=5000), # refer to https://github.com/open-compass/code-evaluator to launch a server + pred_role='BOT') + for lang in ['python', 'cpp', 'go', 'java', 'js'] # do not support rust now +} + +humanevalx_datasets = [ + dict( + type=HumanevalXDataset, + abbr=f'humanevalx-{lang}', + language=lang, + path='./data/humanevalx', + reader_cfg=humanevalx_reader_cfg, + infer_cfg=humanevalx_infer_cfg, + eval_cfg=humanevalx_eval_cfg_dict[lang]) + for lang in ['python', 'cpp', 'go', 'java', 'js'] +] +``` + +### Task Launch + +Refer to the [Quick Start](../get_started.html) + +## Remote Code Evaluation + +Model inference and code evaluation services located in different machines which cannot be accessed directly require prior model inference before collecting the code evaluation results. The configuration file and inference process can be reused from the previous tutorial. + +### Collect Inference Results(Only for Humanevalx) + +In OpenCompass's tools folder, there is a script called `collect_code_preds.py` provided to process and collect the inference results after providing the task launch configuration file during startup along with specifying the working directory used corresponding to the task. +It is the same with `-r` option in `run.py`. More details can be referred through the [documentation](https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html#launching-evaluation). + +```shell +python tools/collect_code_preds.py [config] [-r latest] +``` + +The collected results will be organized as following under the `-r` folder: + +``` +workdir/humanevalx +├── codegeex2-6b +│ ├── humanevalx_cpp.json +│ ├── humanevalx_go.json +│ ├── humanevalx_java.json +│ ├── humanevalx_js.json +│ └── humanevalx_python.json +├── CodeLlama-13b +│ ├── ... +├── CodeLlama-13b-Instruct +│ ├── ... +├── CodeLlama-13b-Python +│ ├── ... +├── ... +``` + +For DS1000, you just need to obtain the corresponding prediction file generated by `opencompass`. + +### Code Evaluation + +Make sure your code evaluation service is started, and use `curl` to request: + +#### The following only supports Humanevalx + +```shell +curl -X POST -F 'file=@{result_absolute_path}' -F 'dataset={dataset/language}' {your_service_ip_address}:{your_service_port}/evaluate +``` + +For example: + +```shell +curl -X POST -F 'file=@./examples/humanevalx/python.json' -F 'dataset=humanevalx/python' localhost:5000/evaluate +``` + +The we have: + +``` +"{\"pass@1\": 37.19512195121951%}" +``` + +Additionally, we offer an extra option named `with_prompt`(Defaults to `True`), since some models(like `WizardCoder`) generate complete codes without requiring the form of concatenating prompt and prediction. You may refer to the following commands for evaluation. + +```shell +curl -X POST -F 'file=@./examples/humanevalx/python.json' -F 'dataset=humanevalx/python' -H 'with-prompt: False' localhost:5000/evaluate +``` + +#### The following only supports DS1000 + +Make sure the code evaluation service is started, then use `curl` to submit a request: + +```shell +curl -X POST -F 'file=@./internlm-chat-7b-hf-v11/ds1000_Numpy.json' localhost:5000/evaluate +``` + +DS1000 supports additional debug parameters. Be aware that a large amount of log will be generated when it is turned on: + +- `full`: Additional print out of the original prediction for each error sample, post-processing prediction, running program, and final error. +- `half`: Additional print out of the running program and final error for each error sample. +- `error`: Additional print out of the final error for each error sample. + +```shell +curl -X POST -F 'file=@./internlm-chat-7b-hf-v11/ds1000_Numpy.json' -F 'debug=error' localhost:5000/evaluate +``` + +You can also modify the `num_workers` in the same way to control the degree of parallelism. + +## Advanced Tutorial + +Besides evaluating the supported HUMANEVAList data set, users might also need: + +### Support New Dataset + +Please refer to the [tutorial on supporting new datasets](./new_dataset.md). + +### Modify Post-Processing + +1. For local evaluation, follow the post-processing section in the tutorial on supporting new datasets to modify the post-processing method. +2. For remote evaluation, please modify the post-processing part in the tool's `collect_code_preds.py`. +3. Some parts of post-processing could also be modified in the code evaluation service, more information will be available in the next section. + +### Debugging Code Evaluation Service + +When supporting new datasets or modifying post-processors, it is possible that modifications need to be made to the original code evaluation service. Please make changes based on the following steps: + +1. Remove the installation of the `code-evaluator` in `Dockerfile`, mount the `code-evaluator` when starting the container instead: + +```shell +docker run -it -p 5000:5000 -v /local/path/of/code-evaluator:/workspace/code-evaluator code-eval:latest bash +``` + +2. Install and start the code evaluation service locally. At this point, any necessary modifications can be made to the local copy of the `code-evaluator`. + +```shell +cd code-evaluator && pip install -r requirements.txt +python server.py +``` diff --git a/docs/en/advanced_guides/contamination_eval.md b/docs/en/advanced_guides/contamination_eval.md new file mode 100644 index 0000000000000000000000000000000000000000..ecbf6ec6920de2342290592239fe3137bdb8bc25 --- /dev/null +++ b/docs/en/advanced_guides/contamination_eval.md @@ -0,0 +1,124 @@ +# Data Contamination Assessment + +**Data Contamination** refers to the phenomenon where data intended for downstream testing tasks appear in the training data of large language models (LLMs), resulting in artificially inflated performance metrics in downstream tasks (such as summarization, natural language inference, text classification), which do not accurately reflect the model's true generalization capabilities. + +Since the source of data contamination lies in the training data used by LLMs, the most direct method to detect data contamination is to collide test data with training data and then report the extent of overlap between the two. The classic GPT-3 [paper](https://arxiv.org/pdf/2005.14165.pdf) reported on this in Table C.1. + +However, today's open-source community often only publishes model parameters, not training datasets. In such cases, how to determine the presence and extent of data contamination remains unsolved. OpenCompass offers two possible solutions. + +## Contamination Data Annotation Based on Self-Built Co-Distribution Data + +Referencing the method mentioned in Section 5.2 of [Skywork](https://arxiv.org/pdf/2310.19341.pdf), we directly used the dataset [mock_gsm8k_test](https://huggingface.co/datasets/Skywork/mock_gsm8k_test) uploaded to HuggingFace by Skywork. + +In this method, the authors used GPT-4 to synthesize data similar to the original GSM8K style, and then calculated the perplexity on the GSM8K training set (train), GSM8K test set (test), and GSM8K reference set (ref). Since the GSM8K reference set was newly generated, the authors considered it as clean, not belonging to any training set of any model. They posited: + +- If the test set's perplexity is significantly lower than the reference set's, the test set might have appeared in the model's training phase; +- If the training set's perplexity is significantly lower than the test set's, the training set might have been overfitted by the model. + +The following configuration file can be referenced: + +```python +from mmengine.config import read_base + +with read_base(): + from .datasets.gsm8k_contamination.gsm8k_contamination_ppl_ecdd22 import gsm8k_datasets # includes training, test, and reference sets + from .models.qwen.hf_qwen_7b import models as hf_qwen_7b_model # model under review + from .models.yi.hf_yi_6b import models as hf_yi_6b_model + +datasets = [*gsm8k_datasets] +models = [*hf_qwen_7b_model, *hf_yi_6b_model] +``` + +An example output is as follows: + +```text +dataset version metric mode internlm-7b-hf qwen-7b-hf yi-6b-hf chatglm3-6b-base-hf qwen-14b-hf baichuan2-13b-base-hf internlm-20b-hf aquila2-34b-hf ... +--------------- --------- ----------- ------- ---------------- ------------ ---------- --------------------- ------------- ----------------------- ----------------- ---------------- ... +gsm8k-train-ppl 0b8e46 average_ppl unknown 1.5 0.78 1.37 1.16 0.5 0.76 1.41 0.78 ... +gsm8k-test-ppl 0b8e46 average_ppl unknown 1.56 1.33 1.42 1.3 1.15 1.13 1.52 1.16 ... +gsm8k-ref-ppl f729ba average_ppl unknown 1.55 1.2 1.43 1.35 1.27 1.19 1.47 1.35 ... +``` + +Currently, this solution only supports the GSM8K dataset. We welcome the community to contribute more datasets. + +Consider cite the following paper if you find it helpful: + +```bibtex +@misc{2023opencompass, + title={OpenCompass: A Universal Evaluation Platform for Foundation Models}, + author={OpenCompass Contributors}, + howpublished = {\url{https://github.com/open-compass/opencompass}}, + year={2023} +} +@misc{wei2023skywork, + title={Skywork: A More Open Bilingual Foundation Model}, + author={Tianwen Wei and Liang Zhao and Lichang Zhang and Bo Zhu and Lijie Wang and Haihua Yang and Biye Li and Cheng Cheng and Weiwei Lü and Rui Hu and Chenxia Li and Liu Yang and Xilin Luo and Xuejie Wu and Lunan Liu and Wenjun Cheng and Peng Cheng and Jianhao Zhang and Xiaoyu Zhang and Lei Lin and Xiaokun Wang and Yutuan Ma and Chuanhai Dong and Yanqi Sun and Yifu Chen and Yongyi Peng and Xiaojuan Liang and Shuicheng Yan and Han Fang and Yahui Zhou}, + year={2023}, + eprint={2310.19341}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## Contamination Data Annotation Based on Classic Pre-trained Sets + +Thanks to [Contamination_Detector](https://github.com/liyucheng09/Contamination_Detector) and @liyucheng09 for providing this method. + +In this method, the authors search the test datasets (such as C-Eval, ARC, HellaSwag, etc.) using the Common Crawl database and Bing search engine, then mark each test sample as clean / question contaminated / both question and answer contaminated. + +During testing, OpenCompass + +will report the accuracy or perplexity of ceval on subsets composed of these three labels. Generally, the accuracy ranges from low to high: clean, question contaminated, both question and answer contaminated subsets. The authors believe: + +- If the performance of the three is relatively close, the contamination level of the model on that test set is light; otherwise, it is heavy. + +The following configuration file can be referenced [link](https://github.com/open-compass/opencompass/blob/main/examples/eval_contamination.py): + +```python +from mmengine.config import read_base + +with read_base(): + from .datasets.ceval.ceval_clean_ppl import ceval_datasets # ceval dataset with contamination tags + from .models.yi.hf_yi_6b import models as hf_yi_6b_model # model under review + from .models.qwen.hf_qwen_7b import models as hf_qwen_7b_model + from .summarizers.contamination import ceval_summarizer as summarizer # output formatting + +datasets = [*ceval_datasets] +models = [*hf_yi_6b_model, *hf_qwen_7b_model] +``` + +An example output is as follows: + +```text +dataset version mode yi-6b-hf - - qwen-7b-hf - - ... +---------------------------------------------- --------- ------ ---------------- ----------------------------- --------------------------------------- ---------------- ----------------------------- --------------------------------------- ... +- - - accuracy - clean accuracy - input contaminated accuracy - input-and-label contaminated accuracy - clean accuracy - input contaminated accuracy - input-and-label contaminated ... +... +ceval-humanities - ppl 74.42 75.00 82.14 67.44 50.00 70.54 ... +ceval-stem - ppl 53.70 57.14 85.61 47.41 52.38 67.63 ... +ceval-social-science - ppl 81.60 84.62 83.09 76.00 61.54 72.79 ... +ceval-other - ppl 72.31 73.91 75.00 58.46 39.13 61.88 ... +ceval-hard - ppl 44.35 37.50 70.00 41.13 25.00 30.00 ... +ceval - ppl 67.32 71.01 81.17 58.97 49.28 67.82 ... +``` + +Currently, this solution only supports the C-Eval, MMLU, HellaSwag and ARC. [Contamination_Detector](https://github.com/liyucheng09/Contamination_Detector) also includes CSQA and WinoGrande, but these have not yet been implemented in OpenCompass. We welcome the community to contribute more datasets. + +Consider cite the following paper if you find it helpful: + +```bibtex +@misc{2023opencompass, + title={OpenCompass: A Universal Evaluation Platform for Foundation Models}, + author={OpenCompass Contributors}, + howpublished = {\url{https://github.com/open-compass/opencompass}}, + year={2023} +} +@article{Li2023AnOS, + title={An Open Source Data Contamination Report for Llama Series Models}, + author={Yucheng Li}, + journal={ArXiv}, + year={2023}, + volume={abs/2310.17589}, + url={https://api.semanticscholar.org/CorpusID:264490711} +} +``` diff --git a/docs/en/advanced_guides/custom_dataset.md b/docs/en/advanced_guides/custom_dataset.md new file mode 100644 index 0000000000000000000000000000000000000000..d4b90dc1ee6e2576d6955ff8f2ccf123912f98c2 --- /dev/null +++ b/docs/en/advanced_guides/custom_dataset.md @@ -0,0 +1,267 @@ +# Dataset Quick Evaluation Tutorial + +OpenCompass provides two paths for quickly evaluating the provided data, the data format protocol based on ChatMLDataset and the data format protocol based on CustomDataset. +Compared to the complete dataset integration process in [new_dataset.md](./new_dataset.md), these two evaluation paths are more convenient and efficient, being able to directly enter the evaluation process without adding new configuration files. +But if you have specific needs for custom reading/inference/evaluation, it is recommended to still follow the complete integration process to add a new dataset. + +## Data Format Protocol and Fast Evaluation Based on ChatMLDataset + +OpenCompass has recently launched a dataset evaluation mode based on the ChatML dialogue template, which allow users to provide a dataset .json file that conforms to the ChatML dialogue template, and simply set the dataset information config like model configs to start evaluating directly. + +### Format Requirements for Data Files + +This evaluation method only supports data files in `.json` format, and each sample must comply with the following format: + +The format of a text-only dataset with a simple structure: + +```jsonl +{ + "question":[ + { + "role": "system" # Omittable + "content": Str + }, + { + "role": "user", + "content": Str + } + ], + "answer":[ + Str + ] +} +{ + ... +} +... +``` + +The format of multiple rounds and multiple modes datasets: + +```jsonl +{ + "question":[ + { + "role": "system", + "content": Str, + }, + { + "role": "user", + "content": Str or List + [ + { + "type": Str, # "image" + "image_url": Str, + }, + ... + { + "type": Str, # "text" + "text": Str, + }, + ] + }, + { + "role": "assistant", + "content": Str + }, + { + "role": "user", + "content": Str or List + }, + ... + ], + "answer":[ + Str, + Str, + ... + ] +} +{ + ... +} +... +``` + +(As OpenCompass currently does not support multi-mode evaluation, the template above is for reference only.) + +When ChatMLDataset reading `.json` files, it will use `pydantic` to perform simple format validation on the files. +You can use `tools/chatml_fformat_test.py` to check your provided data file. + +After format checking, please add a config dictionary named `chatml_datasets` in your running config file to convert the data file into an OpenCompass dataset at runtime. +An example is as follows: + +```python +chatml_datasets = [ + dict( + abbr='YOUR_DATASET_NAME', + path='YOUR_DATASET_PATH', + evaluator=dict( + type='cascade_evaluator', + rule_evaluator=dict( + type='math_evaluator', + ), + llm_evaluator=dict( + type='llm_evaluator', + prompt="YOUR_JUDGE_PROMPT", + judge_cfg=dict(), # YOUR Judge Model Config + ) + ), + n=1, # Repeat Number + ), +] +``` + +The ChatML evaluation module currently provides four preset evaluators, `mcq_rule_evaluator` used for MCQ evaluation, `math_evaluator` used for latex mathematical formula evaluation, `llm_evaluator` used for evaluating answers that are open-ended or difficult to extract), and `cascade_evaluator`, an evaluation mode composed of rule and LLM evaluators cascaded together. + +In addition, if you have a long-term need to use datasets based on ChatML templates, you can contribute your dataset config to `opencompass/config/chatml_datasets`. +An eval example of calling these dataset configs is provided in `examples/evalchat_datasets.py`. + +## Data Format Protocol and Fast Evaluation Based on CustomsDataset + +(This module is no longer being updated, but it can still be used if there is a need for cli- quick evaluation.) + +This module support two types of tasks: multiple choice (`mcq`) and question & answer (`qa`). For `mcq`, both ppl and gen inferences are supported; for `qa`, gen inference is supported. + +### Dataset Format + +We support datasets in both `.jsonl` and `.csv` formats. + +#### Multiple Choice (`mcq`) + +For `mcq` datasets, the default fields are as follows: + +- `question`: The stem of the multiple-choice question. +- `A`, `B`, `C`, ...: Single uppercase letters representing the options, with no limit on the number. Defaults to parsing consecutive letters strating from `A` as options. +- `answer`: The correct answer to the multiple-choice question, which must be one of the options used above, such as `A`, `B`, `C`, etc. + +Non-default fields will be read in but are not used by default. To use them, specify in the `.meta.json` file. + +An example of the `.jsonl` format: + +```jsonl +{"question": "165+833+650+615=", "A": "2258", "B": "2263", "C": "2281", "answer": "B"} +{"question": "368+959+918+653+978=", "A": "3876", "B": "3878", "C": "3880", "answer": "A"} +{"question": "776+208+589+882+571+996+515+726=", "A": "5213", "B": "5263", "C": "5383", "answer": "B"} +{"question": "803+862+815+100+409+758+262+169=", "A": "4098", "B": "4128", "C": "4178", "answer": "C"} +``` + +An example of the `.csv` format: + +```csv +question,A,B,C,answer +127+545+588+620+556+199=,2632,2635,2645,B +735+603+102+335+605=,2376,2380,2410,B +506+346+920+451+910+142+659+850=,4766,4774,4784,C +504+811+870+445=,2615,2630,2750,B +``` + +#### Question & Answer (`qa`) + +For `qa` datasets, the default fields are as follows: + +- `question`: The stem of the question & answer question. +- `answer`: The correct answer to the question & answer question. It can be missing, indicating the dataset has no correct answer. + +Non-default fields will be read in but are not used by default. To use them, specify in the `.meta.json` file. + +An example of the `.jsonl` format: + +```jsonl +{"question": "752+361+181+933+235+986=", "answer": "3448"} +{"question": "712+165+223+711=", "answer": "1811"} +{"question": "921+975+888+539=", "answer": "3323"} +{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"} +``` + +An example of the `.csv` format: + +```csv +question,answer +123+147+874+850+915+163+291+604=,3967 +149+646+241+898+822+386=,3142 +332+424+582+962+735+798+653+214=,4700 +649+215+412+495+220+738+989+452=,4170 +``` + +### Command Line List + +Custom datasets can be directly called for evaluation through the command line. + +```bash +python run.py \ + --models hf_llama2_7b \ + --custom-dataset-path xxx/test_mcq.csv \ + --custom-dataset-data-type mcq \ + --custom-dataset-infer-method ppl +``` + +```bash +python run.py \ + --models hf_llama2_7b \ + --custom-dataset-path xxx/test_qa.jsonl \ + --custom-dataset-data-type qa \ + --custom-dataset-infer-method gen +``` + +In most cases, `--custom-dataset-data-type` and `--custom-dataset-infer-method` can be omitted. OpenCompass will + +set them based on the following logic: + +- If options like `A`, `B`, `C`, etc., can be parsed from the dataset file, it is considered an `mcq` dataset; otherwise, it is considered a `qa` dataset. +- The default `infer_method` is `gen`. + +### Configuration File + +In the original configuration file, simply add a new item to the `datasets` variable. Custom datasets can be mixed with regular datasets. + +```python +datasets = [ + {"path": "xxx/test_mcq.csv", "data_type": "mcq", "infer_method": "ppl"}, + {"path": "xxx/test_qa.jsonl", "data_type": "qa", "infer_method": "gen"}, +] +``` + +### Supplemental Information for Dataset `.meta.json` + +OpenCompass will try to parse the input dataset file by default, so in most cases, the `.meta.json` file is **not necessary**. However, if the dataset field names are not the default ones, or custom prompt words are required, it should be specified in the `.meta.json` file. + +The file is placed in the same directory as the dataset, with the filename followed by `.meta.json`. An example file structure is as follows: + +```tree +. +├── test_mcq.csv +├── test_mcq.csv.meta.json +├── test_qa.jsonl +└── test_qa.jsonl.meta.json +``` + +Possible fields in this file include: + +- `abbr` (str): Abbreviation of the dataset, serving as its ID. +- `data_type` (str): Type of dataset, options are `mcq` and `qa`. +- `infer_method` (str): Inference method, options are `ppl` and `gen`. +- `human_prompt` (str): User prompt template for generating prompts. Variables in the template are enclosed in `{}`, like `{question}`, `{opt1}`, etc. If `template` exists, this field will be ignored. +- `bot_prompt` (str): Bot prompt template for generating prompts. Variables in the template are enclosed in `{}`, like `{answer}`, etc. If `template` exists, this field will be ignored. +- `template` (str or dict): Question template for generating prompts. Variables in the template are enclosed in `{}`, like `{question}`, `{opt1}`, etc. The relevant syntax is in [here](../prompt/prompt_template.md) regarding `infer_cfg['prompt_template']['template']`. +- `input_columns` (list): List of input fields for reading data. +- `output_column` (str): Output field for reading data. +- `options` (list): List of options for reading data, valid only when `data_type` is `mcq`. + +For example: + +```json +{ + "human_prompt": "Question: 127 + 545 + 588 + 620 + 556 + 199 =\nA. 2632\nB. 2635\nC. 2645\nAnswer: Let's think step by step, 127 + 545 + 588 + 620 + 556 + 199 = 672 + 588 + 620 + 556 + 199 = 1260 + 620 + 556 + 199 = 1880 + 556 + 199 = 2436 + 199 = 2635. So the answer is B.\nQuestion: {question}\nA. {A}\nB. {B}\nC. {C}\nAnswer: ", + "bot_prompt": "{answer}" +} +``` + +or + +```json +{ + "template": "Question: {my_question}\nX. {X}\nY. {Y}\nZ. {Z}\nW. {W}\nAnswer:", + "input_columns": ["my_question", "X", "Y", "Z", "W"], + "output_column": "my_answer", +} +``` diff --git a/docs/en/advanced_guides/evaluation_lightllm.md b/docs/en/advanced_guides/evaluation_lightllm.md new file mode 100644 index 0000000000000000000000000000000000000000..5d27a5b6d217d1514e40fec5908f511453c74aac --- /dev/null +++ b/docs/en/advanced_guides/evaluation_lightllm.md @@ -0,0 +1,71 @@ +# Evaluation with Lightllm + +We now support the evaluation of large language models using [Lightllm](https://github.com/ModelTC/lightllm) for inference. Developed by SenseTime, LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. Lightllm provides support for various large Language models, allowing users to perform model inference through Lightllm, locally deploying it as a service. During the evaluation process, OpenCompass feeds data to Lightllm through an API and processes the response. OpenCompass has been adapted for compatibility with Lightllm, and this tutorial will guide you on using OpenCompass to evaluate models with Lightllm as the inference backend. + +## Setup + +### Install OpenCompass + +Please follow the [instructions](https://opencompass.readthedocs.io/en/latest/get_started/installation.html) to install the OpenCompass and prepare the evaluation datasets. + +### Install Lightllm + +Please follow the [Lightllm homepage](https://github.com/ModelTC/lightllm) to install the Lightllm. Pay attention to aligning the versions of relevant dependencies, especially the version of the Transformers. + +## Evaluation + +We use the evaluation of Humaneval with the llama2-7B model as an example. + +### Step-1: Deploy the model locally as a service using Lightllm. + +```shell +python -m lightllm.server.api_server --model_dir /path/llama2-7B \ + --host 0.0.0.0 \ + --port 1030 \ + --nccl_port 2066 \ + --max_req_input_len 4096 \ + --max_req_total_len 6144 \ + --tp 1 \ + --trust_remote_code \ + --max_total_token_num 120000 +``` + +\*\*Note: \*\* tp can be configured to enable TensorParallel inference on several gpus, suitable for the inference of very large models. + +\*\*Note: \*\* The max_total_token_num in the above command will affect the throughput performance during testing. It can be configured according to the documentation on the [Lightllm homepage](https://github.com/ModelTC/lightllm). As long as it does not run out of memory, it is often better to set it as high as possible. + +\*\*Note: \*\* If you want to start multiple LightLLM services on the same machine, you need to reconfigure the above port and nccl_port. + +You can use the following Python script to quickly test whether the current service has been successfully started. + +```python +import time +import requests +import json + +url = 'http://localhost:8080/generate' +headers = {'Content-Type': 'application/json'} +data = { + 'inputs': 'What is AI?', + "parameters": { + 'do_sample': False, + 'ignore_eos': False, + 'max_new_tokens': 1024, + } +} +response = requests.post(url, headers=headers, data=json.dumps(data)) +if response.status_code == 200: + print(response.json()) +else: + print('Error:', response.status_code, response.text) +``` + +### Step-2: Evaluate the above model using OpenCompass. + +```shell +python run.py examples/eval_lightllm.py +``` + +You are expected to get the evaluation results after the inference and evaluation. + +\*\*Note: \*\*In `eval_lightllm.py`, please align the configured URL with the service address from the previous step. diff --git a/docs/en/advanced_guides/evaluation_lmdeploy.md b/docs/en/advanced_guides/evaluation_lmdeploy.md new file mode 100644 index 0000000000000000000000000000000000000000..bfacd48816f6bf5b765eecf2ea1f8878cba2c72f --- /dev/null +++ b/docs/en/advanced_guides/evaluation_lmdeploy.md @@ -0,0 +1,88 @@ +# Evaluation with LMDeploy + +We now support evaluation of models accelerated by the [LMDeploy](https://github.com/InternLM/lmdeploy). LMDeploy is a toolkit designed for compressing, deploying, and serving LLM. It has a remarkable inference performance. We now illustrate how to evaluate a model with the support of LMDeploy in OpenCompass. + +## Setup + +### Install OpenCompass + +Please follow the [instructions](https://opencompass.readthedocs.io/en/latest/get_started/installation.html) to install the OpenCompass and prepare the evaluation datasets. + +### Install LMDeploy + +Install lmdeploy via pip (python 3.8+) + +```shell +pip install lmdeploy +``` + +The default prebuilt package is compiled on CUDA 12. However, if CUDA 11+ is required, you can install lmdeploy by: + +```shell +export LMDEPLOY_VERSION=0.6.0 +export PYTHON_VERSION=310 +pip install https://github.com/InternLM/lmdeploy/releases/download/v${LMDEPLOY_VERSION}/lmdeploy-${LMDEPLOY_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux2014_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118 +``` + +## Evaluation + +When evaluating a model, it is necessary to prepare an evaluation configuration that specifies information such as the evaluation dataset, the model, and inference parameters. + +Taking [internlm2-chat-7b](https://huggingface.co/internlm/internlm2-chat-7b) as an example, the evaluation config is as follows: + +```python +# configure the dataset +from mmengine.config import read_base + + +with read_base(): + # choose a list of datasets + from .datasets.mmlu.mmlu_gen_a484b3 import mmlu_datasets + from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets + from .datasets.triviaqa.triviaqa_gen_2121ce import triviaqa_datasets + from opencompass.configs.datasets.gsm8k.gsm8k_0shot_v2_gen_a58960 import \ + gsm8k_datasets + # and output the results in a chosen format + from .summarizers.medium import summarizer + +datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), []) + +# configure lmdeploy +from opencompass.models import TurboMindModelwithChatTemplate + + + +# configure the model +models = [ + dict( + type=TurboMindModelwithChatTemplate, + abbr=f'internlm2-chat-7b-lmdeploy', + # model path, which can be the address of a model repository on the Hugging Face Hub or a local path + path='internlm/internlm2-chat-7b', + # inference backend of LMDeploy. It can be either 'turbomind' or 'pytorch'. + # If the model is not supported by 'turbomind', it will fallback to + # 'pytorch' + backend='turbomind', + # For the detailed engine config and generation config, please refer to + # https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/messages.py + engine_config=dict(tp=1), + gen_config=dict(do_sample=False), + # the max size of the context window + max_seq_len=7168, + # the max number of new tokens + max_out_len=1024, + # the max number of prompts that LMDeploy receives + # in `generate` function + batch_size=5000, + run_cfg=dict(num_gpus=1), + ) +] +``` + +Place the aforementioned configuration in a file, such as "configs/eval_internlm2_lmdeploy.py". Then, in the home folder of OpenCompass, start evaluation by the following command: + +```shell +python run.py configs/eval_internlm2_lmdeploy.py -w outputs +``` + +You are expected to get the evaluation results after the inference and evaluation. diff --git a/docs/en/advanced_guides/llm_judge.md b/docs/en/advanced_guides/llm_judge.md new file mode 100644 index 0000000000000000000000000000000000000000..f27b34daabd0e4fb71d1208c71c083e5a4d228be --- /dev/null +++ b/docs/en/advanced_guides/llm_judge.md @@ -0,0 +1,370 @@ +# LLM as Judge Evaluation + +## Introduction + +The GenericLLMEvaluator is particularly useful for scenarios where rule-based methods (like regular expressions) cannot perfectly judge outputs, such as: + +- Cases where models output answer content without option identifiers +- Factual judgment datasets that are difficult to evaluate with rules +- Open-ended responses requiring complex understanding and reasoning +- Evaluation that requires a lot of rules to be designed + +OpenCompass provides the GenericLLMEvaluator component to facilitate LLM-as-judge evaluations. + +## Dataset Format + +The dataset for LLM judge evaluation should be in either JSON Lines (.jsonl) or CSV format. Each entry should contain at least: + +- A problem or question +- A reference answer or gold standard +- (The model's prediction will be generated during evaluation) + +Example JSONL format: + +```json +{"problem": "What is the capital of France?", "answer": "Paris"} +``` + +Example CSV format: + +```csv +problem,answer +"What is the capital of France?","Paris" +``` + +## Configuration + +### Using LLM for Evaluation via Command Line + +Some datasets in OpenCompass already include LLM judge configurations. +You need to use a model service (such as OpenAI or DeepSeek's official API) or start a model service locally using tools like LMDeploy, vLLM, or SGLang. + +Then, you can set the environment variables for the evaluation service and evaluate models using the following commands: + +```bash +export OC_JUDGE_MODEL=Qwen/Qwen2.5-32B-Instruct +export OC_JUDGE_API_KEY=sk-1234 +export OC_JUDGE_API_BASE=http://172.30.56.1:4000/v1 +``` + +Note that by default, OpenCompass will use these three environment variables, but if you use configuration files to configure the evaluation service, these environment variables will not take effect. + +### Using LLM for Evaluation via Configuration Files + +To set up an LLM judge evaluation, you'll need to configure three main components: + +1. Dataset Reader Configuration + +```python +reader_cfg = dict( + input_columns=['problem'], # Column name for the question + output_column='answer' # Column name for the reference answer +) +``` + +2. Inference Configuration + +```python +infer_cfg = dict( + prompt_template=dict( + type=PromptTemplate, + template=dict( + round=[ + dict( + role='HUMAN', + prompt='{problem}', # Template for prompting the model + ), + ] + ), + ), + retriever=dict(type=ZeroRetriever), + inferencer=dict(type=GenInferencer), +) +``` + +3. Evaluation Configuration with LLM Judge + +```python +eval_cfg = dict( + evaluator=dict( + type=GenericLLMEvaluator, # Using LLM as evaluator + prompt_template=dict( + type=PromptTemplate, + template=dict( + begin=[ + dict( + role='SYSTEM', + fallback_role='HUMAN', + prompt="You are a helpful assistant who evaluates the correctness and quality of models' outputs.", + ) + ], + round=[ + dict(role='HUMAN', prompt=YOUR_JUDGE_TEMPLATE), # Template for the judge + ], + ), + ), + dataset_cfg=dict( + type=CustomDataset, + path='path/to/your/dataset', + file_name='your_dataset.jsonl', + reader_cfg=reader_cfg, + ), + judge_cfg=YOUR_JUDGE_MODEL_CONFIG, # Configuration for the judge model + dict_postprocessor=dict(type=generic_llmjudge_postprocess), # Post-processing the judge's output + ), +) +``` + +## Using CustomDataset with GenericLLMEvaluator + +Here's how to set up a complete configuration for LLM judge evaluation: + +```python +from mmengine.config import read_base +from opencompass.models import TurboMindModelwithChatTemplate +from opencompass.datasets import CustomDataset +from opencompass.evaluator import GenericLLMEvaluator +from opencompass.datasets import generic_llmjudge_postprocess +from opencompass.openicl.icl_prompt_template import PromptTemplate +from opencompass.openicl.icl_retriever import ZeroRetriever +from opencompass.openicl.icl_inferencer import GenInferencer + +# Import your judge model configuration +with read_base(): + from opencompass.configs.models.qwen2_5.lmdeploy_qwen2_5_14b_instruct import ( + models as judge_model, + ) + +# Define your judge template +JUDGE_TEMPLATE = """ +Please evaluate whether the following response correctly answers the question. +Question: {problem} +Reference Answer: {answer} +Model Response: {prediction} + +Is the model response correct? If correct, answer "A"; if incorrect, answer "B". +""".strip() + +# Dataset reader configuration +reader_cfg = dict(input_columns=['problem'], output_column='answer') + +# Inference configuration for the model being evaluated +infer_cfg = dict( + prompt_template=dict( + type=PromptTemplate, + template=dict( + round=[ + dict( + role='HUMAN', + prompt='{problem}', + ), + ] + ), + ), + retriever=dict(type=ZeroRetriever), + inferencer=dict(type=GenInferencer), +) + +# Evaluation configuration with LLM judge +eval_cfg = dict( + evaluator=dict( + type=GenericLLMEvaluator, + prompt_template=dict( + type=PromptTemplate, + template=dict( + begin=[ + dict( + role='SYSTEM', + fallback_role='HUMAN', + prompt="You are a helpful assistant who evaluates the correctness and quality of models' outputs.", + ) + ], + round=[ + dict(role='HUMAN', prompt=JUDGE_TEMPLATE), + ], + ), + ), + dataset_cfg=dict( + type=CustomDataset, + path='path/to/your/dataset', + file_name='your_dataset.jsonl', + reader_cfg=reader_cfg, + ), + judge_cfg=judge_model[0], + dict_postprocessor=dict(type=generic_llmjudge_postprocess), + ), + pred_role='BOT', +) + +# Dataset configuration +datasets = [ + dict( + type=CustomDataset, + abbr='my-dataset', + path='path/to/your/dataset', + file_name='your_dataset.jsonl', + reader_cfg=reader_cfg, + infer_cfg=infer_cfg, + eval_cfg=eval_cfg, + ) +] + +# Model configuration for the model being evaluated +models = [ + dict( + type=TurboMindModelwithChatTemplate, + abbr='model-to-evaluate', + path='path/to/your/model', + # ... other model configurations + ) +] + +# Output directory +work_dir = './outputs/llm_judge_eval' +``` + +## GenericLLMEvaluator + +The GenericLLMEvaluator is designed to use an LLM as a judge for evaluating model outputs. Key features include: + +1. Flexible prompt templates for instructing the judge +2. Support for various judge models (local or API-based) +3. Customizable evaluation criteria through prompt engineering +4. Post-processing of judge outputs to extract structured evaluations + +**Important Note**: The current generic version of the judge template only supports outputs in the format of "A" (correct) or "B" (incorrect), and does not support other output formats (like "CORRECT" or "INCORRECT"). This is because the post-processing function `generic_llmjudge_postprocess` is specifically designed to parse this format. + +The evaluator works by: + +1. Taking the original problem, reference answer, and model prediction +2. Formatting them into a prompt for the judge model +3. Parsing the judge's response to determine the evaluation result (looking for "A" or "B") +4. Aggregating results across the dataset + +If you would like to see the full details of evaluation results, you can add `--dump-eval-details` to the command line when you start the job. +Example evaluation output: + +```python +{ + 'accuracy': 75.0, # Percentage of responses judged as correct + 'details': [ + { + 'origin_prompt': """ + Please evaluate whether the following response correctly answers the question. + Question: What is the capital of France? + Reference Answer: Paris + Model Response: Paris + Is the model response correct? If correct, answer "A"; if incorrect, answer "B". +""", + 'gold': 'Paris', + 'prediction': 'A', + }, + # ... more results + ] +} +``` + +## CascadeEvaluator + +OpenCompass also provides a CascadeEvaluator that combines the strengths of rule-based evaluation and LLM-based evaluation. The cascade evaluator has two modes: + +1. **Cascade Mode (parallel=False)**: First evaluates all samples with a rule-based evaluator, then only sends samples that were deemed incorrect by the rule-based evaluation to an LLM judge for re-evaluation. This approach reduces reliance on LLM judgments while maintaining accuracy, thus lowering evaluation costs and time. + +2. **Parallel Mode (parallel=True)**: Evaluates all samples with both the rule-based evaluator and LLM judge, then considers a sample correct if either method marks it as correct. This approach can increase the leniency of evaluation but may result in higher costs since all samples require LLM evaluation. + +### Configuring CascadeEvaluator + +Here's an example of how to configure the CascadeEvaluator: + +```python +# Define a rule-based evaluator +rule_evaluator = dict(type=MATHVerifyEvaluator) + +# Define an LLM judge evaluator +llm_judge_evaluator = dict( + type=GenericLLMEvaluator, + prompt_template=dict( + type=PromptTemplate, + template=dict( + begin=[ + dict( + role='SYSTEM', + fallback_role='HUMAN', + prompt="You are a helpful assistant who evaluates the correctness and quality of models' outputs.", + ) + ], + round=[ + dict(role='HUMAN', prompt=YOUR_JUDGE_TEMPLATE), + ], + ), + ), + dataset_cfg=dict( + type=YourDataset, + path='path/to/your/dataset', + reader_cfg=reader_cfg, + ), + judge_cfg=dict(), # Can use environment variables to configure the judge model +) + +# Configure cascade evaluator (cascade mode) +cascade_evaluator = dict( + type=CascadeEvaluator, + llm_evaluator=llm_judge_evaluator, + rule_evaluator=rule_evaluator, + parallel=False # Cascade mode +) + +# For parallel mode, set parallel=True +parallel_evaluator = dict( + type=CascadeEvaluator, + llm_evaluator=llm_judge_evaluator, + rule_evaluator=rule_evaluator, + parallel=True # Parallel mode +) + +# Use the cascade evaluator in your dataset evaluation config +eval_cfg = dict(evaluator=cascade_evaluator) +``` + +### Evaluation Results + +The cascade evaluator outputs detailed evaluation statistics including: + +- Accuracy of the rule-based evaluation +- Accuracy of the LLM evaluation (for samples that failed rule-based evaluation in cascade mode) +- Final combined accuracy + +Example output: + +```python +{ + 'accuracy': 85.0, # Final accuracy + 'cascade_stats': { + 'total_samples': 100, + 'rule_correct': 70, # Number of samples correct by rule evaluation + 'rule_accuracy': 70.0, # Accuracy of rule evaluation + 'llm_evaluated': 30, # Number of samples evaluated by LLM (failed samples in cascade mode) + 'llm_correct': 15, # Number of samples correct by LLM evaluation + 'llm_accuracy': 50.0, # Accuracy of LLM evaluation + 'final_correct': 85, # Total correct samples + 'final_accuracy': 85.0, # Final accuracy + 'parallel_mode': False, # Whether parallel mode was used + }, + 'details': [ + # Detailed evaluation results for each sample + ] +} +``` + +The cascade evaluator is particularly useful for: + +1. Scenarios that require balancing evaluation cost and accuracy +2. Cases where rule-based evaluators are available but might not be comprehensive +3. Evaluation tasks that need more nuanced judgment for edge cases + +## Complete Example + +For a complete working example using GenericLLMEvaluator +, refer to the `eval_llm_judge.py` file in the examples directory, which demonstrates how to evaluate mathematical problem-solving . + +For a complete working example using CascadeEvaluator, refer to the `eval_cascade_evaluator.py` file in the examples directory, which demonstrates how to evaluate mathematical problem-solving . diff --git a/docs/en/advanced_guides/longeval.md b/docs/en/advanced_guides/longeval.md new file mode 100644 index 0000000000000000000000000000000000000000..a75fefda8ba316507abe20207fd7d62bc0d27da2 --- /dev/null +++ b/docs/en/advanced_guides/longeval.md @@ -0,0 +1,169 @@ +# Long Context Evaluation Guidance + +## Introduction + +Although large-scale language models (LLMs) such as GPT-4 have demonstrated significant advantages in handling natural language tasks, most current open-source models can only handle texts with a length of a few thousand tokens, which limits their ability to process long contexts such as reading books and writing text summaries. To explore the performance of models in dealing with long contexts, we use the [L-Eval](https://github.com/OpenLMLab/LEval) and [LongBench](https://github.com/THUDM/LongBench) datasets to test the model's ability to handle long contexts. + +## Existing Algorithms and models + +When dealing with long context inputs, the two main challenges faced by large models are the inference time cost and catastrophic forgetting. Recently, a large amount of research has been devoted to extending the model length, focusing on three improvement directions: + +- Attention mechanisms. The ultimate goal of these methods is to reduce the computation cost of query-key pairs, but they may affect the performance of downstream tasks. +- Input methods. Some studies divide long context inputs into chunks or retrieve pre-existing text segments to enhance the model's ability to handle long contexts, but these methods are only effective for some tasks and are difficult to adapt to multiple downstream tasks. +- Position encoding. This research includes RoPE, ALiBi, Position Interpolation etc., which have shown good results in length extrapolation. These methods have been used to train long context models such as ChatGLM2-6B-32k and LongChat-32k. + +First, we introduce some popular position encoding algorithms. + +### RoPE + +RoPE is a type of positional embedding that injects the information of position in Transformer. It encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. A graphic illustration of RoPE is shown below. + +| Complete Form | +Abbreviated Form | +
|---|---|
| + +```python +infer_cfg=dict( + ice_template=dict( + type=PromptTemplate, + template="Q: {question}\nA: {answer}", + ), + prompt_template=dict( + type=PromptTemplate, + template="Q: {question}\nA: {answer}", + ice_token="", + ), + # ... +) +``` + + | ++ +```python +infer_cfg=dict( + ice_template=dict( + type=PromptTemplate, + template="Q: {question}\nA: {answer}", + ice_token="", + ), + # ... +) +``` + + | +
+ The page you are looking for cannot be found. +
++ If you just switched documentation versions, it is likely that the page you were on is moved. You can look for it in + the content table left, or go to the homepage. +
+ + +{% endblock %} diff --git a/docs/zh_cn/_templates/autosummary/class.rst b/docs/zh_cn/_templates/autosummary/class.rst new file mode 100644 index 0000000000000000000000000000000000000000..4c3a7a9abf5c5b14ac3ef3b00a2f070480295358 --- /dev/null +++ b/docs/zh_cn/_templates/autosummary/class.rst @@ -0,0 +1,13 @@ +.. role:: hidden + :class: hidden-section +.. currentmodule:: {{ module }} + + +{{ name | underline}} + +.. autoclass:: {{ name }} + :members: + +.. + autogenerated from _templates/autosummary/class.rst + note it does not have :inherited-members: diff --git a/docs/zh_cn/_templates/callable.rst b/docs/zh_cn/_templates/callable.rst new file mode 100644 index 0000000000000000000000000000000000000000..3a7b9d2b96c76dfa3eb1d8bef56f58f219fe7760 --- /dev/null +++ b/docs/zh_cn/_templates/callable.rst @@ -0,0 +1,14 @@ +.. role:: hidden + :class: hidden-section +.. currentmodule:: {{ module }} + + +{{ name | underline}} + +.. autoclass:: {{ name }} + :members: + :special-members: __call__ + +.. + autogenerated from _templates/callable.rst + note it does not have :inherited-members: diff --git a/docs/zh_cn/advanced_guides/accelerator_intro.md b/docs/zh_cn/advanced_guides/accelerator_intro.md new file mode 100644 index 0000000000000000000000000000000000000000..e8bd7619cbc0d33e6272a8425ca2920efb3d84f7 --- /dev/null +++ b/docs/zh_cn/advanced_guides/accelerator_intro.md @@ -0,0 +1,142 @@ +# 使用 vLLM 或 LMDeploy 来一键式加速评测推理 + +## 背景 + +在 OpenCompass 评测过程中，默认使用 Huggingface 的 transformers 库进行推理，这是一个非常通用的方案，但在某些情况下，我们可能需要更高效的推理方法来加速这一过程，比如借助 VLLM 或 LMDeploy。 + +- [LMDeploy](https://github.com/InternLM/lmdeploy) 是一个用于压缩、部署和服务大型语言模型（LLM）的工具包，由 [MMRazor](https://github.com/open-mmlab/mmrazor) 和 [MMDeploy](https://github.com/open-mmlab/mmdeploy) 团队开发。 +- [vLLM](https://github.com/vllm-project/vllm) 是一个快速且易于使用的 LLM 推理和服务库，具有先进的服务吞吐量、高效的 PagedAttention 内存管理、连续批处理请求、CUDA/HIP 图的快速模型执行、量化技术（如 GPTQ、AWQ、SqueezeLLM、FP8 KV Cache）以及优化的 CUDA 内核。 + +## 加速前准备 + +首先，请检查您要评测的模型是否支持使用 vLLM 或 LMDeploy 进行推理加速。其次，请确保您已经安装了 vLLM 或 LMDeploy，具体安装方法请参考它们的官方文档，下面是参考的安装方法： + +### LMDeploy 安装方法 + +使用 pip (Python 3.8+) 或从 [源码](https://github.com/InternLM/lmdeploy/blob/main/docs/en/build.md) 安装 LMDeploy： + +```bash +pip install lmdeploy +``` + +### VLLM 安装方法 + +使用 pip 或从 [源码](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source) 安装 vLLM： + +```bash +pip install vllm +``` + +## 评测时使用 VLLM 或 LMDeploy + +### 方法1：使用命令行参数来变更推理后端 + +OpenCompass 提供了一键式的评测加速，可以在评测过程中自动将 Huggingface 的 transformers 模型转化为 VLLM 或 LMDeploy 的模型，以便在评测过程中使用。以下是使用默认 Huggingface 版本的 llama3-8b-instruct 模型评测 GSM8k 数据集的样例代码： + +```python +# eval_gsm8k.py +from mmengine.config import read_base + +with read_base(): + # 选择一个数据集列表 + from .datasets.gsm8k.gsm8k_0shot_gen_a58960 import gsm8k_datasets as datasets + # 选择一个感兴趣的模型 + from ..models.hf_llama.hf_llama3_8b_instruct import models +``` + +其中 `hf_llama3_8b_instruct` 为原版 Huggingface 模型配置，内容如下： + +```python +from opencompass.models import HuggingFacewithChatTemplate + +models = [ + dict( + type=HuggingFacewithChatTemplate, + abbr='llama-3-8b-instruct-hf', + path='meta-llama/Meta-Llama-3-8B-Instruct', + max_out_len=1024, + batch_size=8, + run_cfg=dict(num_gpus=1), + stop_words=['<|end_of_text|>', '<|eot_id|>'], + ) +] +``` + +默认 Huggingface 版本的 Llama3-8b-instruct 模型评测 GSM8k 数据集的方式如下： + +```bash +python run.py config/eval_gsm8k.py +``` + +如果需要使用 vLLM 或 LMDeploy 进行加速评测，可以使用下面的脚本： + +```bash +python run.py config/eval_gsm8k.py -a vllm +``` + +或 + +```bash +python run.py config/eval_gsm8k.py -a lmdeploy +``` + +### 方法2：通过部署推理加速服务API来加速评测 + +OpenCompass 还支持通过部署vLLM或LMDeploy的推理加速服务 API 来加速评测，参考步骤如下： + +1. 安装openai包： + +```bash +pip install openai +``` + +2. 部署 vLLM 或 LMDeploy 的推理加速服务 API，具体部署方法请参考它们的官方文档，下面以LMDeploy为例： + +```bash +lmdeploy serve api_server meta-llama/Meta-Llama-3-8B-Instruct --model-name Meta-Llama-3-8B-Instruct --server-port 23333 +``` + +api_server 启动时的参数可以通过命令行`lmdeploy serve api_server -h`查看。 比如，--tp 设置张量并行，--session-len 设置推理的最大上下文窗口长度，--cache-max-entry-count 调整 k/v cache 的内存使用比例等等。 + +3. 服务部署成功后，修改评测脚本，将模型配置中的路径改为部署的服务地址，如下： + +```python +from opencompass.models import OpenAISDK + +api_meta_template = dict( + round=[ + dict(role='HUMAN', api_role='HUMAN'), + dict(role='BOT', api_role='BOT', generate=True), + ], + reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')], +) + +models = [ + dict( + abbr='Meta-Llama-3-8B-Instruct-LMDeploy-API', + type=OpenAISDK, + key='EMPTY', # API key + openai_api_base='http://0.0.0.0:23333/v1', # 服务地址 + path='Meta-Llama-3-8B-Instruct ', # 请求服务时的 model name + tokenizer_path='meta-llama/Meta-Llama-3.1-8B-Instruct', # 请求服务时的 tokenizer name 或 path, 为None时使用默认tokenizer gpt-4 + rpm_verbose=True, # 是否打印请求速率 + meta_template=api_meta_template, # 服务请求模板 + query_per_second=1, # 服务请求速率 + max_out_len=1024, # 最大输出长度 + max_seq_len=4096, # 最大输入长度 + temperature=0.01, # 生成温度 + batch_size=8, # 批处理大小 + retry=3, # 重试次数 + ) +] +``` + +## 加速效果及性能对比 + +下面是使用 VLLM 或 LMDeploy 在单卡 A800 上 Llama-3-8B-Instruct 模型对 GSM8k 数据集进行加速评测的效果及性能对比表： + +| 推理后端 | 精度（Accuracy） | 推理时间（分钟：秒） | 加速比（相对于 Huggingface） | +| ----------- | ---------------- | -------------------- | ---------------------------- | +| Huggingface | 74.22 | 24:26 | 1.0 | +| LMDeploy | 73.69 | 11:15 | 2.2 | +| VLLM | 72.63 | 07:52 | 3.1 | diff --git a/docs/zh_cn/advanced_guides/circular_eval.md b/docs/zh_cn/advanced_guides/circular_eval.md new file mode 100644 index 0000000000000000000000000000000000000000..4bbe03dde2e5ccc604c7ad8d3ef315bcc8d35df1 --- /dev/null +++ b/docs/zh_cn/advanced_guides/circular_eval.md @@ -0,0 +1,111 @@ +# 循环评测 + +## 背景 + +对于选择题而言，当 LLM 给出正确的选项，并不一定代表着它能真正地理解题意并经过推理得出答案，它也有可能是蒙对的。为了将这两种情形区分开，同时也为了降低 LLM 对选项的偏见，我们可以尝试使用循环评测 (CircularEval)。我们会将一道选择题按照打乱选项的方式进行增广，若 LLM 可以在增广后的每道题上均得到正确的答案，那么我们认为在循环评测的意义下，这道题被做对了。 + +## 新增自己的循环评测数据集 + +一般来说，为了将一个数据集使用循环评测的方式进行评测，它的加载方式和评测方式是需要被重写的，OpenCompass 主库和配置文件均需要进行修改。后续我们以 C-Eval 为例进行讲解。 + +OpenCompass 主库： + +```python +from opencompass.datasets.ceval import CEvalDataset +from opencompass.datasets.circular import CircularDatasetMeta + +class CircularCEvalDataset(CEvalDataset, metaclass=CircularDatasetMeta): + # 被重载的数据集类 + dataset_class = CEvalDataset + + # 若原 load 方法得到一 DatasetDict，其哪些 split 需要被循环评测。CEvalDataset load 得到 [dev, val, test]，我们只需要对 val 和 test 进行循环评测，dev 不需要 + default_circular_splits = ['val', 'test'] + + # 需要被打乱的 key 列表 + default_option_keys = ['A', 'B', 'C', 'D'] + + # 若 answer_key 的内容属于是 ['A', 'B', 'C', 'D'] 之一，并表示正确答案。该字段表示打乱选项后，需要如何更新正确答案。与 default_answer_key_switch_method 二选一 + default_answer_key = 'answer' + + # 如果 answer_key 的内容不属于 ['A', 'B', 'C', 'D'] 之一，那么可以使用函数的方式来指定打乱选项后的正确答案。与 default_answer_key 二选一 + # def default_answer_key_switch_method(item, circular_pattern): + # # item 是原本的数据项 + # # circular_pattern 是一个 tuple，表示打乱选项后的顺序，例如 ('D', 'A', 'B', 'C') 表示原来的 A 选项变成了 D，原来的 B 选项变成了 A，以此类推 + # item['answer'] = circular_pattern['ABCD'.index(item['answer'])] + # return item +``` + +`CircularCEvalDataset` 会接受 `circular_pattern` 参数，它有两个取值: + +- `circular`: 表示单项循环。默认为该值。ABCD 会被扩充为 ABCD, BCDA, CDAB, DABC, 共 4 种 +- `all_possible`: 表示全排列。ABCD 会被扩充为 ABCD, ABDC, ACBD, ACDB, ADBC, ADCB, BACD, ..., 共 24 种 + +另外我们提供了一个 `CircularEvaluator` 用于替换 `AccEvaluator`，该 Evaluator 同样接受 `circular_pattern`，该参数应与上述保持一致。它会产出以下指标： + +- `acc_{origin|circular|all_possible}`: 将打乱后选项顺序后的题目视作多道单独的题目，计算准确率 +- `perf_{origin|circular|all_possible}`: 按照 circular 的逻辑，若选项打乱后的题目都回答正确，才会视为这道题正确，计算准确率 +- `more_{num}_{origin|circular|all_possible}`: 按照 circular 的逻辑，若选项打乱后的题目回答正确的数量大于等于 num，就会视为这道题正确，计算准确率 + +OpenCompass 配置文件： + +```python +from mmengine.config import read_base +from opencompass.datasets.circular import CircularCEvalDataset + +with read_base(): + from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets + +for d in ceval_datasets: + # 重载 load 方法 + d['type'] = CircularCEvalDataset + # 为了与非循环评测版本做区分而进行改名 + d['abbr'] = d['abbr'] + '-circular-4' + # 重载评测方法 + d['eval_cfg']['evaluator'] = {'type': CircularEvaluator} + +# 上述操作后的 dataset 形如下： +# dict( +# type=CircularCEvalDataset, +# path='./data/ceval/formal_ceval', # 未改变 +# name='computer_network', # 未改变 +# abbr='ceval-computer_network-circular-4', +# reader_cfg=dict(...), # 未改变 +# infer_cfg=dict(...), # 未改变 +# eval_cfg=dict(evaluator=dict(type=CircularEvaluator), ...), +# ) +``` + +另外评测时为了针对循环评测有更良好的结果呈现，建议考虑使用以下 summarizer + +```python +from mmengine.config import read_base +from opencompass.summarizers import CircularSummarizer + +with read_base(): + from ...summarizers.groups.ceval import ceval_summary_groups + +new_summary_groups = [] +for item in ceval_summary_groups: + new_summary_groups.append( + { + 'name': item['name'] + '-circular-4', + 'subsets': [i + '-circular-4' for i in item['subsets']], + } + ) + +summarizer = dict( + type=CircularSummarizer, + # 选择具体看哪些指标 + metric_types=['acc_origin', 'perf_circular'], + dataset_abbrs = [ + 'ceval-circular-4', + 'ceval-humanities-circular-4', + 'ceval-stem-circular-4', + 'ceval-social-science-circular-4', + 'ceval-other-circular-4', + ], + summary_groups=new_summary_groups, +) +``` + +更多复杂的评测案例可以参考这个样例代码: https://github.com/open-compass/opencompass/tree/main/examples/eval_circular.py diff --git a/docs/zh_cn/advanced_guides/code_eval.md b/docs/zh_cn/advanced_guides/code_eval.md new file mode 100644 index 0000000000000000000000000000000000000000..3dc1f70325a1bd444ca0618bc48e972ac75a3af8 --- /dev/null +++ b/docs/zh_cn/advanced_guides/code_eval.md @@ -0,0 +1,106 @@ +# 代码评测教程 + +这里以 `humaneval` 和 `mbpp` 为例，主要介绍如何评测模型的代码能力。 + +## pass@1 + +如果只需要生成单条回复来评测pass@1的性能，可以直接使用[configs/datasets/humaneval/humaneval_gen_8e312c.py](https://github.com/open-compass/opencompass/blob/main/configs/datasets/humaneval/humaneval_gen_8e312c.py) 和 [configs/datasets/mbpp/deprecated_mbpp_gen_1e1056.py](https://github.com/open-compass/opencompass/blob/main/configs/datasets/mbpp/deprecated_mbpp_gen_1e1056.py) 并参考通用的[快速上手教程](../get_started/quick_start.md)即可。 + +如果要进行多语言评测，可以参考[多语言代码评测教程](./code_eval_service.md)。 + +## pass@k + +如果对于单个example需要生成多条回复来评测pass@k的性能，需要参考以下两种情况。这里以10回复为例子： + +### 通常情况 + +对于绝大多数模型来说，模型支持HF的generation中带有`num_return_sequences` 参数，我们可以直接使用来获取多回复。可以参考以下配置文件。 + +```python +from opencompass.datasets import MBPPDatasetV2, MBPPPassKEvaluator + +with read_base(): + from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets + from .datasets.mbpp.deprecated_mbpp_gen_1e1056 import mbpp_datasets + +mbpp_datasets[0]['type'] = MBPPDatasetV2 +mbpp_datasets[0]['eval_cfg']['evaluator']['type'] = MBPPPassKEvaluator +mbpp_datasets[0]['reader_cfg']['output_column'] = 'test_column' + +datasets = [] +datasets += humaneval_datasets +datasets += mbpp_datasets + +models = [ + dict( + type=HuggingFaceCausalLM, + ..., + generation_kwargs=dict( + num_return_sequences=10, + do_sample=True, + top_p=0.95, + temperature=0.8, + ), + ..., + ) +] +``` + +对于 `mbpp`，在数据集和评测上需要有新的变更，所以同步修改`type`, `eval_cfg.evaluator.type`, `reader_cfg.output_column` 字段来适应新的需求。 + +另外我们需要模型的回复有随机性，同步需要设置`generation_kwargs`参数。这里注意要设置`num_return_sequences`得到回复数。 + +注意：`num_return_sequences` 必须大于等于k，本身pass@k是计算的概率估计。 + +具体可以参考以下配置文件 +[examples/eval_code_passk.py](https://github.com/open-compass/opencompass/blob/main/examples/eval_code_passk.py) + +### 模型不支持多回复 + +适用于一些没有设计好的API以及功能缺失的HF模型。这个时候我们需要重复构造数据集来达到多回复的效果。这里可以参考以下配置文件。 + +```python +from opencompass.datasets import MBPPDatasetV2, MBPPPassKEvaluator + +with read_base(): + from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets + from .datasets.mbpp.deprecated_mbpp_gen_1e1056 import mbpp_datasets + +humaneval_datasets[0]['abbr'] = 'openai_humaneval_pass10' +humaneval_datasets[0]['num_repeats'] = 10 +mbpp_datasets[0]['abbr'] = 'mbpp_pass10' +mbpp_datasets[0]['num_repeats'] = 10 +mbpp_datasets[0]['type'] = MBPPDatasetV2 +mbpp_datasets[0]['eval_cfg']['evaluator']['type'] = MBPPPassKEvaluator +mbpp_datasets[0]['reader_cfg']['output_column'] = 'test_column' + +datasets = [] +datasets += humaneval_datasets +datasets += mbpp_datasets + +models = [ + dict( + type=HuggingFaceCausalLM, + ..., + generation_kwargs=dict( + do_sample=True, + top_p=0.95, + temperature=0.8, + ), + ..., + ) +] +``` + +由于数据集的prompt并没有修改，我们需要替换对应的字段来达到数据集重复的目的。 +需要修改以下字段： + +- `num_repeats`: 数据集重复的次数 +- `abbr`: 数据集的缩写最好随着重复次数一并修改，因为数据集数量会发生变化，防止与`.cache/dataset_size.json` 中的数值出现差异导致一些潜在的问题。 + +对于 `mbpp`，同样修改`type`, `eval_cfg.evaluator.type`, `reader_cfg.output_column` 字段。 + +另外我们需要模型的回复有随机性，同步需要设置`generation_kwargs`参数。 + +具体可以参考以下配置文件 +[examples/eval_code_passk_repeat_dataset.py](https://github.com/open-compass/opencompass/blob/main/examples/eval_code_passk_repeat_dataset.py) diff --git a/docs/zh_cn/advanced_guides/code_eval_service.md b/docs/zh_cn/advanced_guides/code_eval_service.md new file mode 100644 index 0000000000000000000000000000000000000000..6a28a19f81f68d1ec6466a579a258bf4d8fa9914 --- /dev/null +++ b/docs/zh_cn/advanced_guides/code_eval_service.md @@ -0,0 +1,222 @@ +# 代码评测Docker教程 + +为了完成LLM代码能力评测，我们需要搭建一套独立的评测环境，避免在开发环境执行错误代码从而造成不可避免的损失。目前 OpenCompass 使用的代码评测服务可参考[code-evaluator](https://github.com/open-compass/code-evaluator)项目。接下来将围绕代码评测服务介绍不同需要下的评测教程。 + +1. humaneval-x + +多编程语言的数据集 [humaneval-x](https://huggingface.co/datasets/THUDM/humaneval-x) +数据集[下载地址](https://github.com/THUDM/CodeGeeX2/tree/main/benchmark/humanevalx)，请下载需要评测的语言（××.jsonl.gz）文件，并放入`./data/humanevalx`文件夹。 + +目前支持的语言有`python`, `cpp`, `go`, `java`, `js`。 + +2. DS1000 + +Python 多算法库数据集 [ds1000](https://github.com/xlang-ai/DS-1000) +数据集[下载地址](https://github.com/xlang-ai/DS-1000/blob/main/ds1000_data.zip) + +目前支持的算法库有`Pandas`, `Numpy`, `Tensorflow`, `Scipy`, `Sklearn`, `Pytorch`, `Matplotlib`。 + +## 启动代码评测服务 + +1. 确保您已经安装了 docker，可参考[安装docker文档](https://docs.docker.com/engine/install/) +2. 拉取代码评测服务项目，并构建 docker 镜像 + +选择你需要的数据集对应的dockerfile，在下面命令中做替换 `humanevalx` 或者 `ds1000`。 + +```shell +git clone https://github.com/open-compass/code-evaluator.git +docker build -t code-eval-{your-dataset}:latest -f docker/{your-dataset}/Dockerfile . +``` + +3. 使用以下命令创建容器 + +```shell +# 输出日志格式 +docker run -it -p 5000:5000 code-eval-{your-dataset}:latest python server.py + +# 在后台运行程序 +# docker run -itd -p 5000:5000 code-eval-{your-dataset}:latest python server.py + +# 使用不同的端口 +# docker run -itd -p 5001:5001 code-eval-{your-dataset}:latest python server.py --port 5001 +``` + +**注：** + +- 如在评测Go的过程中遇到timeout，请在创建容器时候使用以下命令 + +```shell +docker run -it -p 5000:5000 -e GO111MODULE=on -e GOPROXY=https://goproxy.io code-eval-{your-dataset}:latest python server.py +``` + +4. 为了确保您能够访问服务，通过以下命令检测推理环境和评测服务访问情况。 (如果推理和代码评测在同一主机中运行服务，就跳过这个操作) + +```shell +ping your_service_ip_address +telnet your_service_ip_address your_service_port +``` + +## 本地代码评测 + +模型推理和代码评测服务在同一主机，或者同一局域网中，可以直接进行代码推理及评测。**注意：DS1000暂不支持，请走异地评测** + +### 配置文件 + +我们已经提供了 huamaneval-x 在 codegeex2 上评估的\[配置文件\]作为参考(https://github.com/open-compass/opencompass/blob/main/examples/eval_codegeex2.py)。 +其中数据集以及相关后处理的配置文件为这个[链接](https://github.com/open-compass/opencompass/tree/main/configs/datasets/humanevalx)， 需要注意 humanevalx_eval_cfg_dict 中的evaluator 字段。 + +```python +from opencompass.openicl.icl_prompt_template import PromptTemplate +from opencompass.openicl.icl_retriever import ZeroRetriever +from opencompass.openicl.icl_inferencer import GenInferencer +from opencompass.datasets import HumanevalXDataset, HumanevalXEvaluator + +humanevalx_reader_cfg = dict( + input_columns=['prompt'], output_column='task_id', train_split='test') + +humanevalx_infer_cfg = dict( + prompt_template=dict( + type=PromptTemplate, + template='{prompt}'), + retriever=dict(type=ZeroRetriever), + inferencer=dict(type=GenInferencer, max_out_len=1024)) + +humanevalx_eval_cfg_dict = { + lang : dict( + evaluator=dict( + type=HumanevalXEvaluator, + language=lang, + ip_address="localhost", # replace to your code_eval_server ip_address, port + port=5000), # refer to https://github.com/open-compass/code-evaluator to launch a server + pred_role='BOT') + for lang in ['python', 'cpp', 'go', 'java', 'js'] # do not support rust now +} + +humanevalx_datasets = [ + dict( + type=HumanevalXDataset, + abbr=f'humanevalx-{lang}', + language=lang, + path='./data/humanevalx', + reader_cfg=humanevalx_reader_cfg, + infer_cfg=humanevalx_infer_cfg, + eval_cfg=humanevalx_eval_cfg_dict[lang]) + for lang in ['python', 'cpp', 'go', 'java', 'js'] +] +``` + +### 任务启动 + +参考[快速上手教程](../get_started.html) + +## 异地代码评测 + +模型推理和代码评测服务分别在不可访问的不同机器中，需要先进行模型推理，收集代码推理结果。配置文件和推理流程都可以复用上面的教程。 + +### 收集推理结果（仅针对Humanevalx） + +OpenCompass 在 `tools` 中提供了 `collect_code_preds.py` 脚本对推理结果进行后处理并收集，我们只需要提供启动任务时的配置文件，以及指定复用对应任务的工作目录，其配置与 `run.py` 中的 `-r` 一致，细节可参考[文档](https://opencompass.readthedocs.io/zh-cn/latest/get_started/quick_start.html#id4)。 + +```shell +python tools/collect_code_preds.py [config] [-r latest] +``` + +收集到的结果将会按照以下的目录结构保存到 `-r` 对应的工作目录中： + +``` +workdir/humanevalx +├── codegeex2-6b +│ ├── humanevalx_cpp.json +│ ├── humanevalx_go.json +│ ├── humanevalx_java.json +│ ├── humanevalx_js.json +│ └── humanevalx_python.json +├── CodeLlama-13b +│ ├── ... +├── CodeLlama-13b-Instruct +│ ├── ... +├── CodeLlama-13b-Python +│ ├── ... +├── ... +``` + +对于 DS1000 只需要拿到 `opencompasss` 对应生成的 prediction文件即可。 + +### 代码评测 + +#### 以下仅支持Humanevalx + +确保代码评测服务启动的情况下，使用 `curl` 提交请求： + +```shell +curl -X POST -F 'file=@{result_absolute_path}' -F 'dataset={dataset/language}' {your_service_ip_address}:{your_service_port}/evaluate +``` + +例如： + +```shell +curl -X POST -F 'file=@./examples/humanevalx/python.json' -F 'dataset=humanevalx/python' localhost:5000/evaluate +``` + +得到结果： + +``` +"{\"pass@1\": 37.19512195121951%}" +``` + +另外我们额外提供了 `with-prompt` 选项（默认为True），由于有些模型生成结果包含完整的代码（如WizardCoder），不需要 prompt + prediciton 的形式进行拼接，可以参考以下命令进行评测。 + +```shell +curl -X POST -F 'file=@./examples/humanevalx/python.json' -F 'dataset=humanevalx/python' -H 'with-prompt: False' localhost:5000/evaluate +``` + +#### 以下仅支持DS1000 + +确保代码评测服务启动的情况下，使用 `curl` 提交请求： + +```shell +curl -X POST -F 'file=@./internlm-chat-7b-hf-v11/ds1000_Numpy.json' localhost:5000/evaluate +``` + +DS1000支持额外 debug 参数，注意开启之后会有大量log + +- `full`: 额外打印每个错误样本的原始prediction，后处理后的predcition，运行程序以及最终报错。 +- `half`: 额外打印每个错误样本的运行程序以及最终报错。 +- `error`: 额外打印每个错误样本的最终报错。 + +```shell +curl -X POST -F 'file=@./internlm-chat-7b-hf-v11/ds1000_Numpy.json' -F 'debug=error' localhost:5000/evaluate +``` + +另外还可以通过同样的方式修改`num_workers`来控制并行数。 + +## 进阶教程 + +除了评测已支持的 `humanevalx` 数据集以外，用户还可能有以下需求: + +### 支持新数据集 + +可以参考[支持新数据集教程](./new_dataset.md) + +### 修改后处理 + +1. 本地评测中，可以按照支持新数据集教程中的后处理部分来修改后处理方法； +2. 异地评测中，可以修改 `tools/collect_code_preds.py` 中的后处理部分； +3. 代码评测服务中，存在部分后处理也可以进行修改，详情参考下一部分教程； + +### 代码评测服务 Debug + +在支持新数据集或者修改后处理的过程中，可能会遇到需要修改原本的代码评测服务的情况，按照需求修改以下部分 + +1. 删除 `Dockerfile` 中安装 `code-evaluator` 的部分，在启动容器时将 `code-evaluator` 挂载 + +```shell +docker run -it -p 5000:5000 -v /local/path/of/code-evaluator:/workspace/code-evaluator code-eval:latest bash +``` + +2. 安装并启动代码评测服务，此时可以根据需要修改本地 `code-evaluator` 中的代码来进行调试 + +```shell +cd code-evaluator && pip install -r requirements.txt +python server.py +``` diff --git a/docs/zh_cn/advanced_guides/compassbench_intro.md b/docs/zh_cn/advanced_guides/compassbench_intro.md new file mode 100644 index 0000000000000000000000000000000000000000..be827296e07d67b0c017f75da311e44fdc7acfea --- /dev/null +++ b/docs/zh_cn/advanced_guides/compassbench_intro.md @@ -0,0 +1,194 @@ +# CompassBench 介绍 + +## CompassBench 2.0 v1.3 版本 + +CompassBench（官方自建榜单）经历了多次更新迭代，从2024年7月起，OpenCompass将会公布自建榜单的评测规则(评测配置文件)和示例数据集文件，以帮助社区更好的了解自建榜单的评测逻辑和方法。 + +### 能力维度 + +2024年8月榜单将会包括以下能力维度： + +| 能力 | 任务介绍 | 评测方式 | 示例数据地址 | +| -------- | -------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------ | +| 语言 | 评测模型在信息抽取、信息抽取、内容总结、对话、创作等多种任务上的能力 | 主观评测 | https://github.com/open-compass/CompassBench/tree/main/v1_3_data/language | +| 推理 | 评测模型在逻辑推理、常识推理、表格推理等多种日常推理任务上的能力 | 主观评测 | https://github.com/open-compass/CompassBench/tree/main/v1_3_data/reasoning | +| 知识 | 评测模型在理科、工科、人文社科等多个领域的知识水平 | 客观评测 | https://github.com/open-compass/CompassBench/tree/main/v1_3_data/knowledge | +| 数学 | 评测模型在数值计算、高中及大学难度的数学问题上的能力 | 客观评测 | https://github.com/open-compass/CompassBench/tree/main/v1_3_data/math | +| 代码 | 评测模型在代码生成、代码补全、代码注释、代码重构、代码改写、计算机知识综合问答上的能力 | 客观评测 + 主观评测 | https://github.com/open-compass/CompassBench/tree/main/v1_3_data/code | +| 指令跟随 | 评测模型在基于各类语言、推理、知识等任务中，能否准确遵循复杂指令的能力 | 主观评测 | https://github.com/open-compass/CompassBench/tree/main/v1_3_data/instruct | +| 智能体 | 评估模型在复杂工具调用的能力，以及数据科学、数学等情况下使用代码解释器的能力 | 客观评测 | https://github.com/open-compass/T-Eval https://github.com/open-compass/CIBench | + +### 评测方法 + +- 对于客观评测，将会采用0-shot + CoT的方式评测。 + - OpenCompass在客观题评测的后处理上已进行较多优化，并在评测时在Prompt中对回答格式进行约束，对于因指令跟随问题带来的无法完成答案提取的情况，将视为回答错误 + - 数学、智能体题目类型与给定的示例数据类似，但真实评测数据与开源数据不同 +- 对于主观评测，将会采用基于大模型评价的方式进行评测。 + - 我们对每一道问题均提供评测时的打分指引。 + + - 比较待测模型相对于参考回复的胜率，共设置为五档 + + - `A++`：回答A远胜于回答B。 + - `A+`：回答A略优于回答B。 + - `A=B`：回答A和回答B质量相同。 + - `B+`：回答B略优于回答A。 + - `B++`：回答B远胜于回答A。 +- 主观评测配置文件 + - [示例评测配置](https://github.com/open-compass/opencompass/blob/main/configs/eval_compassbench_v1_3_subjective.py) +- 主观评价提示词 + +``` + +# Instruction + +You are an expert evaluator. Your task is to evaluate the quality of the \ +responses generated by two AI models. +We will provide you with the user query and a pair of AI-generated \ +responses (Response A and Response B). +You should first read the user query and the conversation history \ +carefully for analyzing the task, and then evaluate the quality of the \ +responses based on and rules provided below. + +# Conversation between User and AI + +## User Query +<|begin_of_query|> + +{question} + +<|end_of_query|> + +## Response A +<|begin_of_response_A|> + +{prediction} + +<|end_of_response_A|> + +## Response B +<|begin_of_response_B|> + +{prediction2} + +<|end_of_response_B|> + +# Evaluation + +## Checklist + +<|begin_of_checklist|> + +{checklist} + +<|end_of_checklist|> + +Please use this checklist to guide your evaluation, but do not limit your \ +assessment to the checklist. + +## Rules + +You should compare the above two responses based on your analysis of the \ +user queries and the conversation history. +You should first write down your analysis and the checklist that you used \ +for the evaluation, and then provide your assessment according to the \ +checklist. +There are five choices to give your final assessment: ["A++", "A+", \ +"A=B", "B+", "B++"], which correspond to the following meanings: + +- `A++`: Response A is much better than Response B. +- `A+`: Response A is only slightly better than Response B. +- `A=B`: Response A and B are of the same quality. Please use this \ +choice sparingly. +- `B+`: Response B is only slightly better than Response A. +- `B++`: Response B is much better than Response A. + +## Output Format +First, please output your analysis for each model response, and \ +then summarize your assessment to three aspects: "reason A=B", \ +"reason A>B", and "reason B>A", and finally make your choice for \ +the final assessment. + +Please provide your evaluation results in the following json \ +format by filling in the placeholders in []: + +{ + "analysis of A": "[analysis of Response A]", + "analysis of B": "[analysis of Response B]", + "reason of A=B": "[where Response A and B perform equally well]", + "reason of A>B": "[where Response A is better than Response B]", + "reason of B>A": "[where Response B is better than Response A]", + "choice": "[A++ or A+ or A=B or B+ or B++]", +} + + +# 指令 + +您是一位专业评估专家。您的任务是评估两个AI模型生成回答的质量。 +我们将为您提供用户问题及一对AI生成的回答（回答A和回答B）。 +您应当首先仔细阅读用户问题，然后根据以下提供的规则评估回答的质量。 + +# 用户与AI之间的对话 + +## 用户问题 +<|begin_of_query|> + +{question} + +<|end_of_query|> + +## 回答A +<|begin_of_response_A|> + +{prediction} + +<|end_of_response_A|> + +## 回答B +<|begin_of_response_B|> + +{prediction2} + +<|end_of_response_B|> + +# 评估 + +## 检查清单 + +<|begin_of_checklist|> + +{checklist} + +<|end_of_checklist|> + +请参考此检查清单来评估回答的质量，但不要局限于此检查清单。 + +## 规则 + +您应当基于用户查询，分析比较上述两种回答。 +您应当基于检查清单写下您的分析，然后提供您的评价。 +有五个选项供您做出最终评估：["A++", "A+", "A=B", "B+", "B++"]，它们对应如下含义： + +- `A++`：回答A远胜于回答B。 +- `A+`：回答A略优于回答B。 +- `A=B`：回答A和回答B质量相同。请谨慎使用此选项。 +- `B+`：回答B略优于回答A。 +- `B++`：回答B远胜于回答A。 + +## 输出格式 +首先，请输出您对每个模型回答的分析， +然后总结您的评估到三个方面："A=B的理由"，"A优于B的理由"，和 "B优于A的理由"， +最后做出您对最终评估的选择。 + +请按照以下json格式提供您的评估结果，通过填充[]中的占位符： + +{ + "回答A的分析": "[回答A的分析]", + "回答B的分析": "[回答B的分析]", + "A=B的理由": "[A和B回答差不多的理由]", + "A优于B的理由": "[回答A优于B的理由]", + "B优于A的理由": "[回答B优于A的理由]", + "choice": "[A++ or A+ or A=B or B+ or B++]", +} + + +``` diff --git a/docs/zh_cn/advanced_guides/compassbench_v2_0.md b/docs/zh_cn/advanced_guides/compassbench_v2_0.md new file mode 100644 index 0000000000000000000000000000000000000000..ff8f85fbd3283c5311da278ae776241b0fda9d5b --- /dev/null +++ b/docs/zh_cn/advanced_guides/compassbench_v2_0.md @@ -0,0 +1,48 @@ +# CompassBench 2.0 介绍 + + +## v1.0介绍 +为支持OpenCompass的年度榜单，本文将提供CompassBench的整体介绍。 + +本次评测将在语言、知识、创作、推理、数学、代码、长文本、智能体能力的多项任务上开展评测，现提供任务介绍和题目示例。 + +- 评测方式采样主观与客观相结合的方式，具体根据各个任务不同进行具体设计。 +- 针对推理、数学、代码、智能体等任务，将会采用Few-shot + CoT的评测方式。 +- 对于填空题，通过在Prompt中提供Few-shot和输出格式约束来协助抽取答案。 +- 对于选择题，针对同一问题，通过变换提问方式，减少随机影响。 +- 对于开放式问题的评测，对同一问题进行多次采样，并采用多维度打分的方式进行评价。 + +> OpenCompass在客观题评测的后处理上已进行较多优化，并在评测时在Prompt中对回答格式进行约束，对于因指令跟随问题带来的无法完成答案提取的情况，将视为回答错误。OpenCompass将会在下一期加入指令跟随能力的评测。 + +| 能力 | 任务 | 介绍 | 题目示例 | +| ---- | ---- | ---- | ---- | +| 语言 | 信息抽取 | 信息抽取是指从文本中提取出特定类型的信息。这类任务通常用于处理结构化数据、知识图谱、问答系统等场景。 | ```"question": "野马队在分区轮以 23–16 击败了匹兹堡钢人队，在比赛的最后三分钟拿下 11 分。然后他们在美式足球联合会 (AFC) 锦标赛上以 20–18 击败了第 49 届超级碗卫冕冠军新英格兰爱国者队，在比赛还剩 17 秒 时拦截了新英格兰队的两分转换传球。尽管曼宁在本赛季的拦截上有问题，但他在两场季后赛中未投任何球。\n野马队在 AFC 锦标赛中打败了谁？"``` | +| 语言 | 意图识别 | 意图识别是对用户输入的文本或语音进行分析，判断其意图或需求。这类任务应用于智能客服、语音助手、聊天机器人等场景。 | ```"question": "中国文化的天人合一思想\n中西文化的基本差异之一就是，在人与自然的关系问题上，中国文化比较重视人与自然的和谐统一，而西方文化则强调，人要征服自然、改造自然才能求得自己的生存和发展。中国文化的这种特色，有时通过“天人合一”的命题表述出来。中国古代思想家一般都反对把天与人割裂开来、对立起来，而主张天人协调、天人合一。\n天人合一问题，就其理论实质而言，是关于人与自然的统一问题，或者说是自然界和精神的统一问题。应当承认，中国传统文化中的天人合一思想，内容十分复杂，其中既有正确的观点，也有错误的观点，我们必须实事求是地予以分析。但是，从文化的民族性以及对民族文化的推进作用和深远影响看，我们应当大胆肯定。中国古代思想家关于天人合一的思想，其最基本的涵义，就是充分肯定自然界和精神的统一，关注人类行为与自然界的协调问题。从这个意思上说，天人合一思想的，是非常有价值的。\n恩格斯对自然和精神的统一问题，有过一系列精辟的论述。他说：“我们一天天地学会更加正确地理解自然规律，学会认识我们对于自然界的惯常行程的干涉所引起的比较近或比较远的影响。”他还说：“自然界和精神是统一的。自然界不能是无理性的……而理性是不能和自然界矛盾的。”“思维规律和自然规律，只要它们被正确地认识，必然是互相一致的。”恩格斯的这些论述，深刻地揭示了自然和精神统一问题的丰富内涵。根据恩格斯的这些论述，考察中国古代的天人合一思想，不难看出，这种思想有着深刻的合理性。\n中国古代的天人合一思想，强调人与自然的统一，人的行为与自然的协调，道德理性与自然理性的一致，充分显示了中国古代思想家对于主客体之间、主观能动性和客观规律之间关系的辩证思考。根据这种思想，人不能违背自然规律，不能超越自然界的承受力去改造自然、征服自然、破坏自然，而只能在顺从自然规律的条件下去利用自然、调整自然，使之更符合人类的需要，也使自然界的万物都能生长发展。另一方面，自然界也不是主宰人其社会的神秘力量，而是可以认识、可以为我所用的客观对象。这种思想长期实践的结果，是达到自然界与人的统一，人的精神、行为与外在自然的统一，自我身心平衡与自然环境平衡的统一，以及由于这些统一而达到的天道与人道的统一，从而实现完满和谐的精神追求。中国文化的天人合一思想，对于解决当今世界由于工业化和无限制地征服自然而带来的自然环境被污染、生态平衡遭破坏等问题，具有重要的启迪意义；对于我们今天正在进行的社会主义现代化建设，更有着防患于未然的重大现实意义。\n（选自张岱年等主编的《中国文化概论》，有删改）\n根据原文提供的信息，下列推断不正确的一项是","A": "对人与自然关系的认识，中国古代天人合一思想有优于西方文化的地方。","B": "现代人重视和研究天人合一思想，是基于对现实及发展问题的思考。", "C": "肯定天人合一思想的合理性，并不意味着对其思想内容的全盘接受。", "D": "以天人合一思想为指导，可解决当今世界因工业化带来的各种社会问题。",``` | +| 语言 | 情感分析 | 情感分析是对文本中的情感或情绪进行识别和分析的任务。这类任务可用于情感倾向分析场景。例如，分析社交媒体上的用户评论，了解新闻或事件的倾向。| ```"question": "请问以下评价是正面评价还是负面评价？\n大众点评网的霸王餐，200份华辉拉肠双人试吃，员村一店是已经有经营两年以上的，年前装修过，干净齐整，下单的服务员亲切有礼，可能我是第一个用代码验证的，中间拖了点时间去验证，幸好周日10点左右没有平时的多人。拉肠一如既往的滑，皮蛋瘦肉粥很绵，皮蛋瘦肉超多，肉肠是一底带肉一底斋肠，以前没吃过鸡蛋肠觉得6蚊不太划算，现在发现是有三底肠粉的哦，不太喜欢吃肉的可以试下，很饱肚，鼓油是吃过这么多家肠粉店味道调得最好的。","A": "正面评价", "B": "负面评价"```| +| 语言 | 内容总结 | 内容总结是将一篇较长的文本压缩成一篇简短的概括性摘要。这类任务适用于需要快速了解文档核心内容的情境，如新闻标题、电子邮件摘要 | ```联合国减灾办公室负责人格拉瑟。联合国减灾办公室2016年2月11日联合国减灾办公室今天表示,2015年是有记录以来最热的一个年份,在这一年当中,自然灾害影响了近1亿人口。减灾办公室呼吁各国采取行动,应对气候变化,在最大程度上做出努力,防止和减少灾害的发生。联合国减灾办公室所公布的最新数据显示,在过去一年当中,受到灾害影响最重的国家都在亚洲,它们是中国、印度、菲律宾和印度尼西亚。自然灾害共导致2万2000人死亡,带来的经济损失约合660亿美元。然而,尽管这一数字惊人,但却低于1400亿的10年平均数字。其中的部分原因是各国政府采取了更好的防范措施。数据显示,2015年有5000万人深受旱灾之苦,增幅达40%。联合国减灾办公室负责人格拉瑟表示,2015年是记载中最热的一个年份,成因是气候变化和厄尔尼诺天气现象。他指出,最令人感到不安的一个趋势是2015年有记录的主要干旱增加了一倍。他强调,数据表明,减少温室气体排放和适应气候变化对于减少灾害风险至关重要。```| +| 语言 | 内容评价 | 内容评价是对文本的质量、价值或观点进行判断和评价的任务。这类任务可用于评论筛选、观点挖掘等场景。 | ```"question": "以下是一个问题以及针对该问题的两个答案，哪个答案更好？\n问题：创建一篇1000字的非剽窃新闻文章，关于任天堂将于2月8日星期三播出新的任天堂直面会，承诺将公布即将推出的Switch游戏的新细节。2月的任天堂直面会将在东部时间下午5点/太平洋时间下午2点，在公司的YouTube频道上直播。\n\n任天堂表示，星期三的任天堂直面会将持续“大约”40分钟，并将重点放在即将在2023年上半年推出的Nintendo Switch游戏上。\n\n任天堂宣布的Nintendo Switch游戏阵容包括《星之卡比：梦之泉豪华版》，这是2011年Wii游戏的重制版；《魔兵雅各：樱花与失落的恶魔》，这是一个以童话为灵感的冒险故事，主角是年轻的魔兵雅各；《塞尔达传说：王国之泪》，是《荒野之息》的续作；以及《皮克敏4》，这是一款长期运行的策略/动作系列的新作，仍然有些神秘。\n\n此外，任天堂的计划中，但日期尚未确定的还有《超级战争大战1+2：重启营》，这是两款Game Boy Advance时代的《超级战争》游戏的重制版。经过长时间的延迟，Switch版的《超级战争》合集据传很快就要推出。当然，任天堂在2023年还有其他一些大计划，包括2月17日在好莱坞环球影城开设的超级任天堂乐园主题公园，以及4月7日上映的《超级马里奥兄弟》电影。格斗游戏确实很酷，但可能会让人望而生畏。你可能会看到铁拳或街霸这样的竞技世界，然后想，“我可能很菜！而且没有正确的装备！而且我不懂这些术语！”但是停下来。你可以进入格斗游戏。对Pat Gill来说，这段旅程真的很有意义。以下是他如何拥抱格斗游戏，以及你也可以。\n答案 1：任天堂粉丝们正急切地期待着最新的任天堂直面发布会，该发布会定于2月8日星期三播出。该发布会将在公司的YouTube频道上以东部时间下午5点/太平洋时间下午2点的直播形式进行，承诺提供有关任天堂Switch即将推出的游戏的令人兴奋的新细节。\n\n据任天堂称，直面发布会将持续大约40分钟，专门关注2023年上半年将为Switch发布的游戏。已经宣布的一些游戏标题包括《星之卡比：梦之泉DX》、《魔兽传说：樱花与失落的恶魔》、《塞尔达传说：王国的泪滴》和《皮克敏4》。\n\n任天堂日程表上最受期待的游戏之一是《超级战争大战1+2：重启训练营》，这是两款经典Game Boy Advance游戏的重制版。尽管尚未正式宣布发行日期，但有传言称游戏将在经过长时间延迟后很快推出。\n\n除了游戏阵容外，任天堂还计划在2023年推出许多其他令人兴奋的项目，包括2月17日在好莱坞环球影城开放的超级任天堂乐园主题公园，以及4月7日上映的《超级马里奥兄弟》电影。\n\n对于格斗游戏爱好者来说，任天堂直面发布会承诺将是一个令人兴奋的活动，将提供任天堂Switch最新和最伟大游戏的一瞥。无论您是经验丰富的玩家还是刚刚开始，直面发布会中肯定会有适合每个人的内容。因此，请务必在2月8日星期三收看，以获取来自任天堂的所有最新消息和更新！\n答案 2：任天堂宣布将于2月8日星期三下午5点EST/下午2点PST，在公司的YouTube频道上播出新的任天堂Direct演示。演示将持续大约40分钟，并将提供有关2023年上半年即将推出的任天堂Switch游戏的新细节。\n\n公司宣布的任天堂Switch阵容包括《星之卡比：梦之返乡豪华版》，这是2011年Wii游戏的重制版；《魔兽师起源：赛瑞莎与失落的恶魔》，一个以童话为灵感的冒险，主角是年轻的贝约奈塔；《塞尔达传说：王国之泪》，《荒野之息》的续作；以及《皮克敏4》，这是一款长寿的策略/动作系列的新作，仍然有些神秘。\n\n此外，任天堂还有一些其他大计划，包括在2023年2月17日在好莱坞环球影城开设超级任天堂乐园主题公园，以及于4月7日上映《超级马里奥兄弟电影》。\n\n格斗游戏是一种受欢迎的游戏类型，可能是一种令人望而生畏的爱好。然而，人们是可以享受格斗游戏的，Pat Gill就是如何拥抱这种爱好的一个很好的例子。他从一个初学者开始，发现这是一段有意义的旅程。只要有正确的心态和资源，任何人都可以参与格斗游戏，并享受它们所提供的刺激和竞争。" ``` | +| 语言 | 多语言翻译 | 多语言翻译是将一种语言的文本转换为另一种语言的文本。这类任务适用于跨语言沟通、在线翻译等场景。|```"question": "Translate the following sentence from English to French: \"He [Wales] basically lied to us from the start. First, by acting as if this was for legal reasons. Second, by pretending he was listening to us, right up to his art deletion."```| +| 语言 | 中华传统文化理解 | 中华传统文化涉及对中国古代文学、艺术、哲学、历史等领域的研究 | ``` "question": "王实甫在《西厢记》中写道：“淋漓襟袖啼红泪，比司马青衫更湿”，其中“司马青衫”指的是什么"``` | +| 语言 | 中文语意理解 | 中文语意理解涉及理解文本中的词汇、短语和句子之间的语义关系，包括但不限于近义词、反义词、整体-部分关系、修饰关系等。 |``` "question": "“繁荣”与以下哪个词具有近义关系？", "A": "盛世", "B": "荣誉", "C": "繁花", "D": "昌盛"```| +| 语言 | 多轮对话 | 评价模型能否在多轮对话中保持上下文一致性和连贯性的能力，评估模型是否能够理解并记住对话的上下文信息，记住之前的对话内容。 |```[{'role': 'user','content': '我在做一项关于智能手机市场的研究，需要整理一些数据成 Markdown 表格。数据包括品牌名称、市场份额和热销型号。品牌有苹果、三星和华为。苹果的市场份额是30%，热销型号是iPhone 13；三星市场份额是25%，热销型号是Galaxy S21；华为市场份额是20%，热销型号是Mate 40。请帮我做一个表格。'},{'role': 'user','content': '看起来不错，不过我希望表格中的市场份额列展示为百分比和实际销量。苹果的销量是8000万部，三星是6000万部，华为是5000万部。'}, {'role': 'user', 'content': '很好。现在请把表格的标题中文改成英文，并且各列改成对齐方式：品牌列左对齐，市场份额列居中对齐，热销型号列右对齐。'},{'role': 'user', 'content': '可以，我注意到我们可能需要添加一列来表示这些品牌的总收入，苹果为500亿美元，三星为400亿美元，华为为350亿美元。此外，请按市场销量对行进行排序。'}]```| +| 知识 | 生活常识 | 考察普通社会上智力正常的人皆有或普遍拥有的，大众化的知识 | ```"question": "世界四大文明古国有哪些？```| +| 知识 | 自然科学（理科） | 关于自然现象的具体科学，研究自然界的本质和规律（理科）：包括不限于数学，物理学，化学，生物学，天文学等 | ```"question": "群的研究对象是什么？"``` | +| 知识 | 自然科学（工科） | 关于自然现象的具体科学，研究自然界的本质和规律（工科）：包括不限于计算机科学，医学，建筑学，材料学，机械学，测量学，气象学，环境学等 | ```"question": "下列关于信息安全的说法，正确的是（ ）。", "options": ["打开朋友转发的网页链接一定是安全的", "安装了杀毒软件后电脑就不会感染病毒", "数据加密是一种提高信息安全性的有效措施", "手机指纹识别技术能确保手机所有信息的安全"]``` | +| 知识 | 社会科学 | 研究社会现象的具体科学，力求揭示社会的本质和规律，例如经济学，政治学，军事学，社会学，管理学，教育学等。社会科学主要以人类社会的组织与结构、体制与关系、功能与效率、秩序与规范为研究认识之对象，并通过这种知识来为人类社会的有序管理、高效运作提供知识、理论和手段 | ```"question": "为了避免资金供应短缺和倒闭，企业经营者需要做什么？"``` | +| 知识 | 人文科学 | 设设计对人的问题的类型思考与情感体验，围绕着关乎人的心灵世界、关乎人的精神生命主题而展开的种种思想、观念、知识和理论的探索。它以人类自身，特别是人的内心情感世界为研究中心，以人自身的发展和完善作为学术探索的出发点和归宿。包括不限于文学，历史学、哲学、艺术、语言等 | ```"question": "光绪二十四年(1898)五月，维新派代表人物康有为从“中体西用”的角度论述了科举制度改革的必要性。这表明他( )", "options": ["在戊戌变法初期思想趋于保守", "认同洋务派的“中体西用”思想", "在教育改革方面与洋务派观点一致", "所说的“体”和“用”与洋务派不同"]``` | +| 创作 | 内容扩写 | 给定标题或者大纲的基础上，通过增加细节、描述和解释，使内容更加丰富、饱满和具有表现力。这种方法主要用于散文、小说等文学创作，以及学术论文、报告等实用文本 | ```请根据我给出的[外星人入侵、核弹、流亡]这些关键词来撰写一篇[科幻]题材的短篇故事。 \n故事需要拥有[引人入胜]的开头以及[反转]的结局，故事线[跌宕起伏]。\n注意请使用[刘慈欣]的写作风格为我撰写这篇故事。减少赘述，内容中不要有重复或意思相近的段落，大约800字``` | +| 创作 | 内容续写 | 现有文本的基础上，继续编写后面的内容。这种方法主要用于小说、故事等叙事性文本。续写部分通常要保持与原有文本的风格、情节和人物设定相一致，同时要求作者具备较强的想象力和创造力。 | ```题目：《新型能源技术在工业生产中的应用与效益》随着能源需求的不断增长和传统能源的有限性，新型能源技术在工业领域的应用备受瞩目。本文将着重探讨新型能源技术对工业生产的潜在影响，以及其在提高生产效益和减少环境影响方面的作用。请按照以上题目和摘要，完成一篇不少于1000字的论文``` | +| 创作 | 内容改写 | 不改变原文主题和基本结构的前提下，对文本进行一定程度的修改、重组和优化。这种方法主要用于修改学术论文、报告、文章等。内容改写的目的是提高文本的表达能力、逻辑性和可读性，同时避免重复。 | ```请帮我总结一封电子邮件的内容，总结需要包含以下四个部分：\n【重要性】根据内容判断事项是否重要，结果包含重要、不重要\n【紧急性】根据内容判断事项是否紧急，结果包含紧急、不紧急\n【核心内容】使用一句简短的话总结邮件最核心的内容。\n【需要回复内容】请判断邮件中哪些内容需要获得我的回复/确认，以列表形式呈现。\n 接下来，请根据下面邮件的内容，进行摘要：\n亲爱的全体员工:\n为了改善大家的身心健康,增强工作效率,公司特别安排了一场瑜伽兴趣培训,现将培训内容通知如下:\n日期及时间:8月15日(周六)上午9:00至11:00\n地点:公司三楼活动室(面积120平米,可容纳30人参加培训)\n培训内容:\n专业瑜伽教练将为大家进行基础的瑜伽技能和健康知识培训。 瑜伽是一种低强度有氧运动,适合各年龄层人群。它能够通过姿势练习、呼吸技巧等,改善身体的柔韧性和平衡感,帮助人体各系统更好地运行,有效减压提神。\n本次培训重点讲解:\n1)基本的瑜伽哲学及其健康效果介绍\n2)冥想和呼吸技巧演练\n3)10多个常见的基础瑜伽姿势示范及练习(包括猿人式、波浪式、斜 Supported Headstand 等)\n4)瑜伽练习时需要注意的安全事项\n5)瑜伽适宜穿戴的服装和个人物品\n6)参与培训后如何延续瑜伽运动\n培训具体流程:\n9:00-9:30 瑜伽基本概念介绍\n9:30-10:10 练习冥想、呼吸及基础姿势\n10:10-10:30 小休10分钟\n10:30-11:00 继续练习高难度姿势并解答问题\n如有意参加本次瑜伽兴趣培训,请于8月10日前用邮件或电话方式告知我们,我方将安排培训。\n若您有任何问题或建议,也欢迎与我联系。感谢您的收听与参与。```| +| 推理 | 逻辑推理 | 综合考察模型的几种常见逻辑推理模式：如演绎、归纳和溯因。 | ```"question": "在接下来的文本中，符号 -> 代表着一个简单的数学运算。\n695 - 472 -> 229\n222 - 62 -> 166\n689 - 439 -> ?",```| +| 推理 | 常识推理 | 常识推理是指基于日常生活中积累的知识和经验，对事物进行合理推断和判断的过程。它涉及到对常见事物、现象和规律的理解，通过综合分析得出合理的结论。 | ```"question": "美即好效应，指对一个外表英俊漂亮的人，人们很容易误认为他或她的其他方面也很不错。根据上述定义，下列哪项属于美即好效应？（ ）", "A": "外表英俊漂亮的人在应聘中更受招聘者的青睐", "B": "小芳认为自己的女儿是幼儿园中最漂亮的孩子", "C": "人们常说女孩因为可爱而美丽并非因为美丽而可爱", "D": "购物网站上有一个漂亮的模特往往会提高产品的销量"``` | +| 数学 | 初等数学 | 初等教育数学能力（小学数学） | ```"question": "小芳手上有40元。她的爸爸又给了她100元。她花了30元买了一条牛仔裤，又花了20元买了一个包。那么小芳还剩下多少钱呢？"```| +| 数学 | 中等数学 | 中等教育数学能力（初中和高中数学） | ```"question": "某地开展建设绿色家园活动,活动期间,计划每天种植相同数量的树木.该活动开始后,实际每天比原计划每天多植树$50$棵,实际植树$400$棵所需时间与原计划植树$300$棵所需时间相同.设实际每天植树$x$棵,则下列方程正确的是（ ）", "options": ["$\\frac{{400}}{{x-50}}=\\frac{{300}}{x}$", "$\\frac{{300}}{{x-50}}=\\frac{{400}}{x}$", "$\\frac{{400}}{{x+50}}=\\frac{{300}}{x}$", "$\\frac{{300}}{{x+50}}=\\frac{{400}}{x}$"]```| +| 数学 | 高等 | 高教育数学能力（大学和研究生数学） | ```"question": "已知有向曲线 $L$ 为球面 $x^2+y^2+z^2=2x$ 与平面 $2x-z-1=0$ 的交线，从 $z$ 轴正向往 $z$ 轴负向看去为逆时针方向，计算曲线积分$\\int_L(6xyz-yz^2)dx+2x^2zdy+xyzdz$.", "options": [ "$\\frac{4\\pi}{7\\sqrt5}$", "$\\frac{3\\pi}{7\\sqrt5}$", "$\\frac{3\\pi}{5\\sqrt5}$", "$\\frac{4\\pi}{5\\sqrt5}$"]``` | +| 代码 | 代码理解 | 输入为用户的需求文字或者部分代码，考察模型的逻辑推理能力和代码生成能力，考察模型对各类编程语言的掌握程度。内容包括不限于：算法和数据结构能力考察编程语言语法考察跨编程语言转换 | ```"question": "编写一个 Python 函数，用于检查两个数字是否仅在一个位置上不同。"```| +| 代码 | 代码分析 | 考察模型对代码的理解和分析能力，给定一段代码，进行代码意图分析，代码规范检查，错误检查等 | ```"question":"\n\ndef truncate_number(number: float) -> float:\n \"\"\" 给定一个正的浮点数，可以将其分解为整数部分（小于给定数字的最大整数）和小数部分（余数部分总是小于1）。\n\n 返回该数字的小数部分。\n >>> truncate_number(3.5)\n 0.5\n \"\"\"",``` | +| 长文本 | 长文本理解与推理 | 考察模型在不同的长度上下文（2k, 4k, 8k, 16k, 32k）情况下的理解和推理能力 | 略 | +| 智能体 | 任务规划 | 智能体根据用户的需求目标和具备工具条件，进行合理的任务拆解，科学地安排子任务的执行顺序和策略，对任务执行路径进行设计和规划，选择合适的策略。 | 略| +| 智能体 | 工具调用 | 评估模型能否准确的调用合适的API，在调用API时能否正确的传递参数 | 略 | +| 智能体 | 反思能力 | 评估模型在子任务执行失败时，是否具有反思和重新规划任务路径的能力 | 略 | +| 智能体 | 任务执行总结 | 评估模型能否根据子任务的执行结果进行总结分析，完成原始任务目标，正确地按指令输出回复 | 略| +| 智能体 | 多轮交互 | 评估模型在进行多轮复杂工具调用时的能力，在多轮情况下能否准确理解意图 | 略 | diff --git a/docs/zh_cn/advanced_guides/contamination_eval.md b/docs/zh_cn/advanced_guides/contamination_eval.md new file mode 100644 index 0000000000000000000000000000000000000000..81001b1bf4fa41f2835dc35466ed5aa0c9bf0fd5 --- /dev/null +++ b/docs/zh_cn/advanced_guides/contamination_eval.md @@ -0,0 +1,122 @@ +# 数据污染评估 + +**数据污染** 是指本应用在下游测试任务重的数据出现在了大语言模型 (LLM) 的训练数据中，从而导致在下游任务 (例如，摘要、自然语言推理、文本分类) 上指标虚高，无法反映模型真实泛化能力的现象。 + +由于数据污染的源头是出现在 LLM 所用的训练数据中，因此最直接的检测数据污染的方法就是将测试数据与训练数据进行碰撞，然后汇报两者之间有多少语料是重叠出现的，经典的 GPT-3 [论文](https://arxiv.org/pdf/2005.14165.pdf)中的表 C.1 会报告了相关内容。 + +但如今开源社区往往只会公开模型参数而非训练数据集，在此种情况下 如何判断是否存在数据污染问题或污染程度如何，这些问题还没有被广泛接受的解决方案。OpenCompass 提供了两种可能的解决方案。 + +## 基于自建同分布数据的污染数据标注 + +我们参考了 [Skywork](https://arxiv.org/pdf/2310.19341.pdf) 中 5.2 节提到的方法，直接使用了 Skywork 上传到 HuggingFace 上的数据集 [mock_gsm8k_test](https://huggingface.co/datasets/Skywork/mock_gsm8k_test)。 + +在该方法中，作者使用 GPT-4 合成了一批与原始 GSM8K 风格类似的数据，然后使用模型分别计算在 GSM8K 训练集 (train)，GSM8K 测试集 (test)，GSM8K 参考集 (ref) 上的困惑度。由于 GSM8K 参考集是最新生成的，作者认为它必然不属于任何模型的任何训练集中，即它是干净的。作者认为： + +- 若 测试集 的困惑度远小于 参考集 的困惑度，那么 测试集 可能出现在了模型的训练阶段； +- 若 训练集 的困惑度远小于 测试集 的困惑度，那么 训练集 可能被模型过拟合了。 + +我们可以参考使用以下配置文件: + +```python +from mmengine.config import read_base + +with read_base(): + from .datasets.gsm8k_contamination.gsm8k_contamination_ppl_ecdd22 import gsm8k_datasets # 包含训练、测试、参考集 + from .models.qwen.hf_qwen_7b import models as hf_qwen_7b_model # 待审查的模型 + from .models.yi.hf_yi_6b import models as hf_yi_6b_model + +datasets = [*gsm8k_datasets] +models = [*hf_qwen_7b_model, *hf_yi_6b_model] +``` + +其样例输出如下： + +```text +dataset version metric mode internlm-7b-hf qwen-7b-hf yi-6b-hf chatglm3-6b-base-hf qwen-14b-hf baichuan2-13b-base-hf internlm-20b-hf aquila2-34b-hf ... +--------------- --------- ----------- ------- ---------------- ------------ ---------- --------------------- ------------- ----------------------- ----------------- ---------------- ... +gsm8k-train-ppl 0b8e46 average_ppl unknown 1.5 0.78 1.37 1.16 0.5 0.76 1.41 0.78 ... +gsm8k-test-ppl 0b8e46 average_ppl unknown 1.56 1.33 1.42 1.3 1.15 1.13 1.52 1.16 ... +gsm8k-ref-ppl f729ba average_ppl unknown 1.55 1.2 1.43 1.35 1.27 1.19 1.47 1.35 ... +``` + +目前该方案仅支持 GSM8K 数据集，我们欢迎社区贡献更多的数据集。 + +如果使用了该方法，请添加引用: + +```bibtex +@misc{2023opencompass, + title={OpenCompass: A Universal Evaluation Platform for Foundation Models}, + author={OpenCompass Contributors}, + howpublished = {\url{https://github.com/open-compass/opencompass}}, + year={2023} +} +@misc{wei2023skywork, + title={Skywork: A More Open Bilingual Foundation Model}, + author={Tianwen Wei and Liang Zhao and Lichang Zhang and Bo Zhu and Lijie Wang and Haihua Yang and Biye Li and Cheng Cheng and Weiwei Lü and Rui Hu and Chenxia Li and Liu Yang and Xilin Luo and Xuejie Wu and Lunan Liu and Wenjun Cheng and Peng Cheng and Jianhao Zhang and Xiaoyu Zhang and Lei Lin and Xiaokun Wang and Yutuan Ma and Chuanhai Dong and Yanqi Sun and Yifu Chen and Yongyi Peng and Xiaojuan Liang and Shuicheng Yan and Han Fang and Yahui Zhou}, + year={2023}, + eprint={2310.19341}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## 基于经典预训练集的污染数据标注 + +感谢 [Contamination_Detector](https://github.com/liyucheng09/Contamination_Detector) 以及 @liyucheng09 提供了本方法。 + +在该方法中，作者将测试数据集 (例如 C-Eval, ARC, HellaSwag 等) 使用 Common Crawl 数据库和 Bing 搜索引擎来进行检索，然后依次标记每条测试样本是 干净的 / 题目被污染的 / 题目和答案均被污染的。 + +测试时，OpenCompass 会分别汇报 ceval 在三种标签所组成的子集上的准确率或困惑度。一般来说，准确率从低到高依次是 干净的，题目被污染的，题目和答案均被污染的 子集。作者认为： + +- 若三者性能较为接近，则模型在该测试集上的污染程度较轻；反之则污染程度较重。 + +我们可以参考使用以下配置文件 [link](https://github.com/open-compass/opencompass/blob/main/examples/eval_contamination.py)： + +```python +from mmengine.config import read_base + +with read_base(): + from .datasets.ceval.ceval_clean_ppl import ceval_datasets # 有污染标记的 ceval 数据集 + from .models.yi.hf_yi_6b import models as hf_yi_6b_model # 待审查的模型 + from .models.qwen.hf_qwen_7b import models as hf_qwen_7b_model + from .summarizers.contamination import ceval_summarizer as summarizer # 输出格式整理 + +datasets = [*ceval_datasets] +models = [*hf_yi_6b_model, *hf_qwen_7b_model] +``` + +其样例输出如下： + +```text +dataset version mode yi-6b-hf - - qwen-7b-hf - - ... +---------------------------------------------- --------- ------ ---------------- ----------------------------- --------------------------------------- ---------------- ----------------------------- --------------------------------------- ... +- - - accuracy - clean accuracy - input contaminated accuracy - input-and-label contaminated accuracy - clean accuracy - input contaminated accuracy - input-and-label contaminated ... +... +ceval-humanities - ppl 74.42 75.00 82.14 67.44 50.00 70.54 ... +ceval-stem - ppl 53.70 57.14 85.61 47.41 52.38 67.63 ... +ceval-social-science - ppl 81.60 84.62 83.09 76.00 61.54 72.79 ... +ceval-other - ppl 72.31 73.91 75.00 58.46 39.13 61.88 ... +ceval-hard - ppl 44.35 37.50 70.00 41.13 25.00 30.00 ... +ceval - ppl 67.32 71.01 81.17 58.97 49.28 67.82 ... +``` + +目前该方案仅支持 C-Eval, MMLU, HellaSwag 和 ARC 数据集，[Contamination_Detector](https://github.com/liyucheng09/Contamination_Detector) 中还包含了 CSQA 和 WinoGrande，但目前还没有在 OpenCompass 中实现。我们欢迎社区贡献更多的数据集。 + +如果使用了该方法，请添加引用: + +```bibtex +@misc{2023opencompass, + title={OpenCompass: A Universal Evaluation Platform for Foundation Models}, + author={OpenCompass Contributors}, + howpublished = {\url{https://github.com/open-compass/opencompass}}, + year={2023} +} +@article{Li2023AnOS, + title={An Open Source Data Contamination Report for Llama Series Models}, + author={Yucheng Li}, + journal={ArXiv}, + year={2023}, + volume={abs/2310.17589}, + url={https://api.semanticscholar.org/CorpusID:264490711} +} +``` diff --git a/docs/zh_cn/advanced_guides/custom_dataset.md b/docs/zh_cn/advanced_guides/custom_dataset.md new file mode 100644 index 0000000000000000000000000000000000000000..c55784d5e6e3e7d5114df1115a447b241ebf9739 --- /dev/null +++ b/docs/zh_cn/advanced_guides/custom_dataset.md @@ -0,0 +1,261 @@ +# 快速评测数据集 + +OpenCompass提供了两种快速对提供的数据进行评测的路径，即基于ChatMLDataset的数据格式协议和基于CustomDataset的数据格式协议。 +相较于 [new_dataset.md](./new_dataset.md) 中的完整数据集集成流程，这两种快速评测路径更加方便快捷，能够在免于增加新配置文件的前提下直接进入评测任务阶段。但如果您存在定制化读取 / 推理 / 评测需求的，建议仍按照完整的集成流程加入新数据集。 + +## 基于ChatMLDataset的数据格式协议和快速评测 + +OpenCompass最新推出的基于ChatML对话模板的数据集评测模式，允许用户提供一个符合ChatML对话模板的数据集.jsonl文件，并像配置模型一样对数据集信息进行简单配置后即可直接开始评测任务。 + +### 数据集文件的格式要求 + +本评测方式仅支持`.jsonl`格式的数据集文件，且其中的每条数据均需遵守以下格式： + +较简易结构的文本数据集： + +```jsonl +{ + "question":[ + { + "role": "system" # 可省略 + "content": Str + }, + { + "role": "user", + "content": Str + } + ], + "answer":[ + Str + ] +} +{ + ... +} +... +``` + +多轮多模等复杂情况的数据集：（由于OpenCompass暂未支持多模评测，因此此处模板仅供参考） + +```jsonl +{ + "question":[ + { + "role": "system", + "content": Str, + }, + { + "role": "user", + "content": Str or List + [ + { + "type": Str, # "image" + "image_url": Str, + }, + ... + { + "type": Str, # "text" + "text": Str, + }, + ] + }, + { + "role": "assistant", + "content": Str + }, + { + "role": "user", + "content": Str or List + }, + ... + ], + "answer":[ + Str, + Str, + ... + ] +} +{ + ... +} +... +``` + +`ChatMLDataset`在读取.jsonl文件时，会使用`pydantic`库对文件进行简易的格式校验。 +您可以使用`tools/chatml_format_test.py`对提供的数据文件进行检查。 + +完成数据检查后，需要在运行配置文件中加入字段名为`chatml_datasets`的配置字典，以在运行时将数据文件转化为OpenCompass的数据集。示例如下： + +```python +chatml_datasets = [ + dict( + abbr='YOUR_DATASET_NAME', + path='YOUR_DATASET_PATH', + evaluator=dict( + type='cascade_evaluator', + rule_evaluator=dict( + type='math_evaluator', + ), + llm_evaluator=dict( + type='llm_evaluator', + prompt="YOUR_JUDGE_PROMPT", + judge_cfg=dict(), # YOUR Judge Model Config + ) + ), + n=1, # Repeat Number + ), +] +``` + +目前，ChatML模块内提供了四种预设的Evaluator，分别是`mcq_rule_evaluator`（用于选择题评估）、`math_evaluator`（用于latex数学公式评估）、`llm_evaluator`（用于评估难以提取答案的题目或开放式题目）、`cascade_evaluator`（规则式和LLM评估器级联组成的评估模式）。 + +此外，如果您有基于ChatML模板长期使用数据集的需求，可以将配置添加到`opencompass/configs/chatml_datasets`中。 +在`examples/eval_chat_datasets.py`中也给出了调用这类数据集配置的评测示例。 + +## 基于CustomDataset的数据格式协议和快速评测 + +(此模块已不再进行更新，但若存在命令行快速运行评测等需求，仍可以使用此模块。) + +基于CustomDataset的数据格式协议支持的任务类型包括选择 (`mcq`) 和问答 (`qa`) 两种，其中 `mcq` 支持 `ppl` 推理和 `gen` 推理；`qa` 支持 `gen` 推理。 + +### 数据集格式 + +我们支持 `.jsonl` 和 `.csv` 两种格式的数据集。 + +#### 选择题 (`mcq`) + +对于选择 (`mcq`) 类型的数据，默认的字段如下： + +- `question`: 表示选择题的题干 +- `A`, `B`, `C`, ...: 使用单个大写字母表示选项，个数不限定。默认只会从 `A` 开始，解析连续的字母作为选项。 +- `answer`: 表示选择题的正确答案，其值必须是上述所选用的选项之一，如 `A`, `B`, `C` 等。 + +对于非默认字段，我们都会进行读入，但默认不会使用。如需使用，则需要在 `.meta.json` 文件中进行指定。 + +`.jsonl` 格式样例如下： + +```jsonl +{"question": "165+833+650+615=", "A": "2258", "B": "2263", "C": "2281", "answer": "B"} +{"question": "368+959+918+653+978=", "A": "3876", "B": "3878", "C": "3880", "answer": "A"} +{"question": "776+208+589+882+571+996+515+726=", "A": "5213", "B": "5263", "C": "5383", "answer": "B"} +{"question": "803+862+815+100+409+758+262+169=", "A": "4098", "B": "4128", "C": "4178", "answer": "C"} +``` + +`.csv` 格式样例如下: + +```csv +question,A,B,C,answer +127+545+588+620+556+199=,2632,2635,2645,B +735+603+102+335+605=,2376,2380,2410,B +506+346+920+451+910+142+659+850=,4766,4774,4784,C +504+811+870+445=,2615,2630,2750,B +``` + +#### 问答题 (`qa`) + +对于问答 (`qa`) 类型的数据，默认的字段如下： + +- `question`: 表示问答题的题干 +- `answer`: 表示问答题的正确答案。可缺失，表示该数据集无正确答案。 + +对于非默认字段，我们都会进行读入，但默认不会使用。如需使用，则需要在 `.meta.json` 文件中进行指定。 + +`.jsonl` 格式样例如下： + +```jsonl +{"question": "752+361+181+933+235+986=", "answer": "3448"} +{"question": "712+165+223+711=", "answer": "1811"} +{"question": "921+975+888+539=", "answer": "3323"} +{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"} +``` + +`.csv` 格式样例如下： + +```csv +question,answer +123+147+874+850+915+163+291+604=,3967 +149+646+241+898+822+386=,3142 +332+424+582+962+735+798+653+214=,4700 +649+215+412+495+220+738+989+452=,4170 +``` + +### 命令行列表 + +自定义数据集可直接通过命令行来调用开始评测。 + +```bash +python run.py \ + --models hf_llama2_7b \ + --custom-dataset-path xxx/test_mcq.csv \ + --custom-dataset-data-type mcq \ + --custom-dataset-infer-method ppl +``` + +```bash +python run.py \ + --models hf_llama2_7b \ + --custom-dataset-path xxx/test_qa.jsonl \ + --custom-dataset-data-type qa \ + --custom-dataset-infer-method gen +``` + +在绝大多数情况下，`--custom-dataset-data-type` 和 `--custom-dataset-infer-method` 可以省略，OpenCompass 会根据以下逻辑进行设置： + +- 如果从数据集文件中可以解析出选项，如 `A`, `B`, `C` 等，则认定该数据集为 `mcq`，否则认定为 `qa`。 +- 默认 `infer_method` 为 `gen`。 + +### 配置文件 + +在原配置文件中，直接向 `datasets` 变量中添加新的项即可即可。自定义数据集亦可与普通数据集混用。 + +```python +datasets = [ + {"path": "xxx/test_mcq.csv", "data_type": "mcq", "infer_method": "ppl"}, + {"path": "xxx/test_qa.jsonl", "data_type": "qa", "infer_method": "gen"}, +] +``` + +### 数据集补充信息 `.meta.json` + +OpenCompass 会默认尝试对输入的数据集文件进行解析，因此在绝大多数情况下，`.meta.json` 文件都是 **不需要** 的。但是，如果数据集的字段名不是默认的字段名，或者需要自定义提示词，则需要在 `.meta.json` 文件中进行指定。 + +我们会在数据集同级目录下，以文件名+`.meta.json` 的形式放置一个表征数据集使用方法的文件，样例文件结构如下： + +```tree +. +├── test_mcq.csv +├── test_mcq.csv.meta.json +├── test_qa.jsonl +└── test_qa.jsonl.meta.json +``` + +该文件可能字段如下： + +- `abbr` (str): 数据集缩写，作为该数据集的 ID。 +- `data_type` (str): 数据集类型，可选值为 `mcq` 和 `qa`. +- `infer_method` (str): 推理方法，可选值为 `ppl` 和 `gen`. +- `human_prompt` (str): 用户提示词模板，用于生成提示词。模板中的变量使用 `{}` 包裹，如 `{question}`，`{opt1}` 等。如存在 `template`，则该字段会被忽略。 +- `bot_prompt` (str): 机器人提示词模板，用于生成提示词。模板中的变量使用 `{}` 包裹，如 `{answer}` 等。如存在 `template`，则该字段会被忽略。 +- `template` (str or dict): 问题模板，用于生成提示词。模板中的变量使用 `{}` 包裹，如 `{question}`，`{opt1}` 等。相关语法见[此处](../prompt/prompt_template.md) 关于 `infer_cfg['prompt_template']['template']` 的内容。 +- `input_columns` (list): 输入字段列表，用于读入数据。 +- `output_column` (str): 输出字段，用于读入数据。 +- `options` (list): 选项列表，用于读入数据，仅在 `data_type` 为 `mcq` 时有效。 + +样例如下： + +```json +{ + "human_prompt": "Question: 127 + 545 + 588 + 620 + 556 + 199 =\nA. 2632\nB. 2635\nC. 2645\nAnswer: Let's think step by step, 127 + 545 + 588 + 620 + 556 + 199 = 672 + 588 + 620 + 556 + 199 = 1260 + 620 + 556 + 199 = 1880 + 556 + 199 = 2436 + 199 = 2635. So the answer is B.\nQuestion: {question}\nA. {A}\nB. {B}\nC. {C}\nAnswer: ", + "bot_prompt": "{answer}" +} +``` + +或者 + +```json +{ + "template": "Question: {my_question}\nX. {X}\nY. {Y}\nZ. {Z}\nW. {W}\nAnswer:", + "input_columns": ["my_question", "X", "Y", "Z", "W"], + "output_column": "my_answer", +} +``` diff --git a/docs/zh_cn/advanced_guides/evaluation_lightllm.md b/docs/zh_cn/advanced_guides/evaluation_lightllm.md new file mode 100644 index 0000000000000000000000000000000000000000..8b425d5cba597fdaa8f0c11e046093966a1c6437 --- /dev/null +++ b/docs/zh_cn/advanced_guides/evaluation_lightllm.md @@ -0,0 +1,71 @@ +# 评测 Lightllm 模型 + +我们支持评测使用 [Lightllm](https://github.com/ModelTC/lightllm) 进行推理的大语言模型。Lightllm 是由商汤科技开发，是一个基于 Python 的 LLM 推理和服务框架，以其轻量级设计、易于扩展和高速性能而著称，Lightllm 对多种大模型都进行了支持。用户可以通过 Lightllm 进行模型推理，并且以服务的形式在本地起起来，在评测过程中，OpenCompass 通过 api 将数据喂给Lightllm，并对返回的结果进行处理。OpenCompass 对 Lightllm 进行了适配，本教程将介绍如何使用 OpenCompass 来对以 Lightllm 作为推理后端的模型进行评测。 + +## 环境配置 + +### 安装 OpenCompass + +请根据 OpenCompass [安装指南](https://opencompass.readthedocs.io/en/latest/get_started/installation.html) 来安装算法库和准备数据集。 + +### 安装 Lightllm + +请根据 [Lightllm 主页](https://github.com/ModelTC/lightllm) 来安装 Lightllm。注意对齐相关依赖库的版本，尤其是 transformers 的版本。 + +## 评测 + +我们以 llama2-7B 评测 humaneval 作为例子来介绍如何评测。 + +### 第一步: 将模型通过 Lightllm 在本地以服务的形式起起来 + +```shell +python -m lightllm.server.api_server --model_dir /path/llama2-7B \ + --host 0.0.0.0 \ + --port 1030 \ + --nccl_port 2066 \ + --max_req_input_len 4096 \ + --max_req_total_len 6144 \ + --tp 1 \ + --trust_remote_code \ + --max_total_token_num 120000 +``` + +**注：** 上述命令可以通过 tp 的数量设置，在 tp 张卡上进行 TensorParallel 推理，适用于较大的模型的推理。 + +**注：** 上述命令中的 max_total_token_num，会影响测试过程中的吞吐性能，可以根据 [Lightllm 主页](https://github.com/ModelTC/lightllm) 上的文档，进行设置。只要不爆显存，往往设置越大越好。 + +**注：** 如果要在同一个机器上起多个 Lightllm 服务，需要重新设定上面的 port 和 nccl_port。 + +可以使用下面的 Python 脚本简单测试一下当前服务是否已经起成功 + +```python +import time +import requests +import json + +url = 'http://localhost:8080/generate' +headers = {'Content-Type': 'application/json'} +data = { + 'inputs': 'What is AI?', + "parameters": { + 'do_sample': False, + 'ignore_eos': False, + 'max_new_tokens': 1024, + } +} +response = requests.post(url, headers=headers, data=json.dumps(data)) +if response.status_code == 200: + print(response.json()) +else: + print('Error:', response.status_code, response.text) +``` + +### 第二步: 使用 OpenCompass 评测上述模型 + +```shell +python run.py examples/eval_lightllm.py +``` + +当模型完成推理和指标计算后，我们便可获得模型的评测结果。 + +**注：** `eval_lightllm.py` 中，配置的 url 要和上一步服务地址对齐。 diff --git a/docs/zh_cn/advanced_guides/evaluation_lmdeploy.md b/docs/zh_cn/advanced_guides/evaluation_lmdeploy.md new file mode 100644 index 0000000000000000000000000000000000000000..14bcbc6bd21110f2a90445a4ee2a76d907fcd11f --- /dev/null +++ b/docs/zh_cn/advanced_guides/evaluation_lmdeploy.md @@ -0,0 +1,86 @@ +# 使用 LMDeploy 加速评测 + +我们支持在评测大语言模型时，使用 [LMDeploy](https://github.com/InternLM/lmdeploy) 作为推理加速引擎。LMDeploy 是涵盖了 LLM 和 VLM 任务的全套轻量化、部署和服务解决方案，拥有卓越的推理性能。本教程将介绍如何使用 LMDeploy 加速对模型的评测。 + +## 环境配置 + +### 安装 OpenCompass + +请根据 OpenCompass [安装指南](https://opencompass.readthedocs.io/en/latest/get_started/installation.html) 来安装算法库和准备数据集。 + +### 安装 LMDeploy + +使用 pip 安装 LMDeploy (python 3.8+)： + +```shell +pip install lmdeploy +``` + +LMDeploy 预编译包默认基于 CUDA 12 编译。如果需要在 CUDA 11+ 下安装 LMDeploy，请执行以下命令： + +```shell +export LMDEPLOY_VERSION=0.6.0 +export PYTHON_VERSION=310 +pip install https://github.com/InternLM/lmdeploy/releases/download/v${LMDEPLOY_VERSION}/lmdeploy-${LMDEPLOY_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux2014_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118 +``` + +## 评测 + +在评测一个模型时，需要准备一份评测配置，指明评测集、模型和推理参数等信息。 + +以 [internlm2-chat-7b](https://huggingface.co/internlm/internlm2-chat-7b) 模型为例，相关的配置信息如下： + +```python +# configure the dataset +from mmengine.config import read_base + + +with read_base(): + # choose a list of datasets + from .datasets.mmlu.mmlu_gen_a484b3 import mmlu_datasets + from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets + from .datasets.triviaqa.triviaqa_gen_2121ce import triviaqa_datasets + from opencompass.configs.datasets.gsm8k.gsm8k_0shot_v2_gen_a58960 import \ + gsm8k_datasets + # and output the results in a chosen format + from .summarizers.medium import summarizer + +datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), []) + +# configure lmdeploy +from opencompass.models import TurboMindModelwithChatTemplate + + + +# configure the model +models = [ + dict( + type=TurboMindModelwithChatTemplate, + abbr=f'internlm2-chat-7b-lmdeploy', + # model path, which can be the address of a model repository on the Hugging Face Hub or a local path + path='internlm/internlm2-chat-7b', + # inference backend of LMDeploy. It can be either 'turbomind' or 'pytorch'. + # If the model is not supported by 'turbomind', it will fallback to + # 'pytorch' + backend='turbomind', + # For the detailed engine config and generation config, please refer to + # https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/messages.py + engine_config=dict(tp=1), + gen_config=dict(do_sample=False), + # the max size of the context window + max_seq_len=7168, + # the max number of new tokens + max_out_len=1024, + # the max number of prompts that LMDeploy receives + # in `generate` function + batch_size=5000, + run_cfg=dict(num_gpus=1), + ) +] +``` + +把上述配置放在文件中，比如 "configs/eval_internlm2_lmdeploy.py"。然后，在 OpenCompass 的项目目录下，执行如下命令可得到评测结果： + +```shell +python run.py configs/eval_internlm2_lmdeploy.py -w outputs +``` diff --git a/docs/zh_cn/advanced_guides/llm_judge.md b/docs/zh_cn/advanced_guides/llm_judge.md new file mode 100644 index 0000000000000000000000000000000000000000..80e0a1119cb2c211e9378ac49b17433bf29aefd6 --- /dev/null +++ b/docs/zh_cn/advanced_guides/llm_judge.md @@ -0,0 +1,368 @@ +# LLM 作为评判器 + +## 简介 + +GenericLLMEvaluator组件特别适用于那些难以通过规则式方法（如正则表达式）进行完美判断的场景，例如： + +- 模型不输出选项标识而只输出选项内容的情况 +- 需要事实性判断的数据集 +- 需要复杂理解和推理的开放式回答 +- 需要设计大量规则的判断 + +OpenCompass提供了GenericLLMEvaluator组件来实现LLM作为评判器的评估。 + +## 数据集格式 + +用于LLM评判的数据集应该是JSON Lines (.jsonl)或CSV格式。每个条目至少应包含： + +- 问题或任务 +- 参考答案或标准答案 +- (模型的预测将在评估过程中生成) + +JSONL格式示例： + +```json +{"problem": "法国的首都是什么？", "answer": "巴黎"} +``` + +CSV格式示例： + +```csv +problem,answer +"法国的首都是什么？","巴黎" +``` + +## 配置说明 + +### 基于命令行使用LLM进行评估 + +OpenCompass中部分数据集已经包含了LLM评判器的配置。 +你需要使用一个模型服务（如OpenAI或DeepSeek官方提供的API）或本地使用LMDeploy、vLLM、SGLang等工具启动一个模型服务。 + +然后，你可以通过以下命令设置相关评估服务的环境变量，并对模型进行评估： + +```bash +export OC_JUDGE_MODEL=Qwen/Qwen2.5-32B-Instruct +export OC_JUDGE_API_KEY=sk-1234 +export OC_JUDGE_API_BASE=http://172.30.56.1:4000/v1 +``` + +注意，默认情况下，OpenCompass会使用这三个环境变量，但如果你使用了基于配置文件的方式配置评估服务，这三个环境变量将不会生效。 + +### 基于配置文件使用LLM进行评估 + +对一个数据集设置LLM评判评估，你需要配置三个主要组件： + +1. 数据集读取配置 + +```python +reader_cfg = dict( + input_columns=['problem'], # 问题列的名称 + output_column='answer' # 参考答案列的名称 +) +``` + +2. 推理配置 + +```python +infer_cfg = dict( + prompt_template=dict( + type=PromptTemplate, + template=dict( + round=[ + dict( + role='HUMAN', + prompt='{problem}', # 提示模型的模板 + ), + ] + ), + ), + retriever=dict(type=ZeroRetriever), + inferencer=dict(type=GenInferencer), +) +``` + +3. 使用LLM评判器的评估配置 + +```python +eval_cfg = dict( + evaluator=dict( + type=GenericLLMEvaluator, # 使用LLM作为评估器 + prompt_template=dict( + type=PromptTemplate, + template=dict( + begin=[ + dict( + role='SYSTEM', + fallback_role='HUMAN', + prompt="你是一个负责评估模型输出正确性和质量的助手。", + ) + ], + round=[ + dict(role='HUMAN', prompt=YOUR_JUDGE_TEMPLATE), # 评判器的模板 + ], + ), + ), + dataset_cfg=dict( + type=CustomDataset, + path='path/to/your/dataset', + file_name='your_dataset.jsonl', + reader_cfg=reader_cfg, + ), + judge_cfg=YOUR_JUDGE_MODEL_CONFIG, # 评判模型的配置 + dict_postprocessor=dict(type=generic_llmjudge_postprocess), # 处理评判器输出的后处理器 + ), +) +``` + +## 使用CustomDataset和GenericLLMEvaluator + +以下是如何设置完整的LLM评判评估配置： + +```python +from mmengine.config import read_base +from opencompass.models import TurboMindModelwithChatTemplate +from opencompass.datasets import CustomDataset +from opencompass.evaluator import GenericLLMEvaluator +from opencompass.datasets import generic_llmjudge_postprocess +from opencompass.openicl.icl_prompt_template import PromptTemplate +from opencompass.openicl.icl_retriever import ZeroRetriever +from opencompass.openicl.icl_inferencer import GenInferencer + +# 导入评判模型配置 +with read_base(): + from opencompass.configs.models.qwen2_5.lmdeploy_qwen2_5_14b_instruct import ( + models as judge_model, + ) + +# 定义评判模板 +JUDGE_TEMPLATE = """ +请评估以下回答是否正确地回答了问题。 +问题：{problem} +参考答案：{answer} +模型回答：{prediction} + +模型回答是否正确？如果正确，请回答"A"；如果不正确，请回答"B"。 +""".strip() + +# 数据集读取配置 +reader_cfg = dict(input_columns=['problem'], output_column='answer') + +# 被评估模型的推理配置 +infer_cfg = dict( + prompt_template=dict( + type=PromptTemplate, + template=dict( + round=[ + dict( + role='HUMAN', + prompt='{problem}', + ), + ] + ), + ), + retriever=dict(type=ZeroRetriever), + inferencer=dict(type=GenInferencer), +) + +# 使用LLM评判器的评估配置 +eval_cfg = dict( + evaluator=dict( + type=GenericLLMEvaluator, + prompt_template=dict( + type=PromptTemplate, + template=dict( + begin=[ + dict( + role='SYSTEM', + fallback_role='HUMAN', + prompt="你是一个负责评估模型输出正确性和质量的助手。", + ) + ], + round=[ + dict(role='HUMAN', prompt=JUDGE_TEMPLATE), + ], + ), + ), + dataset_cfg=dict( + type=CustomDataset, + path='path/to/your/dataset', + file_name='your_dataset.jsonl', + reader_cfg=reader_cfg, + ), + judge_cfg=judge_model[0], + dict_postprocessor=dict(type=generic_llmjudge_postprocess), + ), + pred_role='BOT', +) + +# 数据集配置 +datasets = [ + dict( + type=CustomDataset, + abbr='my-dataset', + path='path/to/your/dataset', + file_name='your_dataset.jsonl', + reader_cfg=reader_cfg, + infer_cfg=infer_cfg, + eval_cfg=eval_cfg, + ) +] + +# 被评估模型的配置 +models = [ + dict( + type=TurboMindModelwithChatTemplate, + abbr='model-to-evaluate', + path='path/to/your/model', + # ... 其他模型配置 + ) +] + +# 输出目录 +work_dir = './outputs/llm_judge_eval' +``` + +## GenericLLMEvaluator + +GenericLLMEvaluator专为使用LLM作为评判器评估模型输出而设计。主要特点包括： + +1. 灵活的提示模板，用于指导评判器 +2. 支持各种评判模型（本地或基于API） +3. 通过提示工程自定义评估标准 +4. 对评判器输出进行后处理以提取结构化评估 + +**重要说明**：目前通用版本的评判模板只支持输出"A"（正确）或"B"（不正确）的格式，不支持其他输出格式（如"正确"或"不正确"）。这是因为后处理函数`generic_llmjudge_postprocess`专门设计为解析这种格式。 + +评估器的工作原理： + +1. 获取原始问题、参考答案和模型预测 +2. 将它们格式化为评判模型的提示 +3. 解析评判器的响应以确定评估结果（寻找"A"或"B"） +4. 汇总整个数据集的结果 + +如果需要查看评估的详细结果，可以在启动任务时添加`--dump-eval-details`到命令行。 +评估输出示例： + +```python +{ + 'accuracy': 75.0, # 被判断为正确的回答百分比 + 'details': [ + { + 'origin_prompt': """ + 请评估以下回答是否正确地回答了问题。 + 问题：法国的首都是什么？ + 参考答案：巴黎 + 模型回答：法国的首都是巴黎。 + 模型回答是否正确？如果正确，请回答"A"；如果不正确，请回答"B"。""", + 'gold': '巴黎', + 'prediction': 'A', + }, + # ... 更多结果 + ] +} +``` + +## 级联评估器 (CascadeEvaluator) + +OpenCompass还提供了级联评估器`CascadeEvaluator`，它结合了规则式评估和LLM评估的优势。级联评估器有两种模式： + +1. **级联模式（Cascade Mode, parallel=False）**：首先使用规则式评估器评估所有样本，然后只将规则式评估认为不正确的样本发送给LLM评判器进行重新评估。这种方式可以在保持准确性的同时减少对LLM评判的依赖，从而降低评估成本和时间。 + +2. **并行模式（Parallel Mode, parallel=True）**：使用规则式评估器和LLM评判器同时评估所有样本，如果任何一个评估器认为样本是正确的，则将该样本视为正确。这种方式可以提高评估的宽容度，但可能会导致更高的成本，因为所有样本都需要LLM评估。 + +### 配置CascadeEvaluator + +以下是配置`CascadeEvaluator`的示例： + +```python +# 定义规则式评估器 +rule_evaluator = dict(type=MATHVerifyEvaluator) + +# 定义LLM评判器 +llm_judge_evaluator = dict( + type=GenericLLMEvaluator, + prompt_template=dict( + type=PromptTemplate, + template=dict( + begin=[ + dict( + role='SYSTEM', + fallback_role='HUMAN', + prompt="你是一个负责评估模型输出正确性和质量的助手。", + ) + ], + round=[ + dict(role='HUMAN', prompt=YOUR_JUDGE_TEMPLATE), + ], + ), + ), + dataset_cfg=dict( + type=YourDataset, + path='path/to/your/dataset', + reader_cfg=reader_cfg, + ), + judge_cfg=dict(), # 可以使用环境变量配置评判模型 +) + +# 配置级联评估器（级联模式） +cascade_evaluator = dict( + type=CascadeEvaluator, + llm_evaluator=llm_judge_evaluator, + rule_evaluator=rule_evaluator, + parallel=False # 级联模式 +) + +# 如果需要并行模式，可以设置parallel=True +parallel_evaluator = dict( + type=CascadeEvaluator, + llm_evaluator=llm_judge_evaluator, + rule_evaluator=rule_evaluator, + parallel=True # 并行模式 +) + +# 在数据集评估配置中使用级联评估器 +eval_cfg = dict(evaluator=cascade_evaluator) +``` + +### 评估结果 + +级联评估器会输出详细的评估统计信息，包括： + +- 规则评估的准确率 +- LLM评估的准确率（针对规则评估失败的样本） +- 最终的综合准确率 + +输出示例： + +```python +{ + 'accuracy': 85.0, # 最终准确率 + 'cascade_stats': { + 'total_samples': 100, + 'rule_correct': 70, # 规则评估认为正确的样本数 + 'rule_accuracy': 70.0, # 规则评估的准确率 + 'llm_evaluated': 30, # LLM评估的样本数（级联模式下为规则评估失败的样本数） + 'llm_correct': 15, # LLM评估认为正确的样本数 + 'llm_accuracy': 50.0, # LLM评估的准确率 + 'final_correct': 85, # 最终正确的样本数 + 'final_accuracy': 85.0, # 最终准确率 + 'parallel_mode': False, # 是否是并行模式 + }, + 'details': [ + # 每个样本的详细评估结果 + ] +} +``` + +级联评估器特别适用于： + +1. 需要平衡评估成本和准确性的场景 +2. 有可用的规则式评估器但可能不够完善的情况 +3. 需要对边界情况进行更精确判断的评估任务 + +## 完整示例 + +如果希望了解通用LLM评判器，请参考examples目录中的`eval_llm_judge.py`文件，该示例展示了如何使用LLM评判器评估数学问题。 + +如果希望了解级联评估器请参考examples目录中的`eval_cascade_evaluator.py`文件，该示例展示了如何使用级联评估器评估数学问题。 diff --git a/docs/zh_cn/advanced_guides/longeval.md b/docs/zh_cn/advanced_guides/longeval.md new file mode 100644 index 0000000000000000000000000000000000000000..1fa43f0409bb1f64d8cedd4ea6b1ebcf8b02ff40 --- /dev/null +++ b/docs/zh_cn/advanced_guides/longeval.md @@ -0,0 +1,169 @@ +# 长文本评测指引 + +## 介绍 + +虽然大语言模型（LLM）如GPT-4在处理自然语言任务已经展现出明显的优势，但目前的开源模型大多只能处理数千个token长度以内的文本，这限制了模型阅读书籍、撰写文本摘要等需要处理长文本的能力。为了探究模型在应对长文本能力时的表现，我们采用[L-Eval](https://github.com/OpenLMLab/LEval)和[LongBench](https://github.com/THUDM/LongBench)两个长文本数据集来测试模型长文本能力。 + +## 现有算法及模型 + +在处理长文本输入时，推理时间开销和灾难性遗忘是大模型面临的两大主要挑战。最近，大量研究致力于扩展模型长度，这些研究集中于以下三个改进方向。 + +- 注意力机制。这些方法的最终目的多为减少query-key对的计算开销，但可能对下游任务的效果产生影响。 +- 输入方法。部分研究将长文本输入分块或将部分已有文本段重复输入模型以增强模型处理长文本能力，但这些方法只对部分任务有效，难以适应多种下游任务。 +- 位置编码。这部分研究包括RoPE, ALiBi，位置插值等，在长度外推方面展现出了良好的效果。这些方法已经被用于训练如ChatGLM2-6b-32k和LongChat-32k等长文本模型。 + +首先，我们介绍一些流行的位置编码算法。 + +### RoPE + +RoPE是一种在Transformer中注入位置信息的位置嵌入方法。它使用旋转矩阵对绝对位置进行编码，并同时在自注意力公式中融入显式的相对位置依赖关系。下图是RoPE机制的一个示例。 + +| 完整写法 | +省略写法 | +
|---|---|
| + +```python +infer_cfg=dict( + ice_template=dict( + type=PromptTemplate, + template="Q: {question}\nA: {answer}", + ), + prompt_template=dict( + type=PromptTemplate, + template="Q: {question}\nA: {answer}", + ice_token="", + ), + # ... +) +``` + + | ++ +```python +infer_cfg=dict( + ice_template=dict( + type=PromptTemplate, + template="Q: {question}\nA: {answer}", + ice_token="", + ), + # ... +) +``` + + | +
| Benchmarks | +CN-Lang | +CSR | +CN-specifics | +Dual-Domain | +Rea-Mem | +
|---|---|---|---|---|---|
| Most benchmarks in davis2023benchmarks | +✘ | +✔ | +✘ | +✘ | +✘ | +
| XNLI, XCOPA,XStoryCloze | +✔ | +✔ | +✘ | +✘ | +✘ | +
| LogiQA, CLUE, CMMLU | +✔ | +✘ | +✔ | +✘ | +✘ | +
| CORECODE | +✔ | +✔ | +✘ | +✘ | +✘ | +
| CHARM (ours) | +✔ | +✔ | +✔ | +✔ | +✔ | +
| 基准 | +汉语 | +常识推理 | +中国特有知识 | +中国和世界知识域 | +推理和记忆的关系 | +
|---|---|---|---|---|---|
| davis2023benchmarks 中提到的基准 | +✘ | +✔ | +✘ | +✘ | +✘ | +
| XNLI, XCOPA,XStoryCloze | +✔ | +✔ | +✘ | +✘ | +✘ | +
| LogiQA,CLUE, CMMLU | +✔ | +✘ | +✔ | +✘ | +✘ | +
| CORECODE | +✔ | +✔ | +✘ | +✘ | +✘ | +
| CHARM (ours) | +✔ | +✔ | +✔ | +✔ | +✔ | +