File size: 3,206 Bytes
9b57ce7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
## Model Performance Evaluation (`evaluate.py`)
This script is used to evaluate the model's performance on a test set. It can operate in two modes:
- **`pair`**: Calculates pairwise accuracy.
- **`ranking`**: Calculates ranking accuracy.
**Pair-wise Sample**
We set path1's image is better than path2's image for simplicity.
```json
[
{
"prompt": ".....",
"path1": ".....",
"path2": "....."
},
{
"prompt": ".....",
"path1": ".....",
"path2": "....."
},
...
]
```
**Rank-wise Sample**
```json
[
{
"id": "005658-0040",
"prompt": ".....",
"generations": [
"path to image1",
"path to image2",
"path to image3",
"path to image4"
],
"ranking": [
1,
2,
5,
3
]
},
...
]
```
### Usage
```bash
python evaluate/evaluate.py \
--test_json /path/to/your/test_data.json \
--config_path config/HPSv3_7B.yaml \
--checkpoint_path checkpoints/HPSv3_7B/model.pth \
--mode pair \
--batch_size 8 \
--num_processes 8
```
**Arguments:**
- `--test_json`: (Required) Path to the JSON file containing evaluation data.
- `--config_path`: (Required) Path to the model's configuration file.
- `--checkpoint_path`: (Required) Path to the model checkpoint.
- `--mode`: The evaluation mode. Can be `pair` or `ranking`. (Default: `pair`)
- `--batch_size`: Batch size for inference. (Default: 8)
- `--num_processes`: Number of parallel processes to use. (Default: 8)
---
## Reward Benchmarking (`benchmark.py`)
This script is used to run inference with a reward model over one or more folders of images. It calculates a reward score for each image based on its corresponding text prompt (expected in a `.txt` file with the same name). The script then outputs statistics (mean, std, min, max) for each folder and saves the detailed results to a JSON file.
It supports multiple reward models through the `--model_type` argument.
### Usage
The script is run using `argparse`. Below is a command-line example:
```bash
python evaluate/benchmark.py \
--config_path config/HPSv3_7B.yaml \
--checkpoint_path checkpoints/HPSv3_7B/model.pth \
--model_type hpsv3 \
--image_folders /path/to/images/folder1 /path/to/images/folder2 \
--output_path ./benchmark_results.json \
--batch_size 16 \
--num_processes 8
```
**Arguments:**
- `--config_path`: (Required) Path to the model's configuration file.
- `--checkpoint_path`: (Required) Path to the model checkpoint.
- `--model_type`: The reward model to use. Choices: `hpsv3`, `hpsv2`, `imagereward`. (Default: `hpsv3`)
- `--image_folders`: (Required) One or more paths to folders containing the images to benchmark.
- `--output_path`: (Required) Path to save the output JSON file with results.
- `--batch_size`: Batch size for processing. (Default: 16)
- `--num_processes`: Number of parallel processes to use. (Default: 8)
- `--num_machines`: For distributed inference, the total number of machines. (Default: 1)
- `--machine_id`: For distributed inference, the ID of the current machine. (Default: 0)
|