winwin-77 commited on
Commit
ef70921
·
verified ·
1 Parent(s): 4d36574

Add README.md from MODEL_CARD_EN.md

Browse files
Files changed (1) hide show
  1. README.md +82 -0
README.md ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # EvoQuality
2
+
3
+ ## 1. Model Overview
4
+
5
+ - **Model Name**: EvoQuality (Self-Evolving VLM for Image Quality Assessment)
6
+ - **Task**: No-Reference Image Quality Assessment (NR-IQA), supporting both single-image quality scoring and pairwise quality comparison (ranking)
7
+ - **Core Idea**: Without relying on any human-annotated quality scores or distortion-type labels, EvoQuality generates pseudo-ranking labels via **pairwise majority voting**, and converts them into an optimizable reward signal through **GRPO** to iteratively self-evolve its quality perception capability
8
+ - **Paper**: [Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Ranking](https://openreview.net/forum?id=INOi0YqI8p) (ICLR 2026, arXiv:2509.25787)
9
+
10
+ ## 2. Model and Framework Details
11
+
12
+ - **Backbone Model (paper setting)**: `Qwen2.5-VL-7B` (used as the baseline policy)
13
+ - **Training Paradigm**: Two-stage cycle, supports multi-round iteration (`T=2` in the paper)
14
+ - **Offline Stage (Pseudo-label)**: Perform `K` comparisons on randomly sampled image pairs, then derive pseudo-preferences `p*(xi, xj)` via majority voting
15
+ - **Online Stage (RL)**: Convert pseudo-preferences into a fidelity reward and update the policy via **Group Relative Policy Optimization (GRPO)** (full fine-tuning of the VLM)
16
+
17
+ ## 3. Prompts
18
+
19
+ - **Offline Comparison** **`c_compare`**:
20
+ - `<image><image> You are performing an image quality assessment task. Compare the two images and decide which one has better perceptual quality. Answer strictly with the index of the better image: 0 if the first image is better, or 1 if the second image is better.`
21
+ - **Online Scoring** **`c_score`**:
22
+ - `<image> You are doing the image quality assessment task. Here is the question: What is your overall rating on the quality of this picture? The rating should be a float between 1 and 5, rounded to two decimal places, with 1 representing very poor quality and 5 representing excellent quality.`
23
+ - **Reasoning Suffix (for self-consistency sampling)**:
24
+ - `You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in boxed{}.`
25
+
26
+ ## 4. Training
27
+
28
+ - **Number of Iterations**: `T = 1` (the open-sourced model weights are the result of the first round of self-evolution)
29
+ - **Training Data**: No additional synthetic distortion data and no extra annotated labels were added when producing the released weights
30
+ - **Offline Stage**:
31
+ - Sample `K=32` responses per pair, then derive pseudo-labels via majority voting
32
+ - Randomly swap image order to mitigate positional bias
33
+ - **Online Stage (GRPO)**:
34
+ - Sample `K=32` responses per sample (`c_score`)
35
+ - Optimizer: AdamW, initial learning rate `3e-7`, with linear decay
36
+ - KL coefficient: `beta = 0.05`
37
+ - Resources (as reported in the paper): 8x NVIDIA A100, per-GPU batch size = 4, ~12 hours/epoch
38
+
39
+ ## 5. Evaluation Metrics
40
+
41
+ - **Evaluation Setting**: zero-shot (no training on the target test sets)
42
+ - **Metrics**: PLCC, SRCC (consistency with human subjective quality)
43
+
44
+ ## 6. Main Results
45
+
46
+ - **Improvement over the Backbone Model (Qwen2.5-VL-7B)**: weighted average (WA VG.) over multiple benchmarks
47
+ - PLCC: `0.615 -> 0.770` (+31.8%)
48
+ - SRCC: `0.570 -> 0.726` (+33.7%)
49
+ - **Generalization**: Achieves significant improvements across diverse distortion types and AI-generated content, matching or surpassing several supervised VLM-IQA approaches on multiple benchmarks (see the paper for detailed tables)
50
+
51
+ ## 7. Intended Use and Usage Guidelines
52
+
53
+ - **Recommended Use**
54
+ - Research and evaluation: NR-IQA, cross-dataset generalization comparison, quality ranking/filtering, auxiliary signals for data cleaning
55
+ - Pre-production assessment: as a perceptual quality proxy, but should be combined with business data and manual spot-check validation
56
+ - **Not Recommended Use**
57
+ - As the sole quality criterion for high-stakes decisions (content moderation, medical imaging diagnostic conclusions, legal evidence adjudication, etc.)
58
+ - Treating model outputs as "absolute objective ground truth" (IQA is inherently subjective and correlated with population preferences)
59
+ - **Output Notes**
60
+ - The paper's prompts require outputs in the form of `<think>...</think>` with `boxed{score}`; for actual integration, it is recommended to parse only the value inside `boxed{}` and consider how temperature/sampling strategies affect consistency
61
+
62
+ ## 8. Limitations and Known Risks
63
+
64
+ - **Self-supervised Pseudo-label Bias**: Pseudo-rankings are derived from the model's own votes, which may amplify the systematic preferences or blind spots of the backbone model
65
+ - **Domain Shift**: May fail on images from specific domains (medical, remote sensing, industrial inspection)
66
+ - **Subjectivity and Population Differences**: Different cultural/aesthetic preferences and task objectives (aesthetics vs. clarity) can change the definition of "quality"
67
+ - **Prompt Sensitivity**: Variations in prompts, sampling count K, and decoding strategies can affect self-consistency voting and final performance
68
+
69
+ <br />
70
+
71
+ <br />
72
+
73
+ ## 9. Citation
74
+
75
+ ```bibtex
76
+ @article{wen2025selfevolving,
77
+ title={Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Ranking},
78
+ author={Wen, Wen and Zhi, Tianwu and Fan, Kanglong and Li, Yang and Peng, Xinge and Zhang, Yabin and Liao, Yiting and Li, Junlin and Zhang, Li},
79
+ journal={arXiv preprint arXiv:2509.25787},
80
+ year={2025}
81
+ }
82
+ ```