upload dataset folder to repo

README.md

@@ -8,156 +8,200 @@ tags:
 - transformers
 ---
 
-<p align="center">
-  <img src="https://github.com/VectorSpaceLab/EditScore/assets/logo.png" width="65%">
-</p>
+# Model Card for Model ID
 
-<p align="center">
-  <a href="https://vectorspacelab.github.io/EditScore"><img src="https://img.shields.io/badge/Project%20Page-EditScore-yellow" alt="project page"></a>
-  <a href="https://arxiv.org/abs/2506.18871"><img src="https://img.shields.io/badge/arXiv%20paper-2506.18871-b31b1b.svg" alt="arxiv"></a>
-  <a href="https://huggingface.co/collections/EditScore/editscore-68d8e27ee676981221db3cfe"><img src="https://img.shields.io/badge/EditScore-🤗-yellow" alt="model"></a>
-  <a href="https://huggingface.co/datasets/EditScore/EditReward-Bench"><img src="https://img.shields.io/badge/EditReward--Bench-🤗-yellow" alt="dataset"></a>
-</p>
+<!-- Provide a quick summary of what the model is/does. -->
 
-<h4 align="center">
-    <p>
-        <a href=#-news>News</a> |
-        <a href=#-quick-start>Quick Start</a> |
-        <a href=#-usage-tips>Usage Tips</a> |
-        <a href=#-limitations-and-suggestions>Limitations</a> |
-        <a href=#-gradio-demo>Online Demos</a> |
-        <a href=#%EF%B8%8F-citing-us>Citation</a>
-    <p>
-</h4>
 
-**EditScore** is a series of state-of-the-art open-source reward models (7B–72B) designed to evaluate and enhance instruction-guided image editing.
-## ✨ Highlights
-- **State-of-the-Art Performance**: Effectively matches the performance of leading proprietary VLMs. With a self-ensembling strategy, **our largest model surpasses even GPT-5** on our comprehensive benchmark, **EditReward-Bench**.
-- **A Reliable Evaluation Standard**: We introduce **EditReward-Bench**, the first public benchmark specifically designed for evaluating reward models in image editing, featuring 13 subtasks, 11 state-of-the-art editing models (*including proprietary models*) and expert human annotations.
-- **Simple and Easy-to-Use**: Get an accurate quality score for your image edits with just a few lines of code.
-- **Versatile Applications**: Ready to use as a best-in-class reranker to improve editing outputs, or as a high-fidelity reward signal for **stable and effective Reinforcement Learning (RL) fine-tuning**.
 
-## 🔥 News
-- **2025-09-29**: We are excited to release **EditScore** and **EditReward-Bench**! Model weights and the benchmark dataset are now publicly available. You can access them on Hugging Face: [Models Collection](https://huggingface.co/collections/EditScore/editscore-68d8e27ee676981221db3cfe) and [Benchmark Dataset](https://huggingface.co/datasets/EditScore/EditReward-Bench), and on ModelScope: [Models Collection](https://www.modelscope.cn/collections/EditScore-8b0d53aa945d4e) and [Benchmark Dataset](https://www.modelscope.cn/datasets/EditScore/EditReward-Bench).
+## Model Details
 
-## 📖 Introduction
-While Reinforcement Learning (RL) holds immense potential for this domain, its progress has been severely hindered by the absence of a high-fidelity, efficient reward signal.
+### Model Description
 
-To overcome this barrier, we provide a systematic, two-part solution:
+<!-- Provide a longer summary of what this model is. -->
 
-- **A Rigorous Evaluation Standard**: We first introduce **EditReward-Bench**, a new public benchmark for the direct and reliable evaluation of reward models. It features 13 diverse subtasks and expert human annotations, establishing a gold standard for measuring reward signal quality.
 
-- **A Powerful & Versatile Tool**: Guided by our benchmark, we developed the **EditScore** model series. Through meticulous data curation and an effective self-ensembling strategy, EditScore sets a new state of the art for open-source reward models, even surpassing the accuracy of leading proprietary VLMs.
 
-<p align="center">
-  <img src="https://github.com/VectorSpaceLab/EditScore/assets/table_reward_model_results.png" width="95%">
-  <br>
-  <em>Benchmark results on EditReward-Bench.</em>
-</p>
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-We demonstrate the practical utility of EditScore through two key applications:
+### Model Sources [optional]
 
-- **As a State-of-the-Art Reranker**: Use EditScore to perform Best-of-*N* selection and instantly improve the output quality of diverse editing models.
-- **As a High-Fidelity Reward for RL**: Use EditScore as a robust reward signal to fine-tune models via RL, enabling stable training and unlocking significant performance gains where general-purpose VLMs fail.
+<!-- Provide the basic links for the model. -->
 
-This repository releases both the **EditScore** models and the **EditReward-Bench** dataset to facilitate future research in reward modeling, policy optimization, and AI-driven model improvement.
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
-<p align="center">
-  <img src="https://github.com/VectorSpaceLab/EditScore/assets/figure_edit_results.png" width="95%">
-  <br>
-  <em>EditScore as a superior reward signal for image editing.</em>
-</p>
+## Uses
 
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-## 📌 TODO
-We are actively working on improving EditScore and expanding its capabilities. Here's what's next:
-- [ ] Release RL training code applying EditScore to OmniGen2.
-- [ ] Provide Best-of-N inference scripts for OmniGen2, Flux-dev-Kontext, and Qwen-Image-Edit.
+### Direct Use
 
-## 🚀 Quick Start
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-### 🛠️ Environment Setup
+[More Information Needed]
 
-#### ✅ Recommended Setup
+### Downstream Use [optional]
 
-```bash
-# 1. Clone the repo
-git clone git@github.com:VectorSpaceLab/EditScore.git
-cd EditScore
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-# 2. (Optional) Create a clean Python environment
-conda create -n editscore python=3.12
-conda activate editscore
+[More Information Needed]
 
-# 3. Install dependencies
-# 3.1 Install PyTorch (choose correct CUDA version)
-pip install torch==2.7.1 torchvision --extra-index-url https://download.pytorch.org/whl/cu126
+### Out-of-Scope Use
 
-# 3.2 Install other required packages
-pip install -r requirements.txt
-```
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-#### 🌏 For users in Mainland China
+[More Information Needed]
 
-```bash
-# Install PyTorch from a domestic mirror
-pip install torch==2.7.1 torchvision --index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu126
+## Bias, Risks, and Limitations
 
-# Install other dependencies from Tsinghua mirror
-pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
-```
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
----
+[More Information Needed]
 
-### 🧪 Usage Example
-Using EditScore is straightforward. The model will be automatically downloaded from the Hugging Face Hub on its first run.
-```python
-from PIL import Image
-from editscore import EditScore
-
-# Load the EditScore model. It will be downloaded automatically.
-# Replace with the specific model version you want to use.
-model_path = "Qwen/Qwen2.5-VL-7B-Instruct"
-lora_path = "EditScore/EditScore-7B"
-
-scorer = EditScore(
-    backbone="qwen25vl", # set to "qwen25vl_vllm" for faster inference
-    model_name_or_path=model_path,
-    enable_lora=True,
-    lora_path=lora_path,
-    score_range=25,
-    num_pass=1, # Increase for better performance via self-ensembling
-)
-
-input_image = Image.open("example_images/input.png")
-output_image = Image.open("example_images/output.png")
-instruction = "Adjust the background to a glass wall."
-
-result = scorer.evaluate([input_image, output_image], instruction)
-print(f"Edit Score: {result['final_score']}")
-# Expected output: A dictionary containing the final score and other details.
-```
+### Recommendations
 
----
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
 
-## 📊 Benchmark Your Image-Editing Reward Model
-We provide an evaluation script to benchmark reward models on **EditReward-Bench**. To evaluate your own custom reward model, simply create a scorer class with a similar interface and update the script.
-```bash
-# This script will evaluate the default EditScore model on the benchmark
-bash evaluate.sh
-
-# Or speed up inference with VLLM
-bash evaluate_vllm.sh
-```
-
-## ❤️ Citing Us
-If you find this repository or our work useful, please consider giving a star ⭐ and citation 🦖, which would be greatly appreciated:
-
-```bibtex
-@article{luo2025editscore,
-  title={EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling},
-  author={Xin Luo and Jiahao Wang and Chenyuan Wu and Shitao Xiao and Xiyan Jiang and Defu Lian and Jiajun Zhang and Dong Liu and Zheng Liu},
-  journal={arXiv preprint arXiv:},
-  year={2025}
-}
-```
+- PEFT 0.17.1