README.md

---
license: apache-2.0
language:
- en
metrics:
- name: accuracy
  valud: 75.39
base_model:
- Qwen/Qwen3-Reranker-0.6B
pipeline_tag: text-classification
---

# EduBenchEvaluator

This is a fine-tuned evaluator designed to assess LLM on the **EduBench** benchmark. 

- 📄 **[Paper](https://arxiv.org/abs/2505.16160)**
- 💻 **[GitHub Repository](https://github.com/DIRECT-BIT/EduBench)**

## Model Details

* **Model Name**: EduBenchEvaluator
* **Model Type**: Fine-tuned language model (0.6B parameters)
* **Base Model**: [Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)

## Training & Methodology

The base model, `Qwen3-Reranker-0.6B`, was fine-tuned to align with human evaluations on the EduBench dataset. 

We approached the fine-tuning process as a text classification task. The model evaluates a given response by taking a `<question, answer, metric>` triplet as input. Based on this context, it is trained to output a precise evaluation score ranging from **1 to 5**. 

This evaluator is specifically constructed to measure an LLM's capability across the diverse educational tasks presented in EduBench. 

## Performance

* **Accuracy**: The model achieves a satisfactory accuracy of **75.28%** on the test set.
* **Human Alignment**: In addition to standard accuracy, we calculated the correlation between the model's predictions and actual human scorers, demonstrating that the model closely mirrors human judgment.

*Note: Further evaluation results and comparisons are reported on our [GitHub](https://github.com/DIRECT-BIT/EduBench).*

## 🫣 Citation

If you find our benchmark, evaluation pipeline, or models useful or interesting, please cite our paper:

```bibtex
@misc{xu2025edubenchcomprehensivebenchmarkingdataset,
      title={EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios}, 
      author={Bin Xu and Yu Bai and Huashan Sun and Yiguan Lin and Siming Liu and Xinyue Liang and Yaolin Li and Yang Gao and Heyan Huang},
      year={2025},
      eprint={2505.16160},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={[https://arxiv.org/abs/2505.16160](https://arxiv.org/abs/2505.16160)}, 
}